IM Engineer Study Notes
← Back to list
Networking
#RDMA#RoCE#InfiniBand#커널바이패스#AI네트워크
Last updated · 2026-10-15

RDMA (Remote Direct Memory Access) and RoCE

1. Overview

A. Definition

RDMA (Remote Direct Memory Access) is a transport technology in which an application on one computer reads and writes the memory of a peer computer directly, without involvement of the operating system or CPU. The network adapter (RNIC/HCA) processes the transport protocol in hardware, bypasses the kernel (Kernel Bypass), and eliminates data copying (Zero-Copy)—three axes that together achieve ultra-low latency, high bandwidth, and low CPU utilization at once. RoCE (RDMA over Converged Ethernet) is a way of implementing this RDMA semantics on standard Ethernet rather than on a dedicated InfiniBand fabric.

To define RDMA in one phrase, it is "communication that strips out the middlemen—the operating system, the protocol stack, the intermediate copies—so that data goes straight from the sending application's buffer to the receiving application's buffer." In traditional TCP/IP communication, a single message is copied several times—application buffer → kernel socket buffer → NIC buffer—and each hop triggers a CPU interrupt and context switch. RDMA has the adapter DMA data directly into a pre-registered Memory Region, fully freeing the CPU from its role as "the porter that carries the data."

Terminologically, RDMA is the abstract semantics of "what to do (direct access to remote memory)," whereas InfiniBand, RoCE, and iWARP are implementations of "on which physical/transport layer to realize it." The three share the same upper-level programming interface (the Verbs API), so from an application's standpoint they largely run on the same code regardless of the underlying fabric. Thanks to this abstraction, RDMA has become a common asset not tied to a particular vendor or cabling technology.

The essential problem RDMA targets is often called the "I/O bottleneck" or the "data-movement wall." Compute capacity has grown steadily with Moore's law and accelerator advances, but the communication and memory paths that ferry data to those compute units have advanced more slowly, intensifying the phenomenon where overall system performance is constrained not by "computing" but by "getting the data in place in time." RDMA is a leading remedy that tears down this wall at the network layer, and it becomes the foundation of disaggregated architectures that treat a datacenter not as a collection of servers but as one large pool of memory and compute.

B. Background and Necessity

The fundamental reason RDMA rose is the gap between stagnating CPU speed and exploding network bandwidth. While server interfaces climbed from 10GbE to 100/200/400/800GbE, the speed of processing the TCP/IP stack on a general-purpose CPU did not grow proportionally. As a rule of thumb, if processing one bit of traffic requires roughly 1Hz of CPU, sustaining a 400Gbps link in software would require pouring hundreds of cores into communication alone. This means communication overhead eats into the very compute resources meant for applications, creating the paradox of "bandwidth to spare, but the CPU is the bottleneck."

The second is the spread of workloads where latency governs throughput. HPC's MPI communication, distributed storage, in-memory databases, and lately large-scale AI training all have thousands of nodes frequently exchanging small messages and waiting on one another. The tens of microseconds of latency and jitter added by the TCP/IP stack sharply degrade overall performance in such synchronization-bound workloads. RDMA lowers latency to the 1–2 microsecond range through kernel bypass and suppresses jitter, fundamentally changing the efficiency of workloads dominated by "waiting."

The third is the communication bottleneck of the AI-accelerator era. Even if a GPU pours out hundreds of TFLOPS, if data cannot be supplied to that GPU without interruption, the expensive accelerator falls idle. In training clusters bundling thousands of GPUs, collective communication such as All-Reduce occupies a substantial share of total training time, and processing it over a CPU software path makes GPU utilization plummet. As paths like GPUDirect RDMA—where the NIC reads and writes GPU memory directly—became essential, RDMA turned from a choice into a basic premise of AI infrastructure.

These three pressures are not independent but mutually amplifying. As bandwidth rises (first), the limits of software processing become stark; as latency-sensitive workloads grow (second), those limits translate directly into performance loss; and the AI accelerator (third) pushes both to the extreme. RDMA, once the preserve of HPC, broadened into cloud and enterprise precisely because these three pressures became a common challenge across the datacenter rather than for one field alone.

C. Characteristics

RDMA's nature is summarized by three axes: ① Kernel Bypass, ② Zero-Copy, and ③ Transport Offload. The application posts commands directly to a hardware queue without going through the kernel (①), data flows from registered memory to peer memory without copying (②), and the adapter handles transport logic such as retransmission, ordering, and flow control in hardware (③). These three characteristics only take effect when interlocked; if even one reverts to software, RDMA's advantage is rapidly diluted.

Another essential characteristic is the existence of one-sided operations. Whereas traditional communication is two-sided, engaging the CPU on both the sending and receiving ends, RDMA WRITE/READ writes to or reads from peer memory directly without the receiving CPU being involved at all. This "silent access" makes RDMA more than merely a fast transport—it becomes a programming model close to distributed shared memory.

Finally, RDMA has the characteristic of practically presupposing lossless transport. In return for simplifying transport logic into hardware, it is weak at flexibly absorbing arbitrary packet drops the way traditional TCP does. When packets drop, the cost of hardware retransmission grows and performance plunges, so the underlying fabric must guarantee that packets are not dropped. This premise is the very starting point that produces the implementation differences of InfiniBand's credit flow control and RoCE's PFC/ECN; "using RDMA" entails the responsibility of "having to design and operate a lossless network."

2. Architecture and Operating Principles

RDMA's performance comes from a structure in which the application skips the kernel and talks directly to the adapter hardware. The structure diagram below shows a typical configuration where two hosts communicate via RDMA.

flowchart LR
  subgraph HOSTA["Host A"]
    APPA["Application(user space)"]
    MRA["Registered memory(Memory Region)"]
    RNICA["RDMA adapter(RNIC/HCA)"]
    APPA -->|"Verbs(control path)"| RNICA
    APPA --- MRA
    RNICA -.->|"DMA(data path)"| MRA
  end
  subgraph HOSTB["Host B"]
    RNICB["RDMA adapter(RNIC/HCA)"]
    MRB["Registered memory(Memory Region)"]
    APPB["Application(user space)"]
    RNICB -.->|"DMA(data path)"| MRB
    APPB --- MRB
  end
  RNICA <-->|"Lossless fabric(InfiniBand/Ethernet)"| RNICB

The core of the structure is the separation of the Control Path and the Data Path. The application goes through the kernel only when starting communication, to prepare resources (queues and memory); thereafter, actual data send/receive posts Work Requests directly to the adapter's hardware queues from user space, without touching the kernel at all. This "set up once, then bypass" structure fundamentally eliminates the per-packet system-call and interrupt costs.

The basic unit of RDMA communication is the Queue Pair (QP), a pair of a Send Queue (SQ) and a Receive Queue (RQ). The application posts work to send or receive as a Work Request (WR) on a queue, and the adapter processes it asynchronously and signals completion on a Completion Queue (CQ). The memory over which data flows is pinned to physical addresses in advance through Memory Registration, and a local key (lkey) and remote key (rkey) carrying access permissions are issued. A remote host can access the peer's memory only if it knows this rkey and virtual address, so the rkey is simultaneously a performance resource and a security control point.

Viewed as a data flow, the sending adapter reads data from a registered local buffer via DMA and sends it onto the fabric, while the receiving adapter writes the arriving data directly into the destination memory the peer designated, via DMA. Throughout this process the data does not pass through the kernel socket buffer, and the CPU only confirms completion (by polling or by event). As a result, the CPU cycles and memory bandwidth consumed by a single message shrink dramatically, yielding far higher effective throughput on the same hardware.

Queue Pairs are divided into transport types by the reliability and connectivity they provide. RC (Reliable Connection) connects two QPs 1:1 like TCP, with hardware responsible for ordering and retransmission, and it supports one-sided WRITE/READ and atomic operations, making it the most widely used. UC (Unreliable Connection) maintains a connection but omits reliability guarantees to reduce overhead, and UD (Unreliable Datagram) is a connectionless type in which one QP communicates with many, offering good scalability for large-scale inter-node communication and multicast. Transport-type choice is a trade-off of "reliability/features ↔ connection-state maintenance cost/scalability," and in AI clusters exceeding thousands of nodes, scalability-oriented designs such as UD or Dynamic Connection (DC) become important to avoid an explosion of per-connection state (QPs).

A point especially worth noting is that memory registration is a double-edged sword. Memory must be pinned to physical addresses for the adapter to DMA it safely, but registration itself is an expensive operation, and if repeated on the communication path it instead increases latency. That is why, in practice, designs that pre-register and reuse buffer pools, or that amortize registration cost with the On-Demand Paging (ODP) feature of modern adapters, are important. Thus RDMA's performance hinges greatly not only on hardware specs but also on memory-management design.

Comparison with the traditional TCP/IP path makes this structural difference vivid. In socket-based communication, a single message is copied from the application buffer to the kernel socket buffer, passes through the protocol stack, and is copied again into the NIC driver buffer before being sent. On the receiving side, the reverse copies and the kernel interrupts, softIRQs, and context switches repeat. In this process the CPU spends time not on computation but on data copying and protocol processing, and the CPU cache is polluted with communication data, indirectly dropping application performance as well. RDMA removes both costs—copying and kernel traversal—at once, so the larger the bandwidth and the smaller the messages—that is, the smaller the per-packet processing budget—the wider its relative edge grows. Understanding that "the reason RDMA is fast" is not that the link is fast but that it removed unnecessary work along the path the data travels is the starting point of design.

3. Transport Operations (Verbs) and Processing Flow

The operations RDMA provides divide broadly into two-sided SEND/RECV and one-sided WRITE/READ/ATOMIC. The detailed process diagram below depicts a typical flow in which a one-sided RDMA WRITE takes place.

flowchart TD
  A["Sending app: obtain target address and rkey"] --> B["Post WR(Post Send to SQ)"]
  B --> C["Local RNIC: DMA read from registered buffer"]
  C --> D["Fabric transmission(ordering and retransmission by HW)"]
  D --> E["Remote RNIC: DMA write to destination memory"]
  E --> F{"Receiving CPU involved?"}
  F -->|"None(one-sided)"| G["Receiving app completes unaware"]
  C --> H["Completion posted to Completion Queue(CQ)"]
  H --> I["Sending app: confirm completion by polling/event"]

SEND/RECV, the two-sided channel semantics, resembles TCP's message exchange. The receiving side must post a receive buffer to the RQ in advance, and the sender's SEND is delivered to that buffer. Because both applications are involved, it suits control and signaling exchange, but it carries the slight burden of pre-posting receive buffers and processing completions on both sides. By contrast, RDMA WRITE/READ, the one-sided memory semantics, lets the sender manipulate memory directly without waking the receiving CPU at all, as long as it knows the peer's virtual address and rkey. Thanks to this asymmetry, one-way communication in which one side "pushes data in (WRITE)" or "pulls it out (READ)" becomes possible, delivering powerful performance in distributed key-value stores, shared logs, and RPC acceleration.

The third category, atomic operations (Compare-and-Swap and Fetch-and-Add), performs read-modify-write on remote memory atomically. This provides synchronization primitives in a single network round trip for distributed locks, counters, and consensus implementations, guaranteeing consistency with far fewer round trips than traditional message-based designs. For example, updating a version number with a remote Compare-and-Swap in a distributed key-value store implements optimistic concurrency without waking the server CPU, bypassing the server-side processing bottleneck.

The practical implication of one-sided operations is the "decentralization" of the server CPU. In traditional RPC every request is processed through the server CPU, making the server a hotspot; but using RDMA READ/WRITE, clients query and update server memory directly, dramatically reducing server CPU load. However, this entails the burden of redesigning data structures into "forms safely and directly accessible from remote," so the exchange between performance gain and design complexity must be weighed carefully. In short, one-sided operations are not a "free accelerator" but a design decision that reshapes even the application's data model.

The choice between one-sided and two-sided operations hinges on whether there is prior knowledge of the communication pattern. In producer-consumer or shared-buffer patterns where the sender can know in advance "where in the peer's memory to write," one-sided WRITE sidesteps the receiving CPU entirely and delivers the best efficiency. Conversely, in request-response patterns where the receiving side must actively process, two-sided SEND/RECV is natural. In practice the two are often mixed—exchanging control and signaling by SEND/RECV while moving the bulk data body by RDMA WRITE/READ. Thus operation choice is a design problem of tuning to the communication pattern between performance and programming simplicity.

The common principle running through the entire processing flow is the asynchronous, completion-driven model. After posting work, the application does not block but does other things, then reaps results by polling the completion queue or receiving event notifications. This asynchrony overlaps many communications to saturate the fabric, but the trade-off between latency and CPU utilization diverges by completion-confirmation method (polling vs. event). Polling gives the lowest latency but burns the CPU (busy-wait), while events save the CPU but add interrupt latency. Which to choose is a design decision based on the workload's latency sensitivity.

4. Comparison of Implementations — InfiniBand · RoCE · iWARP

RDMA is implemented over three transport layers, and each approach has clear trade-offs in performance, compatibility, and operational difficulty. The fundamental reason for this difference lies in the choice of "at which layer and how to guarantee the lossless, ordered transport that RDMA requires."

Category InfiniBand RoCE v2 iWARP
Basis Dedicated IB fabric Ethernet(UDP/IP, port 4791) Ethernet(TCP/IP)
Routing IB subnet L3 routable L3 routable
Lossless means Link credit flow control(built-in) PFC+ECN(DCQCN) required TCP handles loss
Latency/performance Lowest/highest Very high Medium
Operational difficulty Dedicated gear/management Ethernet tuning required Relatively simple
Typical use HPC/AI supercomputing Cloud/AI datacenters General enterprise

InfiniBand builds credit-based flow control into the link layer to provide, by default, a lossless fabric that does not drop packets, and it is composed of dedicated switches, adapters, and a subnet manager. It is best in latency and throughput but requires a dedicated ecosystem separate from Ethernet, imposing heavy cost and operational-expertise burdens. Hence it dominates in top-tier HPC/AI supercomputing where communication cost governs performance.

RoCE is the approach that moved this RDMA onto Ethernet; v1 was an L2 Ethernet frame (non-routable), but RoCEv2, with UDP/IP encapsulation (destination port 4791), became L3-routable and the de facto standard for large-scale datacenters. The crux is reproducing, on lossy Ethernet, the losslessness that RDMA presupposes. To this end, PFC (Priority Flow Control) prevents packet drops for a specific traffic class, and ECN-based congestion control (DCQCN) regulates the send rate. However, if PFC is poorly designed it causes congestion spreading in which congestion propagates upstream, head-of-line blocking, and in severe cases even deadlock, so RoCE's success or failure depends greatly on fabric-tuning capability. The key practical implication is that "RoCE is not an easy InfiniBand but a choice that shifted operational difficulty onto network engineering."

iWARP places RDMA atop standard TCP/IP, delegating the lossless guarantee to TCP's retransmission and congestion control. Thanks to this it works even on ordinary routed networks and WANs without special lossless-Ethernet configuration, lowering the adoption barrier, but the complexity of processing the TCP stack in hardware puts it behind RoCE and InfiniBand in latency and performance. The choice among the three ultimately rests on the triangular trade-off of "top performance (InfiniBand) ↔ Ethernet integration and large-scale expansion (RoCE) ↔ general network compatibility (iWARP)," decided to fit the workload and the organization's capability.

The reason the relative standing of the three has been reshaped over time lies in "changes in the integration and scalability the datacenter demands." In the past, top-performing InfiniBand dominated HPC, but as cloud and AI datacenters were built atop the already-vast Ethernet ecosystem, the demand to "reuse existing Ethernet operations, staff, and gear while gaining RDMA performance" grew, and at that juncture RoCEv2 surged. That is, the tension between the technical optimum (InfiniBand) and ecosystem fit (Ethernet) produced the compromise called RoCE, and the Ultra Ethernet discussed later can be seen as the next-stage evolution that aims to mend this compromise's weaknesses (PFC dependence and congestion-control limits) at the standards level.

5. Deep Dive: AI/HPC Application and Recent Trends

RDMA technology is evolving rapidly in step with the spread of AI infrastructure. First is GPUDirect RDMA and collective-communication acceleration. In large-scale LLM training, thousands of GPUs exchange gradients every step, and GPUDirect RDMA has the NIC read and write GPU memory directly without going through host memory, eliminating copy latency. Add to this collective libraries like NCCL that optimize All-Reduce and All-Gather over RDMA, and In-Network Computing (e.g., SHARP) in which switches perform summation inside the network, and the communication volume itself is reduced to minimize GPU idle time. In practice, improving communication efficiency in large training clusters has a direct impact on total training time and cost.

Second is the combination with storage and memory disaggregation. Implementing NVMe-oF (NVMe over Fabrics) over RDMA transport provides remote NVMe SSDs at latency close to a local disk, enabling composable infrastructure that scales compute and storage independently. Furthermore, CXL-based memory pooling and RDMA complement each other, strengthening the trend of treating the entire datacenter as one giant resource pool. Distributed file systems (e.g., parallel storage) and in-memory DBs also adopt RDMA as their backend transport to lower latency into the few-microsecond range.

Third is the next-generation Ethernet standardization aimed at the AI era. To overcome existing RoCE's PFC dependence and congestion-control limits, the industry is standardizing a new transport layer optimized for AI/HPC workloads, centered on the Ultra Ethernet Consortium (UEC). The core is packet spraying and multipath that scatter flows once concentrated on a single path across multiple paths, delivery less sensitive to ordering, and more sophisticated congestion control to reduce the tail latency of large-scale collective communication. AI-tuned Ethernet platforms like NVIDIA's Spectrum-X are commercial implementations of the same problem awareness. This shows an industry-wide pivot to "preserve RDMA's benefits while redesigning the transport layer to fit the communication patterns of giant AI clusters."

Finance and telecom are also strong application areas for RDMA. In ultra-low-latency trading (HFT) and market-data distribution, microsecond-level latency directly drives revenue, and RDMA provides deterministic low latency with suppressed jitter, creating a competitive edge. Also, parallel file systems like Lustre and GPFS and distributed storage like Ceph adopt RDMA transport as their backend to lower metadata and data-path latency, securing aggregate bandwidth in HPC environments where hundreds of nodes pour out I/O simultaneously. Thus RDMA is not a niche technology for a particular industry but has broadened into every high-performance field where latency and communication overhead are the bottleneck.

In concrete numbers, whereas the TCP/IP path shows tens of microseconds of latency and high CPU utilization, RDMA lowers inter-node latency into the 1–2 microsecond range and holds CPU utilization spent on communication to single-digit percent. However, such numbers vary widely by fabric, message size, and tuning, so the principle at adoption is to validate with a hands-on measurement (PoC) on one's own workload. RoCE in particular changes greatly in performance with a single PFC/ECN parameter, so spec comparison without a benchmark is dangerous.

To sketch a practical application, consider a cluster training a large language model on, say, 1,024 GPUs. Every training step triggers an All-Reduce in which all GPUs exchange gradients, and processing this communication via TCP/IP and host memory lengthens GPU idle time spent waiting on communication, dropping the utilization of the expensive accelerators. If GPUDirect RDMA has the NIC exchange GPU memory directly, NCCL optimizes communication over a RoCEv2 lossless fabric, and switches perform partial summation on behalf of nodes, the communication span shortens, and the same GPU investment can finish training in less time. Because a 1% improvement in communication efficiency translates, across thousands of GPUs and weeks of training, into substantial savings in power, time, and cost, fabric design becomes a key variable in AI-infrastructure ROI.

6. Considerations and Implications

RDMA/RoCE is a key technology for solving the communication bottleneck of the AI/HPC/high-performance-storage era, but understanding it only as "a fast network card" makes adoption easy to fail. From a professional engineer's perspective, the following four axes must be designed in an integrated way.

  • Application strategy (judging workload fit): RDMA's benefit is proportional to "the degree to which communication governs overall performance." Workloads where latency and CPU communication overhead are the bottleneck—synchronization-bound HPC/MPI, large-scale AI distributed training, distributed storage/in-memory DBs—are the primary targets, while in general web and batch workloads with low communication share the return on investment is limited. Before adoption, one must analyze one's own traffic's message size, frequency, and synchronization pattern to gauge the cost-benefit.

  • Trade-off (performance vs. operational complexity): In exchange for integrating into the Ethernet ecosystem, RoCE demands high-difficulty network engineering—PFC/ECN/DCQCN tuning, lossless traffic-class design, and prevention of congestion spreading and deadlock. InfiniBand has high performance and operational stability but requires dedicated gear and separate operational expertise. One must decide "where to place the complexity in exchange for performance" in light of the organization's capability.

  • Security and isolation redesign: Because RDMA bypasses the kernel and accesses remote memory directly, the traditional host firewall and packet-inspection model does not apply as-is. rkey/QP access-permission management, minimizing the scope of memory registration, inter-tenant fabric isolation (VLAN/partition), and transport encryption must be reflected early in design, and in multi-tenant clouds the trust boundary of remote memory access must be made especially clear.

  • Operational observability and fault response: RDMA is hard to inspect with traditional kernel/socket-layer diagnostic tools (packet capture, socket statistics), making it tricky to pinpoint causes of performance degradation (memory-registration bottlenecks, PFC storms, congestion on a particular QP). Without building an observability system that collects adapter counters, congestion-notification statistics, and fabric telemetry in advance, one cannot trace intermittent performance degradation in large clusters and ends up leaving expensive accelerators idle. An RDMA-specific monitoring pipeline must be designed at the same time as adoption.

  • Standardization/lock-in response and outlook: Currently RoCE tuning and congestion control have large per-vendor implementation variance and low portability. Adopting open standards (Verbs, UEC) and abstraction libraries (libibverbs, UCX, etc.) and designing so logic is not directly coupled to specific hardware can lower lock-in risk. Going forward, RDMA is expected to combine with Ultra Ethernet, CXL, GPUDirect, and in-network computing to establish itself as the default communication fabric of the AI datacenter, so an integration roadmap with the related technologies must be drawn together.

In sum, RDMA/RoCE is an architectural shift that "frees the CPU from carrying data and extends memory across the network." Rather than individual equipment specs, the professional engineer must read in an integrated way the design principles of kernel bypass, zero-copy, and transport offload, and the trade-off of where and how to guarantee lossless transport, and cultivate the discernment to judge the fabric approach and the level of tuning based on the nature of one's own workload's communication bottleneck.

References

In one line: RDMA is a communication technology that, through kernel bypass, zero-copy, and transport offload, accesses remote memory directly without CPU involvement to achieve ultra-low latency and high bandwidth; and RoCE implements it on standard Ethernet (RoCEv2/UDP/IP) with PFC/ECN-based losslessness, establishing itself as a core fabric of AI/HPC datacenters.