Zero-Copy I/O and Kernel Bypass
1. Overview
Zero-Copy is an I/O optimization technique that, when transferring data from one storage medium to another—especially between kernel space and user space—eliminates or minimizes the redundant copies in which the CPU directly duplicates bytes, so that data flows from source to destination without CPU involvement. Rather than "removing copies," it means "removing CPU-driven memory-to-memory copies"; the DMA transfers of disk and network devices remain in place.
The background for why zero-copy became necessary lies in the asymmetric advancement of hardware. Networks grew from 1GbE to 100GbE and 400GbE, and storage moved from SATA HDDs to NVMe SSDs—tens to hundreds of times more bandwidth—but memory copy speed (memory bandwidth) and CPU clocks did not speed up proportionally. As a result, "how many times the CPU re-duplicates the data it read" became the throughput bottleneck rather than "where the data is read from and sent to." Saturating a 100GbE link requires transferring about 12.5GB per second; if the same data is copied twice by the CPU as in the traditional path, the memory bus bears several times that traffic, and a single CPU core saturates just doing copies.
The term zero-copy is somewhat misleading, because the DMA transfers by which disks or network cards move in and out of memory are physically indispensable and are therefore not targets for removal. What zero-copy seeks to eliminate is strictly the CPU copy that arises solely because address spaces are separated within the same host. Thus the phrase "zero copies" must be read as "zero redundant CPU copies" to be accurate, and making this distinction clear lets one understand without confusion what each technique actually removes.
Zero-copy targets three costs. The first is wasted CPU cycles, since copying is the least productive work a CPU does. The second is memory bandwidth consumption, since a single copy consumes bandwidth twice over with a read plus a write. The third is CPU cache pollution, since copying large volumes of data evicts the cache lines the application logic actually needs, degrading overall performance. On top of this comes the cost of user/kernel mode switches (context switches). Zero-copy aims to reduce all four together, and it has become a core principle underpinning the performance of high-volume transfer systems such as Apache Kafka, Nginx, and Netflix Open Connect.
2. The Cost Structure of the Traditional I/O Path
To understand the value of zero-copy, one must first see why the traditional path is expensive. Consider the simplest code in which a server reads a file from disk and sends it over a socket (read() then write()): as the figure below shows, the data is copied four times and the user/kernel mode switches four times.
flowchart TB
subgraph US["User Space"]
APP["Application process"]
UB["User buffer"]
end
subgraph KS["Kernel Space"]
PC["Page cache(read buffer)"]
SB["Socket buffer"]
end
DISK[("Disk")]
NIC["NIC(network card)"]
DISK -->|"①DMA copy"| PC
PC -->|"②CPU copy"| UB
UB -->|"③CPU copy"| SB
SB -->|"④DMA copy"| NIC
APP -.->|"read() call"| PC
APP -.->|"write() call"| SB
The first copy (①) is the disk controller's DMA (Direct Memory Access) moving the file content into the kernel's page cache (read buffer), with no CPU involvement. The second copy (②) is the crux of the problem: as the read() system call returns, the data in the kernel buffer is copied directly by the CPU into a user-space buffer. Even if the application will send the data without processing it at all, this copy is forced merely because the address spaces are separate. The third copy (③) is write() again performing a CPU copy of the data in the user buffer into the kernel's socket buffer, and only at the last step (④) does the NIC's DMA send the socket buffer out to the network.
In the end, data that need only be transferred once is copied four times in total (2 DMA, 2 CPU), of which the CPU-performed ② and ③ are pure waste. Mode switches also occur four times in total—two in read() (entry and return) and two in write(). The cost of a mode switch itself is smaller than that of a copy, but each switch perturbs the TLB, branch predictor, and cache state, accumulating indirect costs.
The severity of this waste grows linearly as the data gets larger. For example, transferring a 1GB file causes, from the two CPU copies alone, 2GB read and 2GB written, adding 4GB of extra traffic to the memory bus, and the CPU core cannot handle other requests until that copying finishes. On web and streaming servers with thousands of concurrent connections, this copy cost accumulates and creates the classic bottleneck where throughput hits its limit due to CPU saturation even though the network and disk have headroom. Zero-copy is precisely the technique that removes this "invisible CPU and memory tax." The table below summarizes copy and switch counts for comparison with the techniques explained later. The numbers are an aid to comparison; "why they decrease" must be understood through the principles in the main text.
| Method | Total copies | CPU copies | Mode switches | Note |
|---|---|---|---|---|
| read + write | 4 | 2 | 4 | Traditional path |
| mmap + write | 3 | 1 | 4 | Removes the user-buffer copy |
| sendfile (basic) | 3 | 1 | 2 | Completed inside the kernel |
| sendfile + SG-DMA | 2 | 0 | 2 | CPU copy fully removed |
| splice | 2 | 0 | 2 | Pipe-based reference passing |
The key to read from this table is that "the direction in which copy counts decrease" and "the direction in which CPU copies converge to zero" are the two axes of performance improvement. Dropping the traditional path's two CPU copies to one, and then to zero, is the common goal of zero-copy techniques. Reducing the four mode switches to two is also a secondary but important improvement, made possible by a design like sendfile that delegates the entire process to the kernel in a single system call.
3. Zero-Copy Implementation Techniques
Zero-copy is not a single technology but a set of techniques that remove copies and mode switches step by step. Below we examine the representative means the operating system provides, principle first, in order of reducing copies one stage at a time. Each technique's differences become clear when compared from the viewpoint of "what it removes, and what constraints and costs it bears in exchange."
A. mmap + write
mmap() maps a file directly into the process's virtual address space (memory-map) so that the kernel's page cache and user space share the same physical pages. The CPU copy (②) of "kernel buffer → user buffer" therefore disappears. Simply by reading the mapped address, the application effectively sees the page cache's data as is. However, when subsequently sending to the socket with write(), the CPU copy (③) of "mapped region → socket buffer" still remains, resulting in three copies total (2 DMA + 1 CPU).
The pitfall of the mmap approach is the complexity that arises as the price of reduced copying. If, during transfer, another process truncates or modifies the same file, accessing the mapped region can raise a SIGBUS signal and terminate the process abnormally. In addition, mapping and unmapping (mmap/munmap) incur the cost of updating page tables and invalidating the TLB, so frequently mapping small files erodes the savings from reduced copying. Thus mmap zero-copy must be used carefully in situations where the file lifecycle can be controlled, and it is advantageous for repeatedly transferring large files rather than for small, frequent I/O.
B. sendfile
sendfile() is a system call that transfers data directly inside the kernel from a file descriptor to a socket descriptor. Since the data does not rise into user space, copies ② and ③ appear to both be removed, but in the basic implementation one CPU copy that moves the page cache data to the socket buffer remains, giving three copies total. Yet the mode switches drop to two because it completes in a single system call, and above all, because the data does not pass through user space, cache pollution and address-space switching disappear. Nginx's sendfile on; directive uses exactly this system call.
The constraint of sendfile is that it can be used only when the data is passed through unchanged. If the content must be processed before transfer, or only a portion sent depending on conditions, the data must be pulled up into user space, which negates sendfile's benefit. Also, since the source must be a file that can be loaded into the page cache, it cannot be used for data arriving from a pipe or another socket. What emerged to overcome these limits is the splice family below.
C. sendfile + scatter-gather DMA (true zero-copy)
To remove even the last CPU copy, the NIC must have scatter-gather DMA, that is, the ability to gather multiple scattered memory fragments into a single packet for transmission. Linux kernel 2.4 improved the socket buffer descriptor structure so that the socket buffer holds, instead of the actual data, only the location (address) and length information (descriptor) of the page cache, letting the NIC collect the data directly from the page cache via DMA. With this, CPU copies become entirely zero, and the only remaining copies are the two DMA transfers (disk → page cache, page cache → NIC). This is what is commonly called "true zero-copy."
flowchart LR
APP["Application(sendfile call)"]
subgraph KERNEL["Kernel Space"]
PC["Page cache"]
DESC["Socket buffer descriptor(location and length only)"]
end
DISK[("Disk")]
NIC["NIC(scatter-gather DMA)"]
DISK -->|"①DMA copy"| PC
PC -->|"②pass descriptor only(no data copy)"| DESC
DESC -->|"③SG-DMA gathers directly from page cache"| NIC
PC -.->|"data stays resident in page cache"| NIC
APP -->|"file fd → socket fd"| KERNEL
D. splice / vmsplice and the send-side MSG_ZEROCOPY
splice() moves references to data (page pointers) between two file descriptors via an in-kernel pipe buffer, enabling copy-free transfer across arbitrary fd combinations (file↔file, file↔socket, etc.). Because the pipe buffer holds only references to pages rather than the actual data, it escapes the constraint sendfile imposed that "the source must be a file," so zero-copy can be applied even to proxy patterns such as socket-to-socket relaying. On the send side, where an application wants to send data it generated itself without copying, the MSG_ZEROCOPY flag (the SO_ZEROCOPY socket option) is used. In this case the kernel pins the physical pages of the user buffer, DMAs those pages directly to the NIC, and asynchronously notifies the completion queue when the transfer finishes. There is no copy, but new overhead arises in the form of page pinning and notification handling, so the benefit is large only for high-volume transfers with large packets and may actually be a loss for small transfers. One must always keep in mind the trade-off that "another management cost moves in where the copy was removed."
4. Receive-Side Zero-Copy and Kernel Bypass: RDMA, DPDK, io_uring
The preceding techniques mainly address the send (disk → network) path, but on the receive path and in ultra-high-performance domains, kernel bypass—which circumvents the kernel network stack itself—is used. Comparing three representative approaches clarifies their differences and practical implications.
RDMA (Remote Direct Memory Access) is a technology in which one node's NIC performs reads/writes directly into the application memory of another node across the network, implemented by InfiniBand, RoCE (RDMA over Converged Ethernet), and iWARP. Since the kernel and CPU are out of the data path on both ends, "end-to-end zero-copy" holds. Beyond simply removing copies, RDMA is characterized by providing both one-sided operations that read and write the remote memory directly without the remote CPU's involvement, and kernel-bypassing transmission that posts commands directly to NIC queues from user space. As a result, latency drops to the microsecond range and CPU utilization converges to nearly zero. It is thanks to RDMA's zero-copy that HPC and AI training clusters can connect hundreds of GPUs with 400Gb/s-class InfiniBand and perform collective communication (all-reduce). However, since the NIC is responsible for connection state and flow control in hardware, it pays the cost of being hard to debug on failure and of reduced flexibility (ossification) as feature changes are tied to hardware.
DPDK (Data Plane Development Kit) maps the NIC directly into user space, fetches packets with a poll mode driver (PMD) instead of interrupts, and reduces TLB misses with huge pages to bypass the kernel stack entirely. It is the de facto standard in domains that must process tens of millions of packets per second, such as telecom equipment, firewalls, and L4 load balancers. In exchange, it dedicates one CPU core permanently to polling (busy-poll), sacrificing power and cores.
The reason DPDK uses polling instead of interrupts is also "overhead removal" in the same vein as zero-copy. When tens of millions of packets per second pour in, the cost of taking an interrupt per packet (context switching, cache disturbance) exceeds the processing cost, producing an interrupt storm. By devoting one core to continuously inspecting the NIC queues, DPDK removes this cost entirely and, combined with copy elimination, achieves packet rates impossible with a general-purpose kernel stack. That it gains even determinism of latency in exchange for sacrificing power and cores is why it is preferred in the telecom-equipment domain.
io_uring is Linux's asynchronous I/O interface, which dramatically reduces the number of system calls themselves through submission queue (SQ) and completion queue (CQ) ring buffers shared by user and kernel. Combined with pre-registered buffers, it can suppress both copies and mode switches, and recently a zero-copy receive (zcrx) feature was added, opening a path to receive TCP payloads directly into user memory. Whereas RDMA and DPDK discard the kernel stack entirely, it is practically important that io_uring zero-copy RX removes only the copy while using the kernel TCP stack as is. This is because one can gain the zero-copy benefit while retaining standard TCP compatibility and ease of debugging.
The three approaches share the trait of eliminating copies on the receive side, but what they give up in exchange differs. Organizing their characteristics in the table below clarifies the selection criteria.
| Approach | Kernel stack | Primary use | Main cost |
|---|---|---|---|
| RDMA | Bypass | HPC and AI training fabrics | Hardware dependence, debugging difficulty |
| DPDK | Bypass | Telecom equipment, firewalls, LB | Permanent core occupation, power |
| io_uring zcrx | Retain | General-purpose high-performance servers | Relatively new, kernel-version dependent |
The fundamental reason the three approaches differ lies in the choice of "how far to give up the kernel stack." RDMA and DPDK give up the kernel's generality, safety, and observability for performance, whereas io_uring tries to reduce only copies while preserving them. Thus it is a common design judgment that general-purpose server applications adopt the sendfile/io_uring family, while dedicated network appliances and AI training fabrics choose DPDK/RDMA.
This choice is also directly tied to the operating organization's capability. DPDK and RDMA require low-level tuning such as polling-core management, huge-page configuration, NIC firmware/driver version alignment, and congestion control (RoCE's PFC/ECN), demanding dedicated personnel. The sendfile/io_uring paths, by contrast, run on top of the standard kernel stack and can use existing operations and observability tools as is. In the end, the choice of zero-copy approach involves not only raw performance figures but also an organizational judgment of "how much complexity can be sustained," and this point is especially important in professional-engineer-level design decision-making.
5. Advanced Topics: Practical Applications and Latest Trends
Zero-copy has not remained theoretical but has actually governed the performance of large-scale services. We summarize representative cases and recent trends.
First, Apache Kafka, when a broker transfers log segments on disk to consumers, uses Java's FileChannel.transferTo() (internally sendfile) to stream messages directly from the page cache to the socket without pulling them up into user space. Combined with Kafka's characteristic that messages are consumed as is without processing, zero-copy achieves throughput of millions of messages per second even with little CPU. This is a textbook case demonstrating the principle that "the zero-copy benefit is maximized when the application does not touch the data."
A detail worth noting here is that Kafka reads and sends data not from disk but from the operating system's page cache. Recently produced messages still remain hot in the page cache, so when a consumer consumes them, they flow straight from cache → NIC without disk I/O. Zero-copy combined with the page cache yields "latency close to that of an in-memory queue despite being a disk-based queue," demonstrating that zero-copy exerts its power when designed together with the operating system's memory management rather than as a standalone technique.
Second, Netflix Open Connect CDN servers serve tens to hundreds of Gb/s of video from a single machine with FreeBSD's asynchronous sendfile. The problem was that TLS encryption pulls data up into user space and breaks zero-copy, which was solved with kTLS (kernel TLS). By performing encryption in the kernel (or as a NIC offload), zero-copy—in which data does not pass through user space—was maintained even for encrypted traffic. As a design that reconciles security and zero-copy by moving the encryption point down into the kernel when the two conflict, it has become the standard pattern for preserving zero-copy in modern secure transport.
Third, AI data centers and GPUDirect. In large-scale model training, GPUDirect RDMA and GPUDirect Storage, by which a NIC or NVMe SSD DMAs directly into GPU memory, are central; this extends the zero-copy concept of "data does not route through CPU memory" into the GPU domain. Because it does not pass through the CPU and system memory, it reduces GPU data starvation and eases bottlenecks in the training pipeline.
Fourth, the maturation of io_uring zero-copy networking. The io_uring-based receive zero-copy proposed under Meta's leadership uses header splitting and flow steering so that only the payload is received directly into user memory, and in a Kernel Recipes 2024 talk it advanced toward being efficient even at small transfer sizes and compatible with vanilla TCP. It draws attention as an attempt to narrow the long-standing tension between the performance of kernel bypass and the generality of the kernel stack.
The trend running through these four cases is that zero-copy, having started in the narrow domain of "send and file transfer," is expanding into the receive path, the encryption segment, GPU memory, and asynchronous I/O in general. In an era of AI and high-volume streaming where the cost of moving data dominates the cost of computation, zero-copy is being elevated from the problem of a particular system call into an architectural design principle of "letting data flow from source to consumer without passing through the CPU."
6. Considerations and Implications
From a professional engineer's perspective, when adopting and designing zero-copy, one must judge the following comprehensively.
Application strategy — "does it not touch the data" is the premise: The greatest benefit of zero-copy arises when the application relays data unchanged without transforming it. Pass-through workloads such as file servers, CDNs, message brokers, and proxies are the top candidates. Conversely, if data transformation such as compression, encryption, or filtering is needed, one must first examine whether the zero-copy segment can be preserved by moving that processing point down into the kernel (kTLS), NIC offload, or GPU.
Trade-offs — the cost that arises where the copy was removed: Zero-copy is not free. New management costs and complexity follow, such as page pinning, asynchronous completion notification, mmap's SIGBUS risk, DPDK's permanent core occupation, and RDMA's debugging difficulty and hardware dependence. Especially for small, frequent I/O, the management overhead can exceed the copy savings and actually be slower, so one must verify by benchmark that the transfer unit is sufficiently large before applying it.
Outlook — convergence of kernel bypass and generality: Whereas in the past it was an either/or of abandoning the kernel for extreme performance (DPDK/RDMA) or staying inside the kernel (sendfile), io_uring zero-copy RX, AF_XDP, and kTLS are converging toward "removing only copies while preserving the safety and observability of the kernel stack." When designing a new system, prioritizing a zero-copy path that coexists with the standard stack over full bypass with heavy hardware dependence is advantageous in terms of maintainability and portability.
Alignment with related technologies: Zero-copy is not a standalone technology and must be understood together with DMA, the page cache, virtual memory (mmap), NUMA, huge pages, and the transport layer's TLS and TCP offload. In NUMA environments in particular, the zero-copy benefit materializes fully only when data, NIC, and CPU are on the same node, so interrupt affinity (IRQ affinity) and memory placement must be tuned together. On the observability side, because copies disappear, bottlenecks may not surface from traditional CPU profiling alone, so metrics such as memory bandwidth, DMA, and NIC queue depth must be monitored together.
Security and stability considerations: A structure in which the kernel and NIC directly access user buffers means an expanded attack surface traded for performance. Unless buffer modification during page pinning, DMA region boundary validation, and control over remote memory access permissions in RDMA are designed rigorously, data integrity and isolation can break. Performance optimization should be applied together with hardware protection mechanisms such as the IOMMU so as not to weaken the security model.
References
- IBM Developer, "Efficient data transfer through zero copy." https://developer.ibm.com/articles/j-zerocopy/
- Linux Journal, "Zero Copy I: User-Mode Perspective." https://www.linuxjournal.com/article/6345
- The Linux Kernel Documentation, "MSG_ZEROCOPY." https://www.kernel.org/doc/html/latest/networking/msg_zerocopy.html
- Kernel Recipes 2024, "Efficient zero-copy networking using io_uring." https://kernel-recipes.org/en/2024/schedule/efficient-zero-copy-networking-using-io_uring/
- NVIDIA, "GPUDirect RDMA." https://docs.nvidia.com/cuda/gpudirect-rdma/
In one line: Zero-copy is an I/O optimization that removes CPU copies between kernel and user space to save CPU, memory bandwidth, and cache; it runs from mmap, sendfile, SG-DMA, and splice through RDMA, DPDK, and io_uring kernel bypass, underpinning the high-volume transfer performance of Kafka, Netflix, and AI data centers, yet it is a trade-off-based technique that yields its greatest effect "when the data is not touched."