Zero-Copy Transfers

Zero-copy transfers are fundamentally enabled by Unified Virtual Memory (UVM) architectures, such as NVIDIA's CUDA UVM. UVM creates a single, contiguous virtual address space shared by the CPU and GPU. This allows both processors to access the same memory pointer, meaning the GPU can directly read from or write to host-allocated memory without the programmer orchestrating a manual copy via cudaMemcpy. The memory management unit (MMU) and GPU page fault handler manage the coherence and physical location of data transparently.

For traditional direct memory access (DMA) transfers—a precursor to full zero-copy—the host memory must be page-locked (pinned). Pinning prevents the operating system from paging this memory to disk, guaranteeing a stable physical address. This is essential for the GPU's DMA engine to perform high-bandwidth transfers. In a zero-copy context with UVM, while not always strictly required for functionality, using pinned memory for zero-copy buffers is a critical performance best practice. It avoids the overhead of page faults on the host side during GPU access and ensures optimal transfer speeds over the PCIe bus.

The primary performance benefit is the removal of the latency of an explicit cudaMemcpy operation. In a standard workflow:

• Stage 1: CPU writes data to host buffer.
• Stage 2: cudaMemcpy (blocking or async) transfers data to device buffer.
• Stage 3: GPU kernel processes device buffer.

With zero-copy:

• Stage 1: CPU writes data to a host-resident, GPU-accessible buffer.
• Stage 2: GPU kernel directly accesses the host buffer. The copy latency is eliminated, as data movement occurs implicitly via PCIe loads/stores as the GPU accesses the memory. This is most beneficial for kernels that make sparse, non-coalesced accesses to large data sets, where the cost of a full copy outweighs the cost of slower PCIe accesses.

A standard two-buffer model (host buffer + device buffer) doubles the memory footprint for a given data set. Zero-copy uses a single buffer resident in host memory that is accessible to both processors. This directly halves the memory requirement for that data on systems with unified memory, or at minimum reduces pressure on valuable GPU device memory. This is particularly advantageous when working with very large datasets that exceed GPU memory capacity, as data can be seamlessly accessed from host memory or even SSD via GPU Direct Storage without managing multiple copies.

Zero-copy does not make host memory as fast as GPU device memory. Access characteristics include:

• Higher Latency: GPU access to host memory over PCIe has significantly higher latency (hundreds of nanoseconds) than access to its own high-bandwidth memory (HBM).
• Lower Bandwidth: PCIe bandwidth (e.g., 64 GB/s for PCIe 5.0 x16) is lower than GPU memory bandwidth (e.g., > 1 TB/s for HBM2e).
• Non-Coalesced Penalty: The performance penalty for non-coalesced memory access patterns is more severe over PCIe.

Therefore, zero-copy is an optimization only when the compute-to-memory-access ratio of the kernel is low, or data is accessed very infrequently. It is not suitable for kernels that repeatedly read/write the same data.

Modern zero-copy implementations are tightly integrated with GPU page faulting and demand paging. When a GPU kernel tries to access a page in its virtual address space that resides in host memory, a GPU page fault occurs. The GPU's memory management unit (MMU) then:

1. Stalls the offending threads.
2. Migrates the required page from host memory to GPU memory (if possible and beneficial).
3. Resumes the threads.

This allows for transparent, on-demand migration of frequently accessed "hot" pages to faster GPU memory, while infrequently accessed "cold" pages remain in host memory. This adaptive behavior is key to the efficiency of zero-copy in unified memory systems.

A memory management architecture that creates a single, contiguous virtual address space shared between a CPU and GPU. This allows both processors to access the same memory pointers using a common page table, which is the foundational technology that enables zero-copy transfers. UVM simplifies programming by eliminating the need for explicit data copies, as the memory management unit (MMU) and GPU page fault handler automatically manage data migration between host and device memory on demand.

Host (CPU) memory that is prevented from being paged out to disk by the operating system. Pinning memory is a prerequisite for high-performance Direct Memory Access (DMA) transfers, which are used by both explicit copies and advanced zero-copy techniques. Pinned memory ensures the physical address of the data is stable, allowing the GPU's DMA engine to perform transfers without CPU intervention. It provides the highest bandwidth for host-to-device transfers but consumes scarce system resources that cannot be swapped.

A technology that enables a GPU to directly access data from storage devices (e.g., NVMe SSDs), bypassing the CPU and host memory entirely. While not a host-device zero-copy transfer, GDS embodies the same direct access philosophy at a larger scale. It reduces I/O latency and CPU overhead for data-intensive workloads (like loading massive model checkpoints or datasets) by establishing a direct data path from storage to GPU memory.

A capability that allows multiple GPUs within the same system to directly read from and write to each other's device memory over a high-speed interconnect like NVLink or PCIe. This is a form of zero-copy transfer between GPUs, bypassing host memory. P2P is critical for scaling multi-GPU training and inference workloads, as it enables fast model parallelism and efficient tensor communication without the latency and bandwidth bottleneck of the host.

The mechanisms that make zero-copy transfers practical in unified memory systems.

• Demand Paging: Data resides in host memory until the GPU actually needs it, at which point it is migrated on-demand.
• GPU Page Fault: The exception raised when a GPU kernel accesses a virtual address whose data is not in GPU memory. This fault triggers the automatic migration of the required memory page from host to device. Together, they enable efficient memory oversubscription, where the working set can exceed physical GPU RAM, with a performance penalty proportional to fault frequency.

The performance of zero-copy transfers depends heavily on how GPU threads access the host memory. Coalesced memory access—where consecutive threads read consecutive addresses—is crucial because accessing non-coalesced, unpinned host memory from GPU kernels can be extremely slow due to PCIe protocol overhead per transaction. Optimizing kernels for coalesced access is essential to realize the latency benefits of zero-copy, otherwise, explicit batched copying may be faster.