GPUDirect RDMA

The foundational networking technology that allows data to be transferred directly between the memory of two computers without involving the operating system, CPU, or cache of either host. Zero-copy operations bypass the kernel, dramatically reducing latency and freeing CPU cycles for computation. RDMA is implemented over InfiniBand, RoCE (RDMA over Converged Ethernet), and iWARP fabrics. Key characteristics include:

• Kernel bypass: Data moves directly from NIC to application memory
• One-sided operations: A host can read or write remote memory without the remote CPU's involvement
• Transport offload: The NIC handles reliable delivery, ordering, and congestion control in hardware

A high-bandwidth, low-latency switched fabric interconnect standard used extensively in HPC and AI supercomputing clusters. InfiniBand provides native hardware support for RDMA and achieves sub-microsecond latencies with throughput rates of 400 Gb/s (NDR) and 800 Gb/s (XDR) in modern deployments. Unlike Ethernet, InfiniBand implements credit-based flow control and adaptive routing in silicon, eliminating packet loss and congestion collapse. NVIDIA Mellanox (acquired by NVIDIA) is the dominant supplier, making InfiniBand the de facto fabric for large-scale GPU clusters like those used in GPT-class model training.

A CUDA feature enabling direct data transfers between two GPUs on the same PCIe bus or NVLink fabric without staging data through host system memory. GPUDirect P2P allows CUDA kernels to load/store directly from peer GPU memory, eliminating an intermediate copy. This is distinct from GPUDirect RDMA, which extends the direct path to a network card. Combined, they create an end-to-end GPU-to-GPU data highway across nodes:

• Intra-node: NVLink + NVSwitch provide 900 GB/s GPU-to-GPU bandwidth
• Inter-node: GPUDirect RDMA extends this path over InfiniBand/RoCE
• Unified address space: Enables single-node multi-GPU programming models

NVIDIA's proprietary high-speed interconnect technology that provides direct GPU-to-GPU communication at far greater bandwidth than PCIe. NVLink 4.0 delivers 900 GB/s of bidirectional bandwidth per GPU in the H100 architecture. NVSwitch connects multiple GPUs in a fully connected, non-blocking topology within a single node (up to 8 GPUs in DGX systems). This fabric is the intra-node complement to GPUDirect RDMA's inter-node capability. Together, they form a logically unified memory fabric where all GPUs in a cluster can access each other's memory with minimal software overhead.

A memory management feature that provides a single, unified address space for all CPU and GPU memory in a system. With UVA, the CUDA runtime can determine the physical location of a pointer (host memory, local GPU memory, or peer GPU memory) and transparently route memory accesses. GPUDirect RDMA relies on UVA to expose GPU memory as a flat, addressable region to the network adapter. The NIC's DMA engine uses these UVA pointers to read/write directly to GPU device memory without requiring the application to manage separate address spaces or perform manual copies.

An optimized library implementing multi-GPU and multi-node collective communication primitives such as AllReduce, AllGather, ReduceScatter, and Broadcast. NCCL automatically detects the available interconnects (NVLink, PCIe, InfiniBand, RoCE) and selects the most efficient transport, including GPUDirect RDMA when available. It uses ring and tree algorithms to minimize latency and maximize bandwidth utilization. NCCL is the communication backbone for virtually all distributed deep learning frameworks, including PyTorch Distributed, TensorFlow, and JAX, making GPUDirect RDMA transparent to the ML practitioner.