RDMA

RDMA enables a network adapter to transfer data directly to and from application memory without involving the host operating system kernel. In a traditional TCP/IP stack, data must traverse the kernel's network stack, incurring context switches and memory copies. With kernel bypass, an RDMA-enabled NIC reads or writes data directly from a pinned application buffer, reducing latency to single-digit microseconds and freeing CPU cycles for computation rather than I/O processing. This is critical for GPU clusters where every microsecond of communication delay stalls expensive accelerators.

Conventional network stacks perform multiple memory copies: from application buffer to kernel socket buffer, then to the NIC's transmit ring. RDMA eliminates these intermediate copies. The NIC's DMA engine reads data directly from the source application buffer and writes it directly into the destination application buffer across the network. This zero-copy mechanism dramatically reduces memory bandwidth consumption and CPU overhead, allowing a single server to saturate 200Gb/s or 400Gb/s links without becoming CPU-bound on data movement tasks.

RDMA supports RDMA Read and RDMA Write operations that are executed entirely by the initiator's NIC without waking the remote CPU. The initiator specifies a remote memory address, and the remote NIC services the request autonomously using previously exchanged memory registration keys. This one-sided semantic is transformative for distributed computing patterns like shared-memory emulation and parameter server architectures in deep learning, where a training node can push gradients directly into a remote buffer without interrupting the parameter server's CPU.

The entire transport protocol processing is offloaded to the RDMA network adapter hardware. Functions such as packet segmentation, reassembly, acknowledgment generation, and retransmission logic execute on the NIC's ASIC rather than in software. This hardware offload guarantees deterministic, line-rate performance regardless of host CPU load. For AI training workloads using NCCL collective communications, this ensures that all-reduce operations complete with predictable, low tail latency even when the host CPUs are saturated with data preprocessing tasks.

Before RDMA operations can occur, application memory regions must be registered with the RNIC. This registration pins the virtual memory pages to physical memory, preventing the OS from swapping them out, and provides the NIC with the virtual-to-physical address mappings. The NIC caches these translations in its internal Memory Translation Table, enabling it to perform address translation at line rate without host intervention. While registration introduces a one-time setup cost, it is essential for the safety and performance of direct memory access.

RDMA fabrics offer multiple transport modes tailored to different workload requirements:

• Reliable Connection (RC): Provides in-order, lossless delivery with hardware-based retransmission. Ideal for storage protocols and MPI collectives.
• Unreliable Datagram (UD): Offers connectionless messaging without guaranteed delivery. Used for control plane operations and scalable one-to-many communication.
• Dynamically Connected (DC): Balances scalability and reliability by dynamically establishing connections, commonly used in large-scale AI clusters to avoid connection state explosion.

The dominant high-performance switched fabric interconnect for AI clusters, natively supporting RDMA. InfiniBand provides sub-microsecond latency and bandwidth up to 400 GB/s (NDR), making it the preferred choice for GPU-to-GPU communication in large-scale training.

• Uses a cut-through switching architecture to minimize latency
• Implements adaptive routing to avoid congestion hotspots
• Requires dedicated InfiniBand host channel adapters (HCAs) and switches
• Supports SHARP (Scalable Hierarchical Aggregation and Reduction Protocol) for in-network computing

A network protocol that enables RDMA over standard Ethernet infrastructure, eliminating the need for specialized InfiniBand hardware. RoCEv2 encapsulates RDMA packets in UDP/IP, making it routable across IP networks.

• RoCEv1: Layer 2 only, non-routable, limited to a single broadcast domain
• RoCEv2: Layer 3 routable, uses UDP destination port 4791
• Requires lossless Ethernet via Priority Flow Control (PFC) and Explicit Congestion Notification (ECN)
• Often deployed with DCQCN congestion control for stability at scale

An NVIDIA technology that enables direct data transfers between GPU memory and a network interface card (NIC) without staging through CPU host memory. This eliminates a costly memory copy, reducing latency and freeing CPU cycles.

• Creates a direct PCIe peer-to-peer path between GPU and NIC
• Requires compatible NVIDIA GPUs, Mellanox/ConnectX NICs, and CUDA drivers
• Critical for multi-node training where gradient synchronization must be as fast as possible
• Works with both InfiniBand and RoCE fabrics

The standard collective communication library for multi-GPU and multi-node deep learning. NCCL implements topology-aware all-reduce, all-gather, reduce-scatter, and broadcast primitives optimized to saturate RDMA interconnects.

• Automatically detects GPU topology (NVLink, PCIe, network) and selects optimal algorithms
• Uses ring and tree algorithms depending on message size and node count
• Integrates natively with PyTorch torch.distributed and TensorFlow tf.distribute
• Achieves near-line-rate bandwidth on InfiniBand and RoCE fabrics

A congestion control algorithm specifically designed for RDMA over Converged Ethernet (RoCEv2) networks. DCQCN combines Explicit Congestion Notification (ECN) marking with rate-based throttling to prevent packet loss in lossless fabrics.

• Reacts to congestion before buffers overflow, avoiding PFC storms
• Uses a reaction point (sender) and notification point (switch) architecture
• Critical for maintaining RDMA performance at scale in multi-tenant GPU clusters
• Prevents congestion spreading and head-of-line blocking across the fabric

A two-layer network topology where every leaf switch connects to every spine switch, providing predictable non-blocking east-west bandwidth. This is the standard physical design for GPU cluster fabrics using RDMA.

• Eliminates oversubscription: any leaf can reach any other leaf at full line rate
• Supports equal-cost multipathing (ECMP) for load balancing across spines
• Scales horizontally by adding spines for more bandwidth or leaves for more ports
• Typically deployed with BGP or OSPF for dynamic routing in the fabric