InfiniBand

InfiniBand's foundational capability is kernel-bypass communication. RDMA allows a network adapter to read or write data directly from the memory of one node to another without involving the operating system or CPU on either side.

• Zero-Copy: Data moves straight from GPU memory to the wire, eliminating intermediate buffers.
• Sub-Microsecond Latency: Typical end-to-end latency is under 1 microsecond for small messages.
• CPU Offload: Frees host processors entirely for computation, not data movement.

This is critical for GPUDirect RDMA, where NVIDIA GPUs can directly exchange data over InfiniBand without staging through system memory.

InfiniBand switches, particularly those based on the SHARP (Scalable Hierarchical Aggregation and Reduction Protocol) technology, can perform mathematical operations on data as it traverses the network.

• All-Reduce Acceleration: The switch fabric itself computes summation, minimum, or maximum operations during collective communications.
• Bandwidth Multiplication: Reduces the volume of data that must reach each endpoint, effectively multiplying effective network bandwidth.
• Deterministic Latency: Offloading collectives to the switch eliminates the multi-step, software-coordinated process of traditional MPI all-reduce.

This is a cornerstone of scaling NCCL collective operations in large language model training.

Unlike Ethernet's pause-frame or reactive congestion mechanisms, InfiniBand uses a link-level, credit-based flow control system to prevent packet loss entirely.

• Lossless Fabric: A sender only transmits when it has received buffer credits from the receiver, guaranteeing no buffer overruns.
• No Retransmission Storms: Eliminates the catastrophic performance collapse seen in lossy networks under congestion.
• Predictable Performance: Essential for the synchronized, barrier-bound nature of distributed deep learning training.

This lossless characteristic is why InfiniBand fabrics do not suffer from TCP incast pathologies common in Ethernet-based AI clusters.

InfiniBand employs sophisticated, per-packet adaptive routing algorithms that dynamically distribute traffic across all available paths in the fabric.

• Congestion Avoidance: Packets are sprayed across multiple routes based on real-time link load, avoiding hot spots.
• Full Bisection Bandwidth: Ensures that any node can communicate with any other node at full line rate without contention.
• Packet-Level Granularity: Unlike flow-based ECMP in Ethernet, InfiniBand can reorder packets at the destination, enabling finer-grained load balancing.

This is critical in spine-leaf architectures where multiple parallel paths exist between any two GPUs.

The Subnet Manager (SM) is a centralized entity that continuously discovers, configures, and monitors the entire InfiniBand fabric.

• Plug-and-Play Topology: New nodes are automatically detected, assigned addresses, and integrated into the routing tables.
• Fault Recovery: Upon a link or switch failure, the SM instantly recalculates routes to bypass the fault in milliseconds.
• QoS Enforcement: The SM configures Virtual Lanes (VLs) to provide strict traffic prioritization, isolating management, storage, and compute traffic.

This centralized intelligence eliminates the manual configuration and convergence delays typical of distributed routing protocols.

The foundational networking primitive that gives InfiniBand its performance advantage. RDMA allows a network adapter to write data directly into application memory on a remote machine, completely bypassing the operating system kernel and CPU on both sides.

• Eliminates context switches and data copies
• Achieves sub-microsecond latencies
• Critical for distributed training's all-reduce operations

Without RDMA, InfiniBand would lose its defining low-latency characteristic.

The physical network architecture that maximizes InfiniBand's performance. In a spine-leaf design, every leaf switch connects to every spine switch, creating a non-blocking, full-bisection bandwidth fabric.

• Predictable latency: exactly 2 hops between any two nodes
• Scales horizontally by adding spine switches
• Eliminates oversubscription common in traditional 3-tier architectures

This topology is the physical manifestation of InfiniBand's logical design philosophy: every node should reach every other node at line rate.

A congestion control algorithm critical for lossless RDMA networks. DCQCN combines Explicit Congestion Notification (ECN) marking with Priority Flow Control (PFC) to prevent packet loss in InfiniBand and RoCE fabrics.

• Reacts to congestion before buffers overflow
• Maintains zero packet loss required for RDMA correctness
• Prevents congestion spreading and victim flow problems

Without effective congestion control, a single hot-spot can degrade the entire fabric's throughput.