NVLink

NVLink creates a mesh topology where each GPU has a direct, dedicated link to every other GPU in the node. This eliminates the need to route traffic through a central switch or the CPU, dramatically reducing latency. Unlike PCIe, which forces all devices to share a single bus, NVLink provides point-to-point connections with aggregate bandwidth that scales linearly with the number of links.

• Topology: Fully connected mesh within a node
• Protocol: Proprietary NVIDIA signaling over physical lanes
• Benefit: Non-blocking, simultaneous transfers between all GPU pairs

For systems with more than two GPUs, direct pairwise wiring becomes impractical. NVSwitch is a dedicated switch chip that sits between all GPUs in a node, providing full crossbar connectivity. Each GPU connects to the NVSwitch, which routes data at line rate to any destination GPU. This enables all-to-all communication without the bandwidth degradation of daisy-chaining.

• Function: Non-blocking crossbar switch for GPU traffic
• Deployment: Found in DGX systems and HGX baseboards
• Result: Every GPU sees full bandwidth to every other GPU simultaneously

NVLink enables NVIDIA's Unified Memory architecture to span multiple GPUs. Each GPU can directly load and store to the memory of any other GPU in the NVLink domain without explicit programmer-managed copies. The page migration engine automatically moves data to the GPU that accesses it most frequently, creating a single virtual address space across all GPU memory pools.

• Mechanism: Hardware page faulting and on-demand migration
• Programming Model: cudaMallocManaged() with system-wide atomic operations
• Advantage: Simplifies code for models that exceed single-GPU memory capacity

NVLink-C2C extends the NVLink protocol to connect NVIDIA GPUs directly to other silicon, such as Grace CPUs or custom accelerators, at the die or package level. This provides cache-coherent memory sharing between CPU and GPU with bandwidth up to 900 GB/s, far exceeding what is possible over PCIe or even socket-to-socket CPU interconnects.

• Use Case: Grace Hopper Superchip — connects Grace ARM CPU to Hopper GPU
• Coherency: Full bidirectional cache coherence between CPU and GPU caches
• Bandwidth: Up to 900 GB/s total bandwidth

NVLink Network (formerly NVSwitch Fabric) extends NVLink beyond a single node to connect up to 256 GPUs across multiple nodes in a rack or cluster. It uses NVLink Switches interconnected with copper or optical cables to create a flat, high-radix network. This allows GPUs in different physical servers to communicate with the same direct-load/store semantics as within a node.

• Scale: Up to 256 GPUs in a single NVLink domain
• Physical Layer: Copper for intra-rack, optics for inter-rack
• Protocol: Same NVLink protocol, extended over physical transport

NVLink and NVSwitch support SHARP (Scalable Hierarchical Aggregation and Reduction Protocol) , which performs collective operations like all-reduce directly inside the network switch silicon. Instead of sending all gradient data to a GPU for summation, the switch aggregates data in-flight as it passes through, reducing the result and sending only the final value back. This dramatically cuts all-reduce latency and frees GPU compute for training.

• Operation: In-switch reduction for FP32, FP16, BF16, and FP8
• Impact: Up to 2x improvement in all-reduce throughput
• Relevance: Critical for large-scale distributed training with synchronous SGD

The NVIDIA Collective Communications Library (NCCL) is the software layer that leverages NVLink's physical bandwidth. It provides optimized routines for all-reduce, all-gather, and broadcast operations. During distributed training, NCCL automatically detects the NVLink topology and uses it for intra-node GPU communication, falling back to InfiniBand or RoCE for inter-node transfers. Without NCCL, raw NVLink hardware is inaccessible to most frameworks.

An NVSwitch is a physical switch chip that connects multiple GPUs in a full crossbar topology, eliminating the need for pairwise NVLink connections. In a DGX H100 system, four NVSwitches connect all eight GPUs, allowing any GPU to communicate with any other at full 900 GB/s bidirectional bandwidth. This non-blocking architecture is critical for large tensor parallelism where all GPUs must share model weights simultaneously.

While NVLink handles intra-node GPU communication, InfiniBand (or RoCE) is the dominant inter-node fabric. NVIDIA's GPUDirect RDMA allows data to travel from a GPU's memory on one server, through an InfiniBand NIC, directly to a GPU on another server—bypassing the CPU entirely. NVLink forms the 'scale-up' backbone, while InfiniBand provides the 'scale-out' fabric for clusters with thousands of nodes.

CUDA is the parallel computing platform that makes NVLink useful. CUDA's Unified Memory feature, combined with NVLink's high bandwidth, allows a GPU to transparently access memory on another GPU in the same node. This creates a large, logically unified memory pool. Developers can write code as if all GPUs share a single memory space, with the CUDA runtime and NVLink hardware handling page migration automatically.

NVLink connects the High Bandwidth Memory (HBM3e) stacks of different GPUs. HBM3e provides 4.8 TB/s of internal memory bandwidth per GPU, and NVLink extends this high-speed domain across multiple GPUs. The combination of HBM3e's vertical die stacking and NVLink's horizontal die-to-die connectivity creates a unified, high-bandwidth memory fabric that keeps massive models like GPT-4 resident entirely in GPU memory during training.

PCI Express (PCIe) is the standard bus for connecting GPUs to CPUs, but it is an order of magnitude slower than NVLink for GPU-to-GPU communication. PCIe Gen5 x16 provides 64 GB/s bidirectional bandwidth, compared to NVLink 4.0's 900 GB/s for a single H100. Systems without NVLink force GPU-to-GPU data to traverse the CPU and system memory, creating a severe bottleneck for tensor parallelism and large model training.