NVLink

NVLink creates a high-speed mesh network between processors, bypassing the traditional PCIe root complex. Key architectural elements include:

• Direct GPU-to-GPU Links: Each NVLink connection provides a dedicated, bidirectional lane between two GPUs.
• Multi-GPU Meshes: In systems like NVIDIA DGX, NVLink switches enable complex topologies (e.g., all-to-all connectivity) for up to 256 GPUs.
• CPU-GPU Coherence: With NVLink-C2C, the interconnect extends cache coherency between NVIDIA Grace CPUs and Hopper GPUs, creating a unified memory space. This architecture is fundamental for workloads like large language model training, where terabytes of parameters must be synchronized across hundreds of GPUs.

NVLink's primary advantage is its massive bandwidth, which scales with each generation to alleviate data transfer bottlenecks.

• Generational Scaling: NVLink 4.0 offers 900 GB/s of bidirectional bandwidth per GPU-GPU link, a 1.5x increase over NVLink 3.0 (600 GB/s).
• Comparison to PCIe: A single NVLink 4.0 connection provides over 14x the bandwidth of a PCIe 5.0 x16 lane (128 GB/s).
• Aggregate Bandwidth: An NVIDIA H100 GPU with six NVLinks has an aggregate 3.6 TB/s of GPU-to-GPU bandwidth. This bandwidth is critical for model parallelism, where layers of a neural network are split across GPUs, requiring constant, high-volume activation and gradient transfers.

NVLink enables a critical software feature: a single, unified virtual address space across multiple GPUs and CPUs.

• GPU Direct Access: Any processor in the NVLink fabric can directly access the memory of any other connected processor using standard load/store instructions.
• Simplified Programming: This eliminates the need for explicit cudaMemcpyPeer calls, allowing developers to treat distributed GPU memory as a large, shared pool.
• Larger Model Capacity: Combined with technologies like Unified Virtual Memory (UVM), it allows a model's parameters to be distributed across the aggregate memory of all linked GPUs, enabling training of models that exceed the capacity of a single device. This is the foundation for frameworks like NVIDIA's Collective Communications Library (NCCL) optimized for multi-GPU reduction operations.

For scaling beyond direct point-to-point connections, NVLink employs dedicated switching hardware.

• NVSwitch: A standalone ASIC that acts as a communication hub. The fourth-generation NVSwitch chip supports 64 NVLinks at 50 GB/s per lane.
• Fabric Topology: In an NVIDIA DGX H100 system, eight GPUs are connected via four NVSwitch chips, creating an all-to-all, fully non-blocking network where each GPU can communicate with every other GPU simultaneously at full NVLink bandwidth.
• Scalability: This switch-based fabric is essential for scaling to NVLink Domains in large-scale clusters, maintaining high bandwidth for collective operations like All-Reduce across hundreds of GPUs.

NVLink is designed for high performance per watt, a critical metric in data center-scale deployments.

• Signal Integrity: Advanced signaling technology allows high data rates over fewer pins and at lower power than PCIe.
• Impact on Total Workload Energy: By drastically reducing the time GPUs spend idle waiting for data, NVLink improves overall system utilization, completing workloads faster and reducing total energy consumption.
• Comparative Efficiency: Transferring data over NVLink consumes significantly less energy per gigabyte than an equivalent transfer over PCIe, directly impacting the operational expenditure of large AI training clusters.

Training modern large language models and vision transformers requires distributing the model and its optimizer states across hundreds of GPUs. NVLink enables efficient model parallelism by allowing layers split across GPUs to communicate gradients and activations at speeds far exceeding PCIe. This minimizes the idle time GPUs spend waiting for data, a bottleneck known as the communication wall. For example, frameworks like Megatron-LM rely on NVLink topologies to efficiently train models with trillions of parameters.

In scientific computing, applications like computational fluid dynamics, molecular dynamics, and climate modeling decompose massive spatial domains across GPU clusters. NVLink is essential for neighbor-to-neighbor communication in these stencil-based computations, where boundary data must be exchanged at every timestep. The low latency and high bandwidth reduce simulation time from weeks to days. Systems like the NVIDIA DGX SuperPOD leverage dense NVLink fabrics to power some of the world's fastest supercomputers for research.

While inference often uses a single GPU, models with extremely long context windows (e.g., 1M+ tokens) or massive multimodal models may exceed the memory of one accelerator. NVLink allows these models to be split across multiple GPUs for inference, with attention key-value (KV) caches and intermediate activations shared at near-internal GPU bandwidth. This is critical for latency-sensitive applications like real-time retrieval-augmented generation where the entire context must be kept resident and accessible.

NVLink isn't just for GPU-to-GPU communication. With NVLink-C2C, it creates a coherent memory space between NVIDIA GPUs and Grace CPUs. This enables a unified virtual address space, allowing the CPU and GPU to access the same data without explicit copies. This heterogeneous memory management simplifies programming and is pivotal for workloads with irregular data access patterns or those that dynamically switch between CPU and GPU execution, such as complex graph analytics and database acceleration.

Modern GPU-accelerated databases (e.g., Kinetica, BlazingSQL) and data processing frameworks use NVLink to enable fast joins, sorts, and aggregations across datasets larger than a single GPU's memory. By treating a cluster of NVLink-connected GPUs as a single, massive parallel compute unit, they can perform in-memory analytics on terabytes of data. This architecture is also key for graph neural network training on massive knowledge graphs, where the graph structure and features are distributed across GPU memory.

PCI Express is the standard high-speed serial computer expansion bus that connects a host CPU to peripheral devices, including GPUs, storage, and network cards. It serves as the baseline interconnect against which NVLink is compared.

• Key Differentiator: NVLink provides significantly higher bandwidth and lower latency than PCIe for GPU-to-GPU and CPU-to-GPU communication.
• Typical Role: PCIe remains the primary interface for connecting GPUs to a system's root complex, while NVLink acts as a supplemental, high-performance fabric between accelerators.
• Generations: PCIe 4.0 offers 16 GB/s per lane (x16 = 32 GB/s bidirectional), while PCIe 5.0 doubles that. NVLink 4.0, in contrast, provides 900 GB/s of bidirectional bandwidth per GPU-GPU link.

Peer-to-Peer access is a capability that allows GPUs within the same system to directly read from and write to each other's device memory, bypassing the need to copy data through host (CPU) memory.

• NVLink's Role: NVLink inherently enables P2P access between connected GPUs, providing the high-bandwidth, low-latency pathway necessary for efficient direct communication.
• Performance Impact: Without a fast interconnect like NVLink, P2P transfers may fall back to slower PCIe lanes or be disabled entirely, creating a major bottleneck for multi-GPU model parallelism and large-batch training.
• CUDA Enabler: The CUDA API (cudaDeviceEnablePeerAccess) manages P2P permissions, but physical hardware links (NVLink or PCIe switches) determine the available bandwidth.

Unified Virtual Memory is a memory management architecture that creates a single, contiguous virtual address space shared between a CPU and one or more GPUs.

• Synergy with NVLink: UVM simplifies programming by allowing pointers to be shared between CPU and GPU code. NVLink accelerates this architecture by providing a high-speed cache-coherent interconnect for CPU-GPU communication, making page migrations and accesses across processors much faster.
• Mechanism: The system transparently migrates memory pages (via demand paging) between CPU and GPU memory based on access patterns. NVLink reduces the latency penalty of these migrations.
• Use Case: Essential for applications with complex, pointer-rich data structures (e.g., graphs, databases) that are accessed by both CPU and GPU.

High Bandwidth Memory is a 3D-stacked DRAM technology physically integrated with a GPU or CPU using through-silicon vias (TSVs), offering extreme bandwidth and improved energy efficiency.

• Complementary Technology: While HBM optimizes on-chip memory bandwidth (between GPU cores and its own DRAM), NVLink optimizes off-chip communication bandwidth (between GPUs or to the CPU).
• System Balance: Modern NVIDIA datacenter GPUs (e.g., H100, B200) combine HBM2e/HBM3 memory with multiple NVLink ports. This creates a balanced system: fast local memory (HBM) paired with fast inter-GPU links (NVLink).
• Bandwidth Comparison: HBM3 can provide over 3 TB/s of memory bandwidth per GPU. NVLink 4.0 provides up to 900 GB/s of inter-GPU bandwidth, ensuring the communication fabric does not become a bottleneck for HBM-fed computations.

Non-Uniform Memory Access is a memory design for multiprocessor systems where the memory access time depends on the memory location relative to the processor.

• NVLink as a NUMA Enabler: In multi-GPU systems with NVLink, memory access is inherently NUMA. Accessing memory on a locally attached GPU (over NVLink) is faster than accessing memory on a GPU connected via another node or through PCIe.

• Programming Model: Efficient multi-GPU applications must be NUMA-aware, minimizing cross-GPU memory accesses and leveraging fast NVLink paths for necessary communication. Libraries like NCCL are optimized for this topology.

• Hierarchy: A typical NVLink-connected GPU pod forms a NUMA domain, where intra-pod communication is fast (NVLink) and inter-pod communication is slower (often over InfiniBand or Ethernet).