Peer-to-Peer (P2P) Access

P2P access enables GPUs within the same system to read and write directly to each other's device memory without staging data through the host's CPU memory. This bypasses the PCIe bus for the CPU leg of the transfer, cutting latency and freeing host memory bandwidth for other tasks. It is essential for workloads like large model training where tensors must be shared between GPUs every iteration.

• Mechanism: Managed by the GPU driver and CUDA runtime via APIs like cudaDeviceEnablePeerAccess().
• Benefit: Can double effective bandwidth for inter-GPU communication compared to a host-mediated copy.

P2P performance is dictated by the physical interconnect between GPUs. NVLink is NVIDIA's dedicated, high-bandwidth interconnect designed specifically for GPU-to-GPU communication, offering significantly higher bandwidth and lower latency than standard PCIe.

• NVLink: Provides a direct mesh or switch fabric between GPUs (e.g., up to 900 GB/s with NVLink 4.0). Enables a unified memory address space across multiple GPUs.
• PCIe P2P: Also supports direct transfers but is limited by shared PCIe switch bandwidth and higher protocol overhead. Performance is highly dependent on system topology (e.g., GPUs on the same PCIe root complex).

Unified Virtual Addressing is a memory model that provides a single, contiguous virtual address space across all GPU and host CPU memory in a system. UVA is a prerequisite for transparent P2P access, as it allows pointers to be dereferenced from any GPU without manual translation.

• Function: The CUDA driver and GPU Memory Management Unit (MMU) translate a virtual address to the correct physical location, whether it's in another GPU's memory or host memory.
• Transparency: Enables APIs like cudaMemcpy to perform P2P transfers automatically when source and destination pointers reside in different GPU memories, simplifying programmer effort.

Not all GPU pairs in a system can establish a direct P2P link. Accessibility depends on the physical system topology—how GPUs are connected via PCIe switches and root complexes—and the presence of NVLink bridges.

• Check Peer Access: Use cudaDeviceCanAccessPeer() to determine if a direct path exists between two GPUs.
• PCIe Complexity: In multi-GPU servers, GPUs may be connected via multiple PCIe switches. Optimal P2P bandwidth is achieved when GPUs are on the same NUMA node and PCIe root complex.
• NVIDIA DGX/ HGX Systems: Utilize fully connected NVLink topologies (e.g., NVSwitch) to provide uniform, high-bandwidth P2P access between all GPUs in the pod.

P2P access is critical for scaling deep learning training and large-model inference across multiple GPUs.

• Model Parallelism: Different layers of a single model are split across GPUs. Activations and gradients are passed directly between GPUs each forward/backward pass, making P2P latency a major factor in training speed.
• Data Parallelism: While each GPU works on a separate data batch, operations like all-reduce for gradient synchronization (e.g., via NCCL) rely heavily on P2P links for optimal performance.
• Inference for Large Models: Running models that exceed the memory of a single GPU requires tensor parallelism, where intermediate results are directly exchanged between GPUs during the generation phase.

While powerful, P2P access has specific constraints and performance nuances.

• Bandwidth is Not Symmetric: Transfer speed from GPU A to GPU B may differ from B to A, depending on interconnect topology.
• Memory Allocation Alignment: For best PCIe P2P performance, allocations should be aligned to 64KB boundaries.
• Software Overhead: Enabling peer access has a small, one-time setup cost. The first transfer between a new peer pair may incur additional latency.
• No CPU Access: Direct P2P memory is not directly accessible by the CPU; the CPU must still access it via the GPU's mapped PCIe BAR, which is slower than host memory.
• System BIOS/SW Settings: Features like Above 4G Decoding and SR-IOV must often be configured correctly in the system BIOS for P2P to function.

P2P access is essential for data-parallel and model-parallel training strategies. In data parallelism, gradients must be averaged across all GPUs after each backward pass. P2P enables high-speed all-reduce operations directly between GPU memories, drastically reducing synchronization overhead compared to routing through host memory. For model parallelism, where different layers of a single model are split across devices, P2P allows activations and gradients to flow directly between GPUs during the forward and backward passes, minimizing inter-GPU communication latency.

Running inference for models whose parameters exceed the memory of a single GPU requires model sharding. P2P access allows different layers or components of the model to reside on separate GPUs. During inference, the intermediate activations are passed directly from one GPU to the next via the high-speed interconnect (e.g., NVLink), creating a pipeline. This avoids the latency of copying data back to the CPU between each layer's execution, which is critical for meeting low-latency service-level agreements (SLAs) for large language models (LLMs) and diffusion models.

In scientific computing, applications like computational fluid dynamics (CFD), molecular dynamics, and climate modeling involve massive, spatially decomposed datasets. P2P access allows GPUs working on adjacent spatial domains to exchange halo regions or boundary data directly. This direct exchange is far more efficient than a centralized gather-scatter pattern via the host, enabling stronger scaling and reducing time-to-solution for simulations that require frequent neighbor communication across the GPU grid.

Systems like GPU-accelerated SQL databases and dataframe libraries (e.g., RAPIDS cuDF) use P2P for multi-GPU join, sort, and aggregation operations. When a dataset is partitioned across GPUs, performing a global operation requires shuffling data. P2P enables direct GPU-to-GPU transfers for this shuffle phase, maximizing throughput for in-memory analytics. This is particularly effective for operations that are bandwidth-bound rather than compute-bound.

NVIDIA's NVLink technology provides a switched fabric for connecting multiple GPUs at extremely high bandwidth. P2P access over NVLink creates a unified memory domain across GPUs, allowing them to access each other's memory as if it were their own. This enables:

• GPU Direct RDMA: For MPI-based applications, GPUs can directly read/write to the memory of GPUs in other nodes.
• Large Unified Address Spaces: Applications can allocate a single buffer that is physically distributed across multiple GPUs but accessible from any GPU in the cluster without explicit programming, managed by the UVM driver.

In multi-GPU systems connected via PCIe, all cross-GPU traffic traditionally routes through the CPU's PCIe root complex. This creates a bottleneck, as bandwidth is shared and latency increases. Enabling P2P access allows GPUs on the same PCIe switch (often within the same root complex) to communicate directly, bypassing the root complex. This topology-aware optimization is crucial for maximizing bandwidth in systems without NVLink, such as those using standard PCIe Gen4/Gen5 switches.

Unified Virtual Memory is a memory management architecture that creates a single, contiguous virtual address space shared between a CPU and GPU(s). It simplifies data sharing and is the conceptual framework that enables seamless P2P transfers.

• Mechanism: Allows both processors to access the same memory pointers using a unified page table.
• Relation to P2P: UVM can manage P2P mappings, allowing a GPU to directly access another GPU's memory through a common virtual address, without programmer intervention for data movement.
• Benefit: Eliminates the need for explicit cudaMemcpy calls between devices, enabling pointer-based access across the system.

Non-Uniform Memory Access is a computer memory design where access time depends on the memory location relative to the processor. In multi-GPU systems with P2P, a similar concept applies: accessing local GPU memory is faster than accessing a peer's memory over NVLink/PCIe.

• GPU NUMA: Memory on a directly attached GPU has lower latency than memory on a peer GPU, even with a fast interconnect.
• Design Implication: Optimal algorithm design must consider data locality. Frequently accessed data should reside on the local GPU, with P2P used for less frequent, bulk transfers.
• System Architecture: Modern multi-socket CPU systems and multi-GPU servers are NUMA systems, requiring careful memory allocation and thread placement.

Memory tiering is a system architecture that organizes different types of memory into a hierarchy based on performance and capacity. P2P access effectively creates a new, high-performance tier in this hierarchy for multi-GPU systems.

• Tiers in a GPU Server: 1) Local GPU HBM (fastest), 2) Peer GPU HBM via NVLink (very fast), 3) Peer GPU HBM via PCIe (fast), 4) CPU RAM (slower), 5) Storage (slowest).
• Automated Management: Advanced drivers and runtime systems can perform page migration between these tiers based on access patterns ("heat").
• Goal: Keep the most frequently accessed data in the fastest available memory tier, using P2P to make peer GPU memory a viable high-speed tier.

Atomic operations are read-modify-write instructions guaranteed to complete without interruption. When using P2P access, atomics can be performed directly on a peer GPU's memory, enabling efficient synchronization and communication across devices.

• Function: Operations like atomicAdd, atomicCAS (compare-and-swap) can target peer memory addresses.
• Use Case: Implementing distributed locks, counters, or consensus mechanisms across GPUs without bouncing data through the host.
• Performance: Atomic operations over NVLink have higher latency than local atomics but are far faster than achieving the same result via CPU-mediated communication. This enables fine-grained, multi-GPU parallel algorithms.