Unified Virtual Memory (UVM)

UVM's foundational feature is the creation of a unified, 64-bit virtual address space accessible by both the CPU and GPU. This eliminates the need for developers to manage separate pointers for host and device memory. Key implications include:

• Pointer Consistency: A pointer has the same value and meaning on both processors.
• Simplified APIs: Reduces or removes explicit cudaMemcpy operations for data movement.
• Dynamic Data Structures: Enables the use of complex, pointer-based data structures (like linked lists or trees) that can be traversed by either processor without manual translation.

UVM enables demand paging and automatic page migration. Memory pages are moved between CPU system RAM and GPU device memory transparently based on access patterns.

• Page Fault Driven: When a GPU kernel accesses a page residing in host memory, a GPU page fault occurs. The UVM driver then migrates that page to GPU memory.
• Migration Heuristics: Pages can also be proactively migrated or replicated based on usage to optimize performance.
• Memory Oversubscription: This mechanism allows workloads to allocate more memory than the physical GPU VRAM capacity, using system RAM as a backing store.

UVM maintains cache coherency between the CPU and GPU caches for the shared address space. This ensures both processors see a consistent view of data.

• Granular Coherency: Coherency is typically managed at the granularity of a memory page.
• Protocol Overhead: The coherency protocol (like NVIDIA's HMM) introduces management overhead but is essential for correctness in concurrent access scenarios.
• Access Permissions: Pages have permissions (e.g., CPU-read/write, GPU-read/write) that are managed by the UVM driver to enforce coherency rules.

A direct benefit of UVM is enabling efficient zero-copy transfers. The GPU can access data directly from host-pinned memory without a preliminary copy to device memory.

• Reduced Latency: Eliminates the explicit copy latency for data only accessed once or infrequently.
• Host Memory as Cache: System RAM effectively acts as a slower, larger cache tier for the GPU.
• Use Case: Ideal for applications with sparse, unpredictable access patterns or where data is generated/consumed by the CPU in real-time.

UVM allows the total allocated memory of active GPU processes to exceed the physical GPU memory (VRAM) capacity. This is known as memory overcommit or oversubscription.

• Backing Store: Non-resident pages spill over to system memory (DRAM) and, under extreme pressure, to storage (SSD) via the OS swap.
• Performance Impact: Access to migrated pages incurs higher latency. Performance depends heavily on locality and the speed of the PCIe bus or NVLink.
• Enables Larger Models: Allows running models whose working set size is larger than a single GPU's VRAM, albeit with potential performance degradation.

In multi-GPU systems, UVM extends the unified address space across multiple GPUs, facilitating direct peer-to-peer (P2P) access.

• Direct GPU-to-GPU Access: GPUs can directly read/write to each other's memory using load/store instructions, bypassing the host CPU.
• NVLink Optimization: When GPUs are connected via NVLink, P2P accesses achieve much higher bandwidth and lower latency than over PCIe.
• Simplified Multi-GPU Programming: Developers can program multiple GPUs as if they share a large, pooled memory resource, simplifying data partitioning and communication.

Page-locked memory, also known as pinned memory, is host (CPU) memory that is prevented from being paged out to disk by the operating system. This locking enables high-bandwidth Direct Memory Access (DMA) transfers between the host and GPU device memory, a prerequisite for many UVM optimizations.

• Purpose: Eliminates the need for the operating system to copy data between pageable and pinned buffers before a transfer, reducing latency.
• UVM Role: While UVM can work with pageable memory, using pinned memory for frequently accessed data regions significantly boosts transfer performance within the unified address space.
• Trade-off: Excessive use can reduce available pageable system memory for other processes.

Zero-copy transfers are a technique where a GPU accesses data directly from host memory without an explicit programmer-managed copy into GPU device memory. This is a key performance benefit enabled by UVM architectures.

• Mechanism: The GPU's memory management unit (MMU) maps host memory pages into the GPU's virtual address space. On access, data is transferred over the PCIe bus on-demand.
• Advantage: Reduces host memory pressure by avoiding duplicate copies and can lower latency for infrequently accessed or large datasets.
• Consideration: Access latency is higher than device-resident memory, making it ideal for data used once or when transfer time outweighs slower access speed.

Demand paging is the core virtual memory mechanism that underpins UVM. Data resides in a slower backing store (e.g., system RAM) and is only migrated to faster GPU memory when actively accessed, triggered by a GPU page fault.

• Page Fault: An exception raised when a GPU thread accesses a virtual address whose physical page is not resident in GPU memory. The UVM driver handles this fault by migrating the page.
• Benefit: Enables memory overcommit, allowing applications to allocate more memory than the GPU's physical capacity, with only actively used "hot" pages occupying fast memory.
• Overhead: Frequent page faults and migrations can introduce significant latency, requiring careful data access pattern design.

Memory tiering is a system architecture that organizes heterogeneous memory types (e.g., HBM, GDDR, DDR, NVMe) into a performance hierarchy. UVM acts as the software abstraction that manages data placement and movement across these tiers.

• Tiers: Fast, small capacity (GPU HBM) → Slower, larger capacity (CPU DDR) → Very slow, vast capacity (NVMe Storage).
• UVM's Role: The UVM driver, often with hints from the application or hardware counters, automatically promotes "hot" data to faster tiers and demotes "cold" data to slower ones.
• Goal: Maximize performance-per-dollar and effective capacity by keeping the working set in the fastest possible tier.

Peer-to-Peer (P2P) access allows multiple GPUs in a system to directly read/write each other's memory without routing through host RAM. NVLink is NVIDIA's high-bandwidth interconnect that makes P2P extremely fast. UVM extends this model.

• With UVM: A unified virtual address space can span multiple GPUs. A pointer can reference memory physically located on another GPU; access triggers a P2P transfer over NVLink/PCIe.
• Benefit: Dramatically simplifies multi-GPU programming. Developers can allocate memory without knowing its physical location, and the UVM runtime handles efficient cross-GPU access.
• Requirement: Requires compatible hardware (GPUs connected via NVLink or PCIe with P2P support).

A stream-ordered memory allocator is a GPU memory management scheme where allocations are tied to a specific CUDA stream. Memory is automatically reused or freed based on the completion of work in that stream, minimizing fragmentation.

• Problem it Solves: Traditional cudaMalloc/cudaFree cause fragmentation and global synchronization. UVM introduces similar challenges for managing the unified address space.
• Synergy with UVM: Modern allocators like cudaMallocAsync work with UVM, allowing for asynchronous, stream-ordered allocation of unified memory. This is critical for high-throughput inference servers using techniques like continuous batching.
• Outcome: Reduces allocation latency and fragmentation, leading to more predictable performance in dynamic, multi-tenant GPU environments.