Unified Memory

CUDA Unified Memory is NVIDIA's implementation, introduced with the Kepler architecture and refined in Pascal. It provides a single, coherent address space across CPU and GPU memory, managed by the CUDA driver and runtime. Key features include:

• On-demand page migration: Pages of memory are migrated between host and device upon access, triggered by a page fault.
• Memory oversubscription: Allows allocation of a working set larger than the physical GPU memory, with pages swapped as needed.
• Coherence: Maintains consistency through hardware support in GPUs (since Pascal) and the driver, eliminating the need for explicit cudaMemcpy calls for managed data.
• Access counters: Used for optimizing page placement and prefetching hints via cudaMemAdvise.

ROCm's HIP Unified Memory provides a portable, CUDA-like unified memory model for AMD GPUs and other accelerators. It is a core component of the Heterogeneous-Compute Interface for Portability (HIP) runtime. Its design emphasizes:

• System-wide atomics: Supports atomic operations across CPU and GPU on managed memory.
• Fine-grained coherence: Available on specific GPU architectures (e.g., CDNA2/MI200+), allowing concurrent CPU and GPU access to the same memory region with automatic coherence.
• HIP Managed Memory API: Uses hipMallocManaged and relies on the ROCk kernel driver for page fault handling and migration.
• Integration with hsaAMD for systems with AMD CPUs, enabling tighter integration with the Host System Agent.

Intel oneAPI Unified Shared Memory (USM) is a memory model for heterogeneous programming across CPUs, GPUs, and FPGAs using Data Parallel C++ (DPC++) and SYCL. It defines three allocation types:

• Device allocations: Reside on and are accessible only by a specific device (like traditional device memory).
• Host allocations: Reside in host-accessible memory, accessible by all devices and the host.
• Shared allocations: Can migrate between host and device memory for optimal performance, providing a unified pointer. USM relies on the Level Zero or OpenCL runtime for management and explicit/prefetch operations like mem_advise and mem_prefetch.

Apple's Unified Memory Architecture (UMA) is a hardware design in M-series SoCs where the CPU, GPU, and Neural Engine share a single, physical pool of high-bandwidth, low-latency memory. This is distinct from software-managed unified memory:

• Physically unified: No separate VRAM; all processors access the same DRAM chips via a high-bandwidth fabric.
• Hardware coherence: The interconnect (e.g., Apple Fabric) maintains cache coherence automatically.
• Software transparency: For developers using Metal or ML Compute, memory appears as a single space; the Metal API (MTLHeap, MTLBuffer) and MLCompute frameworks abstract the physical unity.
• Bandwidth benefits: Eliminates the cost of copying data between discrete memory pools, crucial for graphics and machine learning workloads.

Compute Express Link (CXL) is an emerging industry standard that enables a form of hardware-level unified memory at the system scale. While not a software framework itself, it enables new implementations:

• Memory pooling: Allows an accelerator (like an NPU) with CXL to dynamically share a pool of memory attached to the CXL fabric with the host CPU and other devices.
• Coherent access: CXL.mem protocol provides load/store semantics with cache coherence, allowing the CPU and device to access the same memory region without software-driven copies.
• Future frameworks: Upcoming vendor SDKs and OS-level memory managers will leverage CXL to expose a unified, byte-addressable memory space across heterogeneous compute elements, reducing the need for pinned host memory.

Machine learning frameworks provide high-level abstractions that leverage underlying unified memory implementations, simplifying developer workflows.

• PyTorch CUDA Unified Memory: Using torch.cuda.memory or the pin_memory() method for DataLoaders, which often utilizes CUDA unified memory under the hood for asynchronous data transfers.
• TensorFlow with tf.data and prefetch_to_device: Automatically stages data in pinned/unified memory to overlap computation and data transfer.
• Device-agnostic tensors: Frameworks like JAX and newer versions of PyTorch aim for device-agnostic code, where the runtime decides memory placement, potentially using unified memory on supported backends (e.g., PJRT with CUDA).
• Implicit Management: These frameworks handle the complexity of advising, prefetching, and synchronization, allowing data scientists to focus on model logic.

A multiprocessor architecture where memory access time depends on the memory location relative to the processor. In a NUMA system, each processor has its own local memory, which it can access faster than non-local (remote) memory belonging to another processor. This contrasts with Uniform Memory Access (UMA). Unified memory in accelerator frameworks must often operate across NUMA boundaries, adding complexity to data placement decisions for optimal performance.

A high-performance RAM interface for 3D-stacked DRAM, designed to provide vastly higher bandwidth in a compact form factor. HBM stacks multiple DRAM dies vertically and connects them to a logic die via through-silicon vias (TSVs). It is a common physical memory technology used in GPUs and AI accelerators (NPUs) that leverage unified memory architectures. The high bandwidth is critical for feeding data-hungry parallel processors.

• Key Feature: Extremely high bandwidth (e.g., >1 TB/s) in a small physical footprint.
• Use Case: The physical DRAM often accessed via a unified address space in modern accelerators.

An open industry-standard interconnect that provides high-bandwidth, low-latency connectivity between the CPU and devices like accelerators and memory expanders. CXL maintains memory coherency between the CPU and connected devices, which is a foundational enabler for efficient, cache-coherent unified memory systems. It allows an NPU to access host CPU memory seamlessly and vice-versa, reducing the need for explicit data copies.

• Protocols: CXL.mem for memory access, CXL.cache for caching, CXL.io for I/O.
• Impact: Enables a true, low-overhead unified memory paradigm across discrete components.

Host system memory (RAM) that is prevented from being paged out to disk by the operating system, also known as page-locked memory. This is a critical prerequisite for high-speed Direct Memory Access (DMA) transfers to devices like GPUs or NPUs. When using unified memory, the underlying runtime often automatically pins memory pages involved in data transfers or managed allocations to ensure peak transfer bandwidth and guarantee physical address stability for the device.

A hardware feature that allows peripheral devices or accelerators to transfer data directly to and from system memory without continuous intervention from the central processing unit (CPU). DMA engines are the workhorses behind the data movement in a unified memory system. When a CPU instruction accesses a unified memory address that resides on a device, a DMA transfer (often handled transparently by hardware/runtime) is typically initiated to fulfill the request.

Defines the legal ordering of memory operations (loads and stores) in a parallel computing system, specifying the possible values a read operation can return. In a system with unified memory accessed by both a CPU and an accelerator, a well-defined consistency model (e.g., release-acquire semantics) is essential. It dictates when writes by one processor become visible to another, ensuring programmer expectations match hardware behavior and preventing subtle concurrency bugs.