Non-Uniform Memory Access (NUMA)

The defining feature of NUMA is non-uniform latency. Each processor or group of processors (a NUMA node) has its own local memory that it can access with low latency. Accessing memory attached to a remote NUMA node (remote memory) incurs significantly higher latency and potentially lower bandwidth due to the need to traverse an interconnect like a crossbar, HyperTransport, or Intel Ultra Path Interconnect (UPI). Optimizing performance requires data locality—keeping data and the threads that process it on the same NUMA node.

A system's NUMA topology maps the physical relationship between processors and memory banks. Key components include:

• Socket: A physical CPU package, often containing multiple cores.
• NUMA Node: A logical grouping of cores with a local memory controller and attached RAM. A system can have multiple nodes per socket.
• Interconnect: The high-speed links (e.g., AMD's Infinity Fabric, Intel's UPI) that connect nodes, enabling remote memory access. Tools like numactl --hardware or lscpu on Linux reveal this topology, which is critical for process and memory pinning.

Most operating systems (Linux, Windows) use a first-touch policy for memory allocation in NUMA systems. When a process first writes to a page of memory, that page is physically allocated from the local memory of the NUMA node where the writing thread is executing. This policy can lead to suboptimal placement if initialization is done by a single thread, causing all memory to be local to one node while other threads on remote nodes suffer high access latency. Explicit control is achieved via APIs like numactl or libnuma.

NUMA architecture critically affects GPU performance in multi-socket servers. A GPU is physically connected via PCIe to a specific CPU socket (NUMA node). Direct memory access (DMA) transfers between the GPU and local host memory (on the same node) have much higher bandwidth and lower latency than transfers to remote host memory. This makes GPU affinity a key optimization. For optimal GPU Direct Storage or Zero-Copy performance, data buffers must be allocated in the host memory local to the GPU's PCIe root complex.

NUMA is often contrasted with the older Uniform Memory Access (UMA) architecture, also known as Symmetric Multiprocessing (SMP). In UMA, all processors share a single, centralized memory bus. While all memory has equal access latency for any processor, the shared bus becomes a severe contention bottleneck as processor count increases. NUMA solves this scalability limit by distributing memory controllers, but at the cost of introducing complexity in managing non-uniform latency. Modern multi-socket servers are exclusively NUMA-based.

Effective use of NUMA requires explicit software strategies:

• Thread and Memory Pinning: Bind processes/threads to specific cores and allocate memory from the corresponding local node using taskset, numactl, or OpenMP/MPI environment variables.
• NUMA-Aware Allocators: Use allocators (e.g., tcmalloc, jemalloc) that are aware of NUMA topology to allocate memory from the local node of the requesting thread.
• Data Partitioning: Design algorithms to partition data structures so that each NUMA node's threads primarily operate on locally allocated memory, minimizing remote accesses.
• Monitoring: Use performance counters to track NUMA hits/misses and tools like numastat to analyze memory locality.

A memory management architecture that creates a single, contiguous virtual address space shared between a CPU and GPU. This allows both processors to access the same memory pointers using a common page table, simplifying data sharing and enabling on-demand page migration. It is foundational for implementing NUMA-like abstractions across CPU-GPU systems, as the memory management unit (MMU) handles the physical location of data transparently to the programmer.

A capability that allows GPUs within the same system to directly read from and write to each other's device memory over a high-speed interconnect like NVLink or PCIe, bypassing host memory. This creates a NUMA-like topology where accessing a peer GPU's memory has lower latency and higher bandwidth than routing through the CPU. Enabling P2P is critical for scaling multi-GPU training and inference workloads, as it reduces inter-GPU communication bottlenecks.

A system architecture that organizes different types of memory (e.g., HBM, GDDR, DDR, NVMe) into a hierarchy of performance and capacity. Data is automatically promoted to faster tiers (like GPU HBM) or demoted to slower tiers (like CPU DDR or NVMe storage) based on usage heat. This is a dynamic extension of the NUMA principle, managing non-uniform access across media types rather than just physical distance. It's essential for workloads that exceed local GPU memory capacity.

The uniformity of shared resource data that ends up stored in multiple local caches (e.g., L1/L2 caches across CPU cores or GPU SMs). Protocols like MESI ensure all processors see a consistent view of memory by propagating writes. In NUMA systems, maintaining coherency across NUMA nodes introduces significant latency overhead. This is a key differentiator: some high-performance computing (HPC) NUMA systems opt for non-cache-coherent (NCC) NUMA to avoid this penalty, placing the burden of consistency on the software.

A type of stacked memory technology where DRAM dies are vertically integrated with a GPU or CPU using through-silicon vias (TSVs). HBM offers extremely high bandwidth and improved energy efficiency compared to traditional GDDR memory. In a multi-GPU NUMA system, each processor has its own local HBM, which is the fastest memory tier. Optimizing for NUMA involves minimizing accesses to remote HBM across the interconnect and maximizing use of local HBM.

Describes the order and structure in which threads within a GPU warp or CPU core read from or write to memory. Coalesced access patterns are optimal for achieving maximum bandwidth. In a NUMA context, the access pattern must also consider locality. Accessing memory from a thread running on a core within the same NUMA node as the memory (local access) is vastly faster than accessing memory attached to a remote node. Data placement and thread affinity must be aligned.