Non-Uniform Memory Access (NUMA)

The defining characteristic of NUMA is non-uniform memory access latency. Each processor or group of cores (a NUMA node) has its own local memory bank. Access to this local memory is fast. Access to memory attached to a remote NUMA node (remote memory) incurs higher latency due to traversal across an interconnect (e.g., AMD's Infinity Fabric, Intel's Ultra Path Interconnect). Performance optimization hinges on keeping data and computation within the same NUMA node.

A NUMA system is composed of interconnected NUMA nodes. Each node typically contains:

• One or more processor cores (a socket or CPU).
• A local memory controller and attached DRAM.
• A cache hierarchy (L1, L2, L3) shared by the node's cores.
• Links to other NUMA nodes via a coherent interconnect. The system's topology (number of nodes, interconnect bandwidth) is exposed to the operating system via System Resource Affinity Tables (SRAT) and System Locality Information Tables (SLIT) in ACPI.

NUMA systems maintain a single, globally shared address space across all nodes, requiring a cache coherence protocol to ensure all caches have a consistent view of memory. This is typically implemented as a directory-based or snooping protocol over the interconnect. When a core accesses remote data, the protocol fetches the cache line, potentially invalidating copies in other nodes. This generates coherence traffic, which consumes interconnect bandwidth and adds to remote access latency.

A NUMA-aware operating system (e.g., Linux, Windows Server) schedules processes and manages memory to optimize locality. Key mechanisms include:

• CPU affinity: Pinning a process/thread to cores on a specific NUMA node.
• Memory policy: Controlling where a process's pages are allocated (e.g., MPOL_BIND, MPOL_PREFERRED in Linux).
• First-touch policy: The default where a page is allocated from the local memory of the thread that first writes to it. The OS uses /sys/devices/system/node/ (Linux) or GetNumaNodeProcessorMask (Windows) APIs for management.

High-performance applications must be designed for NUMA to avoid severe performance penalties from remote accesses. Strategies include:

• Data partitioning: Structuring algorithms so that each thread primarily works on data allocated in its local node.
• Thread pinning: Explicitly binding threads to specific cores using pthread_setaffinity_np() or numactl.
• NUMA-aware memory allocators: Using allocators like libnuma, jemalloc, or tcmalloc that are conscious of locality.
• Minimizing shared, write-heavy data: To reduce coherence traffic from false sharing.

• UMA (Uniform Memory Access): Traditional SMP where all memory has equal latency for all processors (e.g., via a single shared bus). Simpler but not scalable to many cores.
• ccNUMA (Cache-Coherent NUMA): The modern standard, where the hardware maintains cache coherence across nodes, as described here. This is what is typically meant by "NUMA" today.
• Non-coherent NUMA: Systems without hardware cache coherence, requiring explicit software management—common in some high-performance computing clusters but not in standard server CPUs.

A NUMA architecture organizes processors and memory into interconnected nodes. Each node contains one or more processors and a local bank of memory. Access to local memory (within the same node) is fast, while access to remote memory (in another node) incurs higher latency and lower bandwidth due to traversal across an interconnect (e.g., AMD's Infinity Fabric, Intel's Ultra Path Interconnect). This contrasts with Uniform Memory Access (UMA), where all memory is equidistant from all processors.

• Key Metric: Latency Ratio. Remote access can be 1.5x to 3x slower than local access.
• System View: The Linux command numactl --hardware displays the NUMA topology of a system.

AI training and inference are memory-intensive, moving massive tensors and weights between CPU and accelerator memory. Poor NUMA placement causes severe bottlenecks.

• Data Loading: If a data-loading process is pinned to a CPU core on NUMA Node 0, but the PCIe-attached GPU or NPU is physically connected to NUMA Node 1, all training data must traverse the inter-node link, saturating it and increasing latency.
• Multi-Instance Training: Running multiple model instances (e.g., for hyperparameter search) without NUMA binding can lead to instances competing for remote memory bandwidth, degrading aggregate throughput.
• Solution: NUMA-aware process binding ensures compute threads and their allocated memory reside on the same node as the target accelerator's host interface.

Optimizing for NUMA involves controlling process affinity, memory allocation policy, and device topology.

• CPU and Memory Pinning: Use numactl --cpunodebind=N --membind=N to launch processes, binding them to a specific NUMA node (N).
• First-Touch Policy: In Linux, memory is allocated on the node of the CPU that first writes to it ("touches" it). Initialize data structures on the correct NUMA node.
• Accelerator Affinity: Modern systems report which NUMA node a PCIe device is local to (e.g., via lspci -v or nvidia-smi topo -m). Schedule work on the adjacent CPU cores.
• Interleaved Allocation: For workloads with uniform memory access patterns, numactl --interleave=all distributes memory pages across nodes, averaging out latency but reducing peak local bandwidth.

Modern AI servers feature heterogeneous NUMA domains, where an accelerator like an NPU may have its own on-board memory (HBM) and a host interface tied to a specific CPU NUMA node.

• Direct Memory Access (DMA): Accelerators use DMA engines to transfer data. DMA sourced from local memory completes faster than from remote memory.
• Unified Virtual Addressing (UVA): Frameworks like CUDA simplify programming but do not eliminate NUMA costs. The physical location of the pinned host memory used for DMA still dictates performance.
• CXL (Compute Express Link): Emerging as a game-changer, CXL.mem allows accelerators to act as coherent memory expanders, creating new, software-defined NUMA-like hierarchies that must be managed for optimal data placement.

Identifying NUMA-related performance issues requires specific tools and metrics.

• Hardware Counters: Use perf to monitor node-local vs. remote memory access events (e.g., perf stat -e numa_hit, numa_miss). A high numa_miss rate indicates poor locality.
• Bandwidth Saturation: Monitor interconnect bandwidth using vendor-specific tools (e.g., Intel's pcm). Saturation manifests as high latency for memory operations.
• Latency Profiling: Tools like Intel VTune Profiler or AMD uProf can visualize memory access latency and pinpoint code sections suffering from remote accesses.
• Kernel Statistics: The Linux kernel exposes NUMA statistics in /sys/devices/system/node/nodeN/ for monitoring hit/miss rates and memory usage per node.

NUMA interacts with several other critical system architecture concepts.

• Cache Coherence: NUMA systems maintain a coherent view of memory across all nodes via a directory-based or snooping protocol, but coherence traffic adds to remote access cost.
• False Sharing: Can be exacerbated in NUMA as modified cache lines must be shipped across the interconnect, amplifying performance penalties.
• Virtual Memory & Huge Pages: Using Transparent Huge Pages (THP) or explicit 1GB huge pages reduces page table pressure and can improve the efficiency of NUMA memory management.
• Containers & Orchestration: Orchestrators like Kubernetes must be NUMA-aware. The Kubernetes Topology Manager and associated CPU Manager can align pod resources (CPU, memory, device) to a single NUMA node.

Uniform Memory Access (UMA) is a shared memory architecture where all processors in a multiprocessing system have equal access time to all of main memory. This contrasts directly with NUMA.

• Symmetric Multiprocessing (SMP): The classic UMA architecture, often built on a single system bus or crossbar switch.
• Performance Characteristic: Access latency is uniform but bandwidth is shared, creating a scalability bottleneck as more processors are added.
• Example: Early multi-core servers with a single memory controller.

Cache coherence is the property that ensures all caches in a multiprocessor system have a consistent view of shared memory. It is a critical, complementary protocol to NUMA.

• Problem Solved: Prevents different processors from reading stale or conflicting values for the same memory address.
• Protocols: Common implementations include MESI (Modified, Exclusive, Shared, Invalid) and MOESI.
• NUMA Interaction: In a NUMA system, cache coherence traffic must traverse the interconnect between nodes, adding latency and complexity. Protocols like AMD's HyperTransport and Intel's QuickPath Interconnect (QPI) handle this.

Processor affinity (or CPU pinning) is the practice of binding a software process or thread to specific CPU cores or a NUMA node. It is a primary software optimization for NUMA systems.

• Goal: Minimize remote memory accesses by ensuring a process runs on cores local to the memory where its data resides.
• First-Touch Policy: Many operating systems allocate memory pages on the node where the thread that first accesses them is running, making initial placement critical.
• Tools: Implemented via system calls like sched_setaffinity() on Linux or APIs within OpenMP and MPI.

Cache-Coherent NUMA (ccNUMA) is the predominant form of NUMA in modern servers, where a hardware coherence protocol maintains consistency across all processor caches and distributed memory.

• Industry Standard: All major x86 and ARM server processors (e.g., AMD EPYC, Intel Xeon, ARM Neoverse) implement ccNUMA.
• Transparency: The coherence hardware makes the system appear as a single, shared-memory machine to software, simplifying programming but requiring awareness for peak performance.
• Interconnect: Relies on high-speed, coherent interconnects like Infinity Fabric (AMD) or Ultra Path Interconnect (UPI) (Intel).

Memory latency is the time delay between a memory read request and the data's availability. NUMA architectures explicitly create non-uniform latency.

• Local vs. Remote Access: In a 2-node NUMA system, remote access can be 1.5x to 3x slower than local access.
• Quantifying Impact: Performance profiling tools measure Last-Level Cache (LLC) misses and subsequent DRAM access latency to identify costly remote accesses.
• Mitigation: Strategies include data replication (for read-only data), migration of hot pages, and interleaving allocations across nodes.

A NUMA node is the fundamental building block of a NUMA system, comprising one or more processor cores and their directly attached, local bank of memory.

• Composition: Typically includes CPU cores, their caches (L1/L2/L3), a memory controller, and a physical RAM bank.
• System View: The numactl --hardware command on Linux displays all NUMA nodes, their free memory, and distances (latency/cost) between nodes.
• Distance Matrix: A key system descriptor that quantifies the access cost between nodes, guiding the OS and application scheduler.