Non-Uniform Memory Access (NUMA)

In a NUMA system, each processor or group of cores (a NUMA node) has its own local memory. Accessing this local memory is fast. Accessing memory attached to a remote NUMA node (remote memory) incurs higher latency due to traversal across an interconnect (e.g., Intel's Ultra Path Interconnect, AMD's Infinity Fabric). The key performance goal is to maximize local memory accesses and minimize costly remote accesses through careful data and thread placement.

NUMA affinity is the practice of binding software threads and processes to specific CPU cores and their associated memory nodes. This is achieved via OS or library calls (e.g., numactl on Linux, SetThreadAffinityMask on Windows). Thread pinning ensures a thread's memory allocations come from its local node and its execution stays on cores within that node, preventing the OS scheduler from migrating it across nodes and causing unpredictable remote memory access penalties.

Most operating systems use a first-touch policy for memory allocation in NUMA systems. The physical memory page is allocated from the NUMA node where the thread that first writes to (touches) that page is executing. This makes initialization critical: if a master thread on Node 0 initializes all data, all memory is allocated to Node 0. Subsequent threads on Node 1 will suffer remote accesses. Proper initialization involves touching data in parallel from threads pinned to their respective nodes.

The NUMA interconnect is a shared resource with finite aggregate bandwidth. When many cores simultaneously access remote memory, they saturate this link, creating contention. This can degrade performance more severely than the base latency increase. Performance profiling must monitor interconnect utilization (e.g., via perf or vendor tools). Optimizations aim to reduce both the volume and the synchronization of remote accesses to avoid flooding the interconnect.

NUMA principles extend to systems with discrete accelerators like GPUs or NPUs. Here, the accelerator's device memory is analogous to a remote NUMA node with very high access latency. Frameworks like Compute Express Link (CXL) aim to create a more NUMA-like, cache-coherent memory space between CPUs and accelerators. For optimal performance, data should reside in the executing processor's local memory, whether it's a CPU core or an accelerator's HBM.

Analyzing NUMA performance requires specific tools:

• numactl / numastat: Linux utilities to control policy and view memory allocation statistics per node.
• likwid-perfctr: Provides detailed performance counter monitoring per NUMA node.
• Vendor Profilers: Intel VTune Profiler and AMD uProf have dedicated NUMA analysis views.
• Key Metrics: Local/remote memory access ratios, interconnect bandwidth utilization, and remote cache hit rates. The goal is to identify which processes or data structures are causing excessive remote accesses.

In multi-socket NUMA systems, inter-socket memory access can be 2-3x slower than local memory access. For distributed training frameworks like PyTorch DDP or TensorFlow MirroredStrategy, improper process affinity can cause severe bottlenecks.

• DataLoader processes pinned to the wrong NUMA node incur high latency fetching training batches.
• Gradient synchronization across sockets becomes a major throughput limiter.
• Optimal strategy: Bind each training process to a specific socket and allocate its working memory locally using numactl or framework-specific APIs.

High-throughput inference servers (e.g., TensorRT, Triton) must manage request batching across NUMA nodes to avoid cross-socket memory transfers.

• Batch composition should group requests being processed on the same socket.
• Model weights should be replicated in each socket's local memory to prevent remote fetches during forward passes.
• Static binding of inference threads to cores prevents expensive migrations that break memory locality.

Models exceeding a single socket's memory capacity (e.g., LLMs with >80B parameters) force cross-NUMA memory allocation. This creates a permanent performance penalty.

• Techniques like NUMA-aware model parallelism split layers across sockets, keeping weight gradients local.
• ZeRO-Offload and similar CPU-offload strategies must partition optimizer states according to NUMA topology.
• Transparent Huge Pages (THP) can worsen remote access latency; use explicit 1GB pages allocated locally per node.

GPUs and NPUs connected via PCIe to a specific CPU socket have optimal bandwidth to that socket's local memory. GPU-Direct RDMA performance depends on NUMA alignment.

• CUDA device affinity APIs (cudaSetDeviceFlags) should bind GPU contexts to the local NUMA node.
• Data staging buffers for GPU transfers must be allocated in local CPU memory to avoid extra hops.
• Multi-GPU systems often map GPUs to different sockets; peer-to-peer transfers require careful NUMA-aware placement.

Modern ML frameworks provide NUMA controls, but often require explicit configuration.

• PyTorch: Set torch.set_num_threads() per process and use numactl wrapper scripts. The torch.nn.parallel.DistributedDataParallel benefits from torch.multiprocessing with correct spawn context.
• TensorFlow: Use tf.config.threading APIs and the NUMA_BINDING environment variable for thread placement.
• OpenMP: The OMP_PLACES and OMP_PROC_BIND environment variables are critical for math library kernels (e.g., oneDNN, cuBLAS).

Orchestrators like Kubernetes must be configured to respect NUMA topology for AI workloads.

• Kubernetes Topology Manager policy should be set to restricted or single-numa-node for guaranteed locality.
• CPU Manager policies align with exclusive CPU allocation for pods.
• Device Plugins (e.g., for GPUs) should expose NUMA node affinity in their resource advertisements.
• Docker supports --cpuset-cpus and --cpuset-mems flags to pin containers to specific sockets and memory nodes.

NUMA Affinity is the practice of explicitly binding software processes or threads to specific CPU cores and their associated local memory nodes to minimize access latency. This is a critical manual optimization in high-performance computing (HPC) and database systems.

• Pinning: Using OS or runtime commands (e.g., numactl, taskset) to lock a process to a core.
• First-Touch Policy: Many OSes allocate memory pages on the node where the thread that first accesses them resides, making initial thread placement crucial.
• Impact: Correct affinity can reduce memory latency by 2-3x compared to unbound execution, directly translating to lower tail latency in latency-sensitive applications.

Uniform Memory Access (UMA) is the contrasting memory architecture to NUMA, where all processors in a system share a single, centralized memory bank via a common bus or crossbar switch. Access time to memory is identical for all processors, regardless of physical proximity.

• Symmetric Multiprocessing (SMP): The classic UMA architecture, prevalent in early multi-core systems.
• Simplified Programming: Eliminates the need for affinity tuning, as all memory accesses have uniform cost.
• Scalability Limit: The shared bus becomes a bottleneck as more cores are added, leading to the adoption of NUMA designs for modern multi-socket servers with tens to hundreds of cores.

Cache Coherence is the hardware mechanism that ensures all processor caches in a shared-memory multiprocessor system maintain a consistent view of the same memory location. It is a prerequisite for NUMA systems to function correctly.

• Coherence Protocols: Protocols like MESI (Modified, Exclusive, Shared, Invalid) manage the state of cached data lines across all cores.
• Coherency Domain: In a NUMA system, each node typically has its own coherence domain, and inter-node communication is handled by a coherent interconnect like Intel's QuickPath Interconnect (QPI) or AMD's Infinity Fabric.
• Performance Cost: Maintaining coherence across NUMA nodes introduces additional latency for remote cache misses, a key factor in NUMA performance analysis.

Compute Express Link (CXL) is an open industry-standard interconnect that provides high-bandwidth, low-latency connectivity between the CPU and devices like accelerators (GPUs, FPGAs) and memory expanders, while maintaining memory coherency. It represents the evolution of interconnect technology critical for heterogeneous systems.

• Memory Pooling: CXL 3.0 enables memory pooling, where a block of CXL-attached memory can be dynamically shared by multiple hosts, creating a new, software-defined NUMA topology.
• Coherent Device Access: Allows accelerators to access host CPU memory coherently, simplifying programming models compared to traditional PCIe.
• Industry Adoption: Supported by Intel, AMD, ARM, and major cloud providers, it is foundational for next-generation data center architectures.

Locality of Reference is a fundamental principle of computer science describing the tendency of a processor to access the same set of memory locations repeatedly over a short period (temporal locality) or nearby memory locations (spatial locality). NUMA optimization is essentially the strategic enforcement of this principle at the hardware node level.

• Temporal Locality: Optimized by CPU caches. Data accessed recently is likely to be accessed again.
• Spatial Locality: Optimized by cache line fetches. Accessing one address makes nearby addresses cheap to access.
• NUMA Application: Designing algorithms and data structures to maximize access within a single NUMA node (local locality) minimizes expensive remote accesses, directly improving performance.

The Roofline Model is an analytical performance model that visualizes the attainable performance of a computational kernel as a function of its Operational Intensity. It is used to diagnose whether a kernel is compute-bound or memory-bound, which is directly influenced by NUMA effects.

• Operational Intensity: Meas in FLOPs/byte, it's the ratio of arithmetic work to data movement from main memory.
• Memory-Bound Kernels: Kernels with low operational intensity hit the memory bandwidth "roof." For these, NUMA remote access latency and reduced effective bandwidth are severe performance limiters.
• NUMA-Aware Analysis: A kernel's performance roofline is lower when data is fetched from remote NUMA nodes, making the model essential for predicting the impact of poor data placement.