NUMA (Non-Uniform Memory Access)

The defining characteristic of NUMA is a memory latency hierarchy. Each processor or group of processors (a NUMA node) has its own local memory. Access to this local memory is fast. Access to memory attached to another processor (remote memory) is slower, traversing an interconnect like Intel's Ultra Path Interconnect (UPI) or AMD's Infinity Fabric. This creates a non-uniform access pattern where performance depends on data locality.

A NUMA node is the fundamental building block, typically comprising:

• One or more CPU cores (a socket or a core complex).
• A local bank of DRAM (the node's memory).
• A local memory controller.
• Links to other NUMA nodes via the interconnect. The operating system enumerates these nodes, and software can query topology via interfaces like libnuma on Linux. Efficient programming requires thread and memory affinity to keep processes and their data on the same node.

NUMA nodes communicate via a high-speed interconnect. This hardware also runs a cache coherency protocol (e.g., MESI, MOESI) across the entire system. This protocol ensures that all processor caches have a consistent view of shared memory, even when data is migrated or replicated. The overhead of maintaining coherency across the interconnect for remote accesses is a primary source of NUMA penalty. Protocols are directory-based or snooping-based to track sharing states.

Most operating systems use a first-touch policy for memory allocation. When a process first writes to a page of memory, that page is physically allocated from the DRAM of the NUMA node where the writing thread is executing. This can inadvertently pin data to a remote node if the initializing thread is not carefully placed. Explicit memory policy APIs (e.g., mbind(), numactl) allow developers to control allocation, such as interleaving pages across nodes for uniform bandwidth.

Poor NUMA awareness can degrade performance by 200-300% due to remote memory latency. Key optimization strategies include:

• Thread Pinning: Binding threads to specific cores/nodes.
• Data Placement: Allocating data on the node where it will be processed.
• NUMA-Aware Algorithms: Designing parallel algorithms to minimize cross-node communication.
• Topology Awareness: Using system tools (numactl, lstopo) to understand the hardware layout. The goal is to maximize accesses to local memory.

Most modern systems are Cache-Coherent NUMA (CC-NUMA), where hardware maintains coherence across all processor caches, presenting a single shared memory image to software. Historically, Non-Cache-Coherent NUMA (NCC-NUMA) systems existed, requiring explicit software management for coherence, similar to clusters. CC-NUMA is the standard for commercial servers, workstations, and many high-performance computing systems, providing a easier programming model at the cost of complex interconnect hardware.

The practice of pinning software threads to specific CPU cores and allocating memory from the local NUMA node of those cores. This ensures memory accesses are local, minimizing latency.

• Tools: numactl (Linux), SetThreadAffinityMask (Windows).
• Goal: Prevent the OS scheduler from migrating threads between sockets, which would cause all memory accesses to become remote.
• Example: Binding a database process to cores on NUMA node 0 and allocating its memory from node 0.

A default memory policy in many operating systems where a page of memory is allocated from the NUMA node of the CPU core that first writes to (touches) it.

• Implication: Initializing data structures with the threads that will primarily use them is critical for optimal placement.
• Pitfall: A master thread initializing all data can cause all memory to be allocated on one node, creating a hotspot.
• Optimization: Parallelize initialization so each thread touches the data it will own.

A memory policy that distributes (stripes) pages of a large memory allocation round-robin across all available NUMA nodes.

• Use Case: For applications with very large, uniformly accessed datasets where bandwidth is more critical than latency.
• Effect: Averages out remote access latency and balances memory controller load.
• Trade-off: Increases average latency compared to perfect local placement but prevents worst-case all-remote scenarios.
• Implementation: Using numactl --interleave=all before launching an application.

Designing concurrent data structures (e.g., queues, hash maps, allocators) to partition data per NUMA node, minimizing cross-node communication.

• Principle: Employ a sharding or partitioning scheme aligned with NUMA nodes.
• Example: A concurrent hash map where each NUMA node manages its own stripe of buckets and associated locks. Threads on a node primarily access their local stripe.
• Benefit: Dramatically reduces the frequency of expensive atomic operations and cache-line transfers across the interconnect (e.g., Intel UPI, AMD Infinity Fabric).

Quantifying the performance differential between local and remote memory accesses to guide optimization efforts.

• Typical Latency Ratio: Remote access can be 1.5x to 3x slower than local access, depending on the system architecture and interconnect.
• Bandwidth Impact: Aggregate cross-socket bandwidth is typically lower than total local bandwidth.
• Tools: Microbenchmarks (e.g., Intel MLC), performance counters monitoring UNC_CHA_TOR_INSERTS (Intel) for remote traffic, or numastat for OS-level page placement statistics.

Programmatically querying the system's NUMA topology to make informed scheduling and allocation decisions at runtime.

• APIs: hwloc (Portable Hardware Locality library), Linux /sys/devices/system/node/, Windows GetNumaNodeProcessorMaskEx.
• Information Gathered: Number of nodes, cores per node, memory size per node, and distances (latency/affinity) between nodes.
• Purpose: Enables adaptive software that configures its parallelism and memory use based on the actual hardware it's running on, essential for portable performance.

Symmetric Multiprocessing is a multiprocessor architecture where all processors share a single, uniform memory space and have equal access to all I/O devices. This contrasts with NUMA, where memory access is non-uniform.

• Uniform Memory Access (UMA): All memory accesses have the same latency, regardless of which processor makes the request.
• Centralized Memory Controller: A single memory bus or interconnect serves all processors, which can become a bottleneck.
• Use Case: Found in simpler, smaller-scale multi-core systems where latency uniformity simplifies programming but limits scalability.

Cache-Coherent Non-Uniform Memory Access is the predominant form of NUMA in modern x86 and ARM servers. It extends the basic NUMA model by maintaining a single, coherent view of memory across all processor caches.

• Hardware Coherence Protocol: Uses a directory-based or snooping protocol (e.g., MESI) to ensure all CPUs see the most recent value of a memory location.
• Transparency: The coherence is handled entirely in hardware, making the system appear as a shared-memory machine to software, despite physical non-uniformity.
• Performance Impact: Remote cache hits are faster than accessing remote main memory, but coherence traffic adds overhead to cross-socket communication.

A NUMA Node is the fundamental building block of a NUMA system, consisting of one or more processors and their directly attached, local memory. The system is a collection of interconnected NUMA nodes.

• Local Memory: Memory physically attached to the node's memory controllers. Access is fast (low latency, high bandwidth).
• Remote Memory: Memory attached to a different NUMA node. Access incurs higher latency and potentially lower bandwidth across an interconnect (e.g., Intel UPI, AMD Infinity Fabric).
• Affinity: A core concept where a process or thread is scheduled on a core within the node containing the data it uses most frequently to minimize remote accesses.

The First-Touch Policy is a common operating system and runtime memory allocation strategy in NUMA systems. Memory pages are allocated on the NUMA node where the thread that first writes to (touches) them is currently executing.

• Automatic Placement: Simplifies programming by automatically attempting to co-locate data with the using thread.
• Potential Pitfall: If initialization is performed by a single thread (e.g., a main thread), all memory may be allocated on one node, leading to severe imbalance and remote access for worker threads on other nodes.
• Mitigation: Requires careful parallelization of data initialization or explicit memory placement APIs (e.g., numactl, mbind in Linux).

The NUMA Interconnect is the high-speed link that connects NUMA nodes, enabling processors to access remote memory and maintain cache coherence. Its bandwidth and latency are critical to overall system performance.

• Intel Ultra Path Interconnect (UPI): The point-to-point interconnect used in Intel Xeon Scalable processors for multi-socket communication.
• AMD Infinity Fabric: AMD's scalable, coherent interconnect used within and between EPYC processor dies and sockets.
• Performance Characteristic: Bandwidth for remote access is typically a fraction of local memory bandwidth, and latency can be 1.5x to 3x higher. Optimizing algorithms to minimize cross-interconnect traffic is a key NUMA tuning goal.

Memory Affinity (or NUMA Pinning) is the explicit control of thread-to-core and memory-to-node binding to optimize performance on a NUMA system, overriding the OS's default scheduler and memory allocator.

• Thread Pinning: Binding a specific thread or process to a set of cores on a particular NUMA node using taskset or numactl --cpunodebind.
• Memory Pinning: Directing memory allocations to a specific NUMA node using numactl --membind or APIs like libnuma.
• Goal: Ensures a thread runs on cores local to the memory it uses, eliminating unpredictable remote access penalties. Essential for achieving deterministic, high-performance in latency-sensitive and HPC workloads.