Non-Uniform Memory Access (NUMA)

The defining characteristic of NUMA is non-uniform memory access times. A processor can access its local memory (memory attached to its own socket or NUMA node) with lower latency and higher bandwidth than remote memory (memory attached to another processor's socket). This variance is the core performance consideration. For example, local access might be ~100 nanoseconds, while remote access can be 1.5 to 3 times slower, depending on the system interconnect (e.g., AMD Infinity Fabric, Intel Ultra Path Interconnect).

A NUMA system is organized into NUMA nodes. Each node typically contains:

• One or more CPU cores (a processor group)
• A local bank of main memory (RAM)
• An integrated memory controller Nodes are connected via a high-speed interconnect. The operating system enumerates these nodes, and software can query topology via interfaces like numactl on Linux. Optimal performance is achieved by allocating memory and scheduling threads on the same node.

NUMA nodes communicate via a coherent interconnect that maintains cache coherence across the entire system. This interconnect is a critical bottleneck. Common technologies include:

• AMD's Infinity Fabric
• Intel's QuickPath Interconnect (QPI) / Ultra Path Interconnect (UPI)
• HyperTransport (older systems) As core counts increase, the interconnect must scale to handle the growing volume of cache coherence traffic and remote memory requests, making topology a key design factor.

NUMA-aware operating systems (Linux, Windows, modern UNIX) schedule processes/threads and allocate memory with topology in mind. Policies include:

• Local allocation: Memory is allocated from the node where the thread is running.
• Interleaving: Memory pages are striped across nodes to average out latency.
• Bind policies: Pinning threads to specific nodes. Without awareness, a process can suffer severe performance degradation if its threads are scheduled on one node while its memory is allocated on another (NUMA thrashing).

A common default policy in Linux is first-touch. The physical memory page for a virtual address is allocated from the NUMA node of the CPU that first writes to (or 'touches') that address. This can lead to suboptimal layouts if initialization is done by a single thread. For performance-critical applications, developers must explicitly manage memory placement using libraries or system calls (mbind, set_mempolicy).

NUMA is often contrasted with Uniform Memory Access (UMA), or Symmetric Multiprocessing (SMP), architecture. In UMA, all processors share a single, centralized memory bus and controller, so access time to memory is the same for all CPUs. UMA becomes a bandwidth bottleneck as processor counts increase. NUMA scales better for high-core-count systems by distributing memory controllers, albeit at the cost of introducing access latency non-uniformity.

The organization of memory subsystems in a computing or cognitive architecture into multiple levels with distinct trade-offs between speed, capacity, and cost. A classic hierarchy includes:

• Registers: Fastest, smallest, inside the CPU.
• CPU Caches (L1/L2/L3): Fast SRAM, holds frequently used data.
• Main Memory (RAM): Slower, larger, volatile working memory.
• Persistent Storage (SSD/HDD): Slowest, largest, non-volatile.

NUMA is a design within the main memory layer of this hierarchy, optimizing access for multi-processor systems. In agentic AI, a similar conceptual hierarchy exists with working memory buffers, vector stores, and knowledge graphs.

The multi-level structure of small, fast memory caches integrated into a CPU. Each level balances latency and size:

• L1 Cache: Fastest, smallest (e.g., 64KB per core), split into instruction and data caches.
• L2 Cache: Larger and slower than L1 (e.g., 512KB per core), often shared between cores on a single chip.
• L3 Cache: Largest and slowest shared cache (e.g., 32MB), shared across all cores on a CPU die or socket.

This hierarchy exists within a NUMA node. When a core accesses data, it checks L1, then L2, then L3, before finally going to local or remote NUMA memory. Efficient cache usage minimizes costly remote NUMA accesses.

A critical performance principle stating that memory accesses tend to cluster. There are two main types:

• Temporal Locality: Recently accessed data is likely to be accessed again soon. Exploited by caching.
• Spatial Locality: Data near a recently accessed address is likely to be accessed soon. Exploited by prefetching and fetching memory in blocks (cache lines).

NUMA architectures heavily penalize poor locality. If a thread's data is scattered across remote NUMA nodes, access latency spikes. Software must be designed for NUMA awareness, allocating memory and scheduling threads to maximize access to local memory.

A hardware component that handles memory access requests from the CPU. Its core functions are:

• Virtual-to-Physical Address Translation: Using page tables to map a process's virtual address space to physical RAM.
• Memory Protection: Enforcing access permissions (read/write/execute) to prevent processes from accessing unauthorized memory.
• Cache Control: Interfacing with the cache hierarchy.

The MMU operates transparently to the NUMA architecture. However, the operating system's memory allocator must be NUMA-aware to ensure the physical pages it assigns reside on the local node, which the MMU then maps for the process.

A dynamic storage management technique that automatically moves data between different classes of memory or storage media based on access patterns and policies. Tiers are defined by performance and cost:

• Tier 1 (Hot): Fast, expensive media (e.g., DRAM, Intel Optane Persistent Memory).
• Tier 2 (Warm/Cold): Slower, cheaper media (e.g., NVMe SSDs, SATA SSDs, HDDs).

While NUMA deals with latency differences within the same tier (DRAM), memory tiering deals with latency and performance differences across storage media types. Modern systems often combine both: using NUMA-optimized DRAM as the primary tier, with slower, non-volatile memory as a secondary tier.

A type of CPU instruction that enforces ordering constraints on memory operations issued before and after the barrier. It is crucial for correct execution in multi-threaded and concurrent programming on modern systems, including NUMA.

Why it's critical in NUMA: In a NUMA system, memory accesses to different nodes can complete out of order due to varying latencies. A memory fence ensures that all writes from one thread are visible to other threads in a predictable order, preventing subtle, architecture-dependent bugs. Without proper fencing, data corruption and race conditions can occur, which are harder to debug in a NUMA environment.