NUMA Affinity

In a Non-Uniform Memory Access (NUMA) architecture, each CPU socket has its own local memory controller. Accessing local memory offers full bandwidth and minimal latency. Accessing remote memory attached to another socket traverses the inter-socket interconnect (e.g., AMD Infinity Fabric, Intel UPI), introducing a significant latency penalty—often 50-100% higher—and reduced bandwidth. NUMA affinity ensures the process's memory allocations are satisfied from the local node.

The Linux kernel's default memory policy is first-touch. A physical memory page is allocated on the NUMA node where the CPU first writes to it. To leverage this without explicit membind, an initialization routine can be run on the target NUMA node that sequentially writes to every element of a large tensor. This ensures the data is physically placed in the local memory banks of the socket that will perform the inference, maximizing bandwidth for subsequent reads.

Modern CPUs maintain cache coherence across sockets using protocols like MESI. When a process bounces between sockets, it generates excessive cross-socket snoop traffic as the new socket's cache controller must probe the old socket's caches for modified data. This consumes QPI/UPI bandwidth and introduces latency. NUMA pinning eliminates this by ensuring all cache lines are owned and modified within a single socket's coherent domain.

For multi-process serving engines like NVIDIA Triton Inference Server, NUMA affinity is applied per model instance. A common pattern is to launch one instance per NUMA node, each bound to its local GPUs and memory. This avoids the catastrophic scenario where a single instance's threads span sockets, causing half of every tensor operation to incur remote memory latency. This design treats each socket as an independent, high-performance compute island.

The latency penalty for ignoring NUMA affinity is non-trivial for real-time inference. Benchmarks show that cross-socket memory access can increase P99 latency by 30-60% for large transformer models. This is because the time to load model weights from remote memory dominates the fast matrix multiplication on the local accelerator. For a Service Level Objective (SLO) of 10ms, a 4ms penalty from remote memory access is an unacceptable degradation.

The practice of binding a process or thread to a specific CPU core using taskset or sched_setaffinity. While NUMA affinity operates at the socket level, CPU pinning provides finer-grained control at the individual core level. Combined, they prevent the OS scheduler from migrating latency-sensitive inference threads across cores or sockets, eliminating cache thrashing and translation lookaside buffer (TLB) invalidation overhead. In Kubernetes, this is configured via the cpuset manager policy in the kubelet.

A shared-memory multiprocessor design where each CPU socket has its own local memory bank with lower latency and higher bandwidth than accessing memory attached to a remote socket. On a dual-socket AMD EPYC or Intel Xeon server, a core accessing local DRAM might see ~100ns latency, while cross-socket access can exceed ~300ns. This asymmetry is the root cause of performance variance that NUMA affinity policies seek to eliminate by colocating compute and data.

A Kubernetes scheduling constraint ensuring a pod's CPU and memory resources are allocated from a single NUMA node. Configured via the Topology Manager with the single-numa-node policy. This is critical for inference workloads where a model's weights must fit entirely within one socket's local memory. If the Topology Manager cannot satisfy the constraint, the pod is admitted with a TopologyAffinityError rather than silently degrading performance through remote memory access.

Memory pages larger than the default 4KB, typically 2MB or 1GB, that reduce TLB misses during model inference. When combined with NUMA affinity, huge pages must be pre-allocated on the target NUMA node to prevent the kernel from faulting in pages from a remote socket. The hugetlbfs mount with NUMA awareness ensures that a model's KV cache and weight tensors reside in pinned, physically contiguous local memory, eliminating both TLB miss penalties and cross-socket traffic simultaneously.

A condition where all memory channels on a NUMA node are fully utilized, causing queuing delays. In model serving, this occurs when multiple inference replicas on the same socket compete for local DRAM bandwidth. NUMA affinity without replica anti-affinity can inadvertently colocate too many workers on a single node, creating a bottleneck. Monitoring tools like perf stat and Intel PCM expose local_dram vs remote_dram counters to diagnose whether saturation is local or cross-socket induced.