NUMA Alignment

In a Non-Uniform Memory Access architecture, each CPU socket owns local memory banks. Accessing local memory occurs at full speed, but accessing remote memory attached to another socket incurs a significant latency penalty—often 50-100% slower. For GPU-attached processes like data loaders or inference preprocessors, a thread running on socket 0 but accessing memory allocated on socket 1 forces every read across the inter-socket interconnect (UPI/Infinity Fabric) , creating a bottleneck that starves the GPU of data.

Determining which NUMA node a GPU is attached to requires querying the system's PCIe topology. The command nvidia-smi topo -m displays the affinity matrix between GPUs, NICs, and NUMA nodes. A GPU connected via the CPU's integrated PCIe controller will show PIX (PCIe internal) affinity to its local NUMA node. Misalignment occurs when a workload's CPU threads are scheduled on a different NUMA node than the GPU's attached node, forcing all GPU Direct RDMA transfers and command submissions to traverse the inter-socket link.

Linux kernels with automatic NUMA balancing enabled attempt to migrate pages closer to accessing threads. However, for GPU workloads, this can be actively harmful:

• The kernel may migrate memory pages to a remote node based on transient access patterns
• GPU-driven DMA transfers are invisible to the CPU's page access tracking
• Balancing scans consume CPU cycles and memory bandwidth

Best practice is to disable automatic NUMA balancing (echo 0 > /proc/sys/kernel/numa_balancing) on GPU compute nodes and rely on explicit numactl binding.

In multi-GPU systems, each GPU should be paired with a dedicated CPU process bound to its local NUMA node. Common patterns include:

• One process per GPU: Launch N processes, each bound with numactl --cpunodebind=X --membind=X where X matches the GPU's NUMA node
• Thread affinity within a process: Use pthread_setaffinity_np() or taskset to pin specific threads to cores on the correct node
• CUDA IPC and NUMA: When using CUDA Inter-Process Communication for multi-GPU sharing, ensure the producer and consumer processes reside on the same NUMA node to avoid inter-socket IPC overhead

Beyond numactl, applications can enforce alignment programmatically:

• libnuma API: Provides numa_alloc_onnode() and numa_run_on_node() for fine-grained control
• CUDA managed memory: cudaMallocManaged() with cudaMemAdviseSetPreferredLocation can hint the GPU's local NUMA node
• Huge pages: Combining NUMA binding with 1GB huge pages (hugetlbfs) ensures large contiguous memory regions stay local, reducing TLB misses during high-throughput GPU data transfers

Failure to align huge pages results in the same remote access penalty amplified across larger memory regions.

A BIOS-configurable NUMA memory policy that distributes memory allocations in a round-robin fashion across all NUMA nodes. While memory interleaving increases average latency, it prevents bandwidth saturation on a single memory controller.

• Contrasts with NUMA alignment's goal of strict locality
• Useful when a workload's memory access pattern is unpredictable or uniformly distributed
• Often disabled for GPU servers to enforce explicit numactl-based locality control

The practice of mapping the physical PCIe bus hierarchy to NUMA domains to ensure GPU accelerators are used by CPUs on the same socket. PCIe topology awareness requires understanding which root complex a GPU is attached to.

• lspci -vt displays the tree of PCIe bridges and devices
• NVIDIA's nvidia-smi topo -m shows the NUMA affinity of each GPU
• Misalignment forces GPU DMA transfers to traverse the inter-socket QPI/UPI link, introducing latency and bandwidth bottlenecks