Huge Pages

Huge pages are memory pages that are significantly larger than the standard 4KB page size used by most operating systems. Common sizes are 2MB (for x86-64 architectures) and 1GB. By using larger contiguous blocks of memory, the system reduces the number of individual page table entries (PTEs) required to map the virtual address space of a process. This directly reduces pressure on the Translation Lookaside Buffer (TLB), a critical CPU cache that stores recent virtual-to-physical address translations. Fewer TLB misses mean less time spent walking the page table in memory, which translates to lower memory access latency and higher throughput for applications with large memory footprints, such as large language model inference and scientific simulations.

The primary performance benefit of huge pages stems from improving TLB hit rates. The TLB has a limited number of entries. With standard 4KB pages, a process using 1GB of memory requires 262,144 page table entries. A typical TLB might hold only 1,024 entries, causing constant TLB thrashing—where needed translations are constantly evicted. With 2MB huge pages, that same 1GB memory region requires only 512 entries, easily fitting within the TLB. Key impacts:

• Reduced Miss Penalty: A TLB miss triggers a multi-level page table walk, which can take hundreds of CPU cycles.
• Increased Effective Memory Bandwidth: More cycles are spent on actual data movement rather than address translation overhead.
• Predictable Latency: Fewer TLB misses lead to more consistent memory access times, crucial for real-time inference workloads.

On Linux, huge pages are managed via the hugetlbfs filesystem. Administrators pre-allocate a pool of huge pages at boot time or runtime. There are two primary types:

• Transparent Huge Pages (THP): An automatic system where the kernel attempts to coalesce standard 4KB pages into huge pages (typically 2MB) transparently to the application. Enabled via sysfs (/sys/kernel/mm/transparent_hugepage/enabled).
• Explicit Huge Pages: Statically allocated at boot via kernel parameters (e.g., hugepagesz=1G hugepages=16). Applications must explicitly request memory from this pool using mmap() on a hugetlbfs mount point.

For high-performance computing and GPU workloads, explicit huge pages are often preferred because they guarantee availability and eliminate the runtime coalescing overhead and potential latency variability introduced by THP.

Huge pages are critical for GPU-accelerated workloads, especially those using Unified Virtual Memory (UVM) or GPU Direct Storage. When a GPU accesses host memory (system RAM) via a PCIe bus or NVLink, every access requires a virtual-to-physical translation.

• Direct Performance Gain: Using huge pages for host memory buffers accessed by the GPU drastically reduces the number of host-side TLB misses, decreasing the latency of these translations.
• Efficient Page Migration: In UVM systems with demand paging, migrating a 2MB huge page between host and GPU memory is more efficient than migrating 512 individual 4KB pages, reducing migration overhead and page fault handling time.
• NVIDIA Driver Support: The NVIDIA driver can leverage pre-allocated huge pages for its internal buffers and user allocations when applications use APIs like cudaMallocManaged with appropriate hints.

Implementing huge pages requires careful system configuration and involves trade-offs.

Configuration Steps:

1. Allocate Explicit Pages: Set kernel parameters, e.g., default_hugepagesz=1G hugepagesz=1G hugepages=4.
2. Mount hugetlbfs: mount -t hugetlbfs nodev /dev/hugepages.
3. Application Integration: Link with libhugetlbfs or use explicit mmap() calls.

Key Trade-offs:

• Memory Fragmentation: Allocating large, contiguous physical blocks can be challenging on long-running systems, leading to allocation failures.
• Over-commitment Risk: Pre-allocated huge pages are reserved and unavailable for other uses, even if idle, potentially reducing overall system memory flexibility.
• Administrative Overhead: Requires root privileges and upfront capacity planning versus the 'set-and-forget' nature of standard 4KB pages.

Huge pages interact with several other memory optimization techniques:

• Page-Locked (Pinned) Memory: Huge pages are often combined with pinned memory to ensure the large page resides in physical RAM and is not swapped, enabling maximum Direct Memory Access (DMA) bandwidth for GPU transfers.
• Memory Tiering: In systems with High Bandwidth Memory (HBM) and slower DDR, huge pages ensure large, hot working sets stay efficiently mapped in the faster memory tier.
• Non-Uniform Memory Access (NUMA): On multi-socket systems, it's crucial to allocate huge pages from the NUMA node local to the accessing GPU to avoid cross-socket latency.
• Stream-Ordered Memory Allocators: Modern GPU allocators (like cudaMallocAsync) can be more efficient when underlying host allocations use huge pages, reducing per-allocation overhead.

For inference serving, enabling huge pages is a foundational step before applying more advanced techniques like continuous batching or KV cache optimization.

The standard memory page size on most x86-64 Linux systems is 4 Kilobytes (KB). Huge pages are significantly larger, typically 2 Megabytes (MB) or 1 Gigabyte (GB).

• Impact: A single 2MB huge page replaces 512 standard 4KB pages.
• Benefit: This drastically reduces the number of page table entries (PTEs) the system must manage, which in turn increases the Translation Lookaside Buffer (TLB) hit rate. The TLB is a cache for these translations; fewer entries mean more relevant translations stay cached, speeding up virtual-to-physical address lookup.

Huge page support must be enabled and configured in the Linux kernel. Key parameters are set via the /proc filesystem or kernel boot arguments.

• /proc/sys/vm/nr_hugepages: Defines the number of persistent huge pages to statically allocate at boot. Setting this reserves physical memory exclusively for huge pages.
• /proc/sys/vm/nr_overcommit_hugepages: Defines the maximum number of huge pages the kernel can create dynamically on-demand, allowing for flexible allocation beyond the static pool.
• Kernel Boot Arg (hugepagesz=, hugepages=): Parameters like hugepagesz=1G hugepages=16 can be set in the bootloader (e.g., GRUB) to reserve sixteen 1GB huge pages at system startup.

Applications request huge pages through specific system calls and flags.

• mmap() with MAP_HUGETLB: The mmap() system call can be used with the MAP_HUGETLB flag to map files or anonymous memory using huge pages. The page size can be specified via the MAP_HUGE_2MB or MAP_HUGE_1GB flags.
• shmget() with SHM_HUGETLB: The shmget() system call, used for creating shared memory segments, also supports the SHM_HUGETLB flag. This is commonly used by applications like databases and some high-performance computing (HPC) runtimes.
• Transparent Huge Pages (THP): For applications not explicitly coded for huge pages, the kernel can automatically promote standard 4KB pages to huge pages (typically 2MB) in the background via the Transparent Huge Pages feature, controlled via /sys/kernel/mm/transparent_hugepages/enabled.

Linux provides a pseudo-filesystem, hugetlbfs, for easier huge page allocation and management.

• Mounting: mount -t hugetlbfs hugetlbfs /dev/hugepages
• Usage: Applications can create files in this mounted directory. Writing to the file triggers the allocation of huge pages. The memory is backed by the static huge page pool.
• Page Size Specification: Different mount points can be created for different page sizes (e.g., 2MB vs 1GB) using the pagesize mount option: mount -t hugetlbfs -o pagesize=1G none /dev/hugepages_1G.
• Access Control: Standard filesystem permissions apply, allowing for secure multi-tenant allocation.

While beneficial, huge pages introduce specific trade-offs that must be managed.

• Memory Fragmentation: Statically allocated huge pages are non-swappable and can lead to physical memory fragmentation, making it harder for the OS to satisfy large contiguous requests for standard pages.
• Allocation Latency: Dynamic allocation of huge pages (via overcommit) can be slower than standard page allocation due to the need to find a large contiguous physical block.
• Overhead Reduction: The primary performance gain comes from reducing TLB miss penalties. This is most impactful for workloads with large, contiguous memory access patterns, such as large matrix operations in machine learning or in-memory databases.
• Monitoring: Usage can be monitored via /proc/meminfo (fields like HugePages_Total, HugePages_Free, Hugepagesize).

In LLM inference, the Key-Value (KV) Cache for the transformer's attention mechanism can consume tens of gigabytes. This cache is accessed sequentially and repeatedly during token generation.

• Benefit: Using huge pages for the KV cache memory allocation can reduce TLB pressure, decreasing the latency of each attention lookup.
• Implementation: Inference servers like Triton Inference Server or custom serving frameworks can be configured to use mmap with MAP_HUGETLB for their large, persistent memory pools.
• Result: This leads to more predictable and lower latency, especially for long-context prompts, by minimizing a critical memory subsystem bottleneck.

The Translation Lookaside Buffer (TLB) is a small, high-speed cache within a processor's memory management unit (MMU) that stores recent virtual-to-physical address translations. Its purpose is to avoid the slower process of walking the page table in memory for every memory access.

• TLB Miss: When a required translation is not in the TLB, a costly page table walk is triggered.
• TLB Coverage: The amount of physical memory that can be addressed by the translations currently held in the TLB. Larger page sizes (like huge pages) dramatically increase TLB coverage, reducing miss rates.
• Impact on AI: In transformer-based models, the attention mechanism's key-value (KV) cache can span large, contiguous memory regions. Using huge pages for the KV cache improves TLB hit rates, directly reducing memory access latency during inference.

A page table is a data structure used by a computer's operating system to store the mapping between virtual addresses (used by processes) and physical addresses (in RAM). It is a core component of virtual memory systems.

• Page Table Entry (PTE): Each mapping in the table. For a standard 4KB page system addressing 512GB of memory, over 134 million PTEs are required.
• Huge Page Effect: A 2MB huge page replaces 512 standard 4KB page table entries with a single entry. This drastically reduces the size of the page table, lowering memory overhead for the OS and improving the efficiency of page table walks on a TLB miss.
• Multi-Level Page Tables: Modern systems use hierarchical page tables (e.g., 4-level). Huge pages can be entered at higher levels, simplifying the lookup process.

Memory fragmentation is a state where free memory is broken into many small, non-contiguous blocks, preventing the allocation of a large contiguous segment even if the total free memory is sufficient. It is a primary challenge for using huge pages.

• External Fragmentation: Free memory is scattered between allocated blocks. This can prevent the OS from finding a free, contiguous 2MB or 1GB block needed for a huge page.
• Internal Fragmentation: Wasted space within an allocated page. If an application only uses 5KB of a 2MB huge page, the remainder is wasted, reducing efficiency.
• Mitigation for Huge Pages: Techniques include:
  • Memory Pooling: Pre-allocating huge pages at system boot or application start.
  • Transparent Huge Pages (THP): A Linux kernel feature that automatically coalesces adjacent standard pages into huge pages, though it can introduce latency variability.

Non-Uniform Memory Access (NUMA) is a computer memory design for multiprocessor systems where memory access time depends on the memory location relative to the processor. Memory attached directly to a processor's local bus is faster than memory attached to a remote processor's bus.

• NUMA Node: A set of CPU cores and their directly attached, local memory.
• Huge Page Allocation: For optimal performance, huge pages should be allocated from the NUMA node local to the GPU or CPU core that will use them most frequently. Remote accesses incur higher latency and lower bandwidth.
• AI System Impact: In multi-GPU servers, ensuring the CPU-side buffers (e.g., for pre-processing) and the GPU memory itself use locally allocated huge pages is critical for minimizing data transfer stalls.

High Bandwidth Memory (HBM) is a type of stacked memory technology where DRAM dies are vertically integrated with a GPU or CPU using through-silicon vias (TSVs). It provides extremely high bandwidth and improved energy efficiency compared to traditional GDDR memory.

• Stacked Design: Multiple memory dies are stacked on top of a base logic die, connected by a wide, high-speed interface.
• Bandwidth vs. Capacity: HBM offers massive bandwidth (e.g., over 1 TB/s per stack) but typically at lower capacities than GDDR. Efficient use via techniques like huge pages is paramount.
• Synergy with Huge Pages: The immense bandwidth of HBM can only be fully utilized with efficient memory access patterns. Reducing TLB misses via huge pages ensures the memory controller's bandwidth is spent on useful data transfer, not servicing page walk overhead.

Page-locked memory, also known as pinned memory, is host (CPU) memory that is prevented from being paged out to disk by the operating system. This guarantees the physical memory address remains fixed, enabling high-bandwidth Direct Memory Access (DMA) transfers to and from GPU device memory.

• DMA Requirement: For a GPU to directly copy data from host memory, the physical pages must not move. Pinning ensures this stability.
• Relationship to Huge Pages: While distinct concepts, they are often used together for peak I/O performance. Pinning a large buffer composed of huge pages minimizes both TLB pressure and OS paging overhead during sustained data transfers.
• Use Case: The input/output pipelines for batched inference servers often use pinned, huge-page memory for the host-side tensors to achieve maximum PCIe transfer throughput to the GPU.