Demand Paging

The core mechanism where data pages are transferred from a backing store (system RAM or SSD) into GPU device memory only when a kernel attempts to access an address that is not currently resident. This access attempt triggers a GPU page fault, which halts the executing thread and initiates a page migration operation to fetch the required data before resuming execution. This lazy loading principle is what defines 'demand' paging.

A key benefit is that demand paging operates transparently beneath the CUDA Unified Memory programming model. Developers can write code using single pointers (e.g., allocated with cudaMallocManaged) that are accessible from both CPU and GPU. The memory management unit (MMU) and driver handle all migration logic, freeing the programmer from manually staging data via cudaMemcpy. This simplifies code but requires an understanding of the performance implications of page faults.

Demand paging is the essential enabler for memory overcommit (oversubscription). It allows an application to allocate a total working set larger than the physical GPU memory (VRAM) capacity. The system relies on host memory and even storage (via technologies like GPU Direct Storage) as a slower, larger backing tier. Only the actively accessed 'hot' pages reside in fast GPU memory, while 'cold' pages remain in slower tiers, effectively creating a large virtual address space.

The technique introduces a fundamental trade-off:

• Benefit: Maximizes GPU memory utilization by keeping only needed data resident, allowing more or larger models to run concurrently.
• Cost: Page fault latency. Handling a fault involves significant overhead: interrupting execution, migrating data over the PCIe bus (or NVLink), and resuming. Excessive faulting can devastate throughput, turning a compute-bound kernel into a memory-bound or I/O-bound operation. Performance optimization focuses on minimizing faults through access pattern optimization and prefetching hints.

Demand paging is a runtime behavior of a Unified Virtual Memory (UVM) system. UVM establishes a single virtual address space across CPU and GPU. Demand paging is the dynamic data movement policy within that space. Other UVM features, like CPU-initiated page faults and memory prefetching APIs (cudaMemPrefetchAsync), work in concert with demand paging to optimize data locality and hide migration latency.

This characteristic is pivotal for deploying large language models (LLMs) where the model weights exceed GPU memory. With demand paging, weights can be staged in CPU RAM or NVMe storage. During inference, the attention mechanism generates a sequential access pattern across layers and the KV cache. While naive demand paging would cause severe stuttering, it is combined with predictive prefetching algorithms that proactively load the weights for the next layer or sequence position, overlapping compute with data migration to sustain high throughput.

A foundational technique that enables efficient demand paging by ensuring host memory is ready for rapid transfer.

• Pinned (Page-Locked) Memory: Host memory allocated with cudaHostAlloc or hipHostMalloc is prevented from being paged to disk by the OS, guaranteeing a fixed physical address.
• Direct Memory Access (DMA): The GPU's DMA engine can directly read from/write to this pinned memory without CPU intervention, which is essential for fast page migration during a fault.
• Zero-Copy Fallback: In some implementations, the GPU can directly access pinned host memory (a 'zero-copy' transfer) as a temporary measure while the target page is being migrated to device memory.

The software stack that orchestrates demand paging, sitting between the application and the kernel-mode driver.

• Virtual Address Management: The runtime (e.g., libcuda.so, libhip.so) manages the GPU's page tables, mapping virtual addresses to physical pages in GPU memory, system memory, or indicating 'not present'.
• Fault Service Routine: When the GPU MMU raises a page fault, the user-space driver's fault handler is invoked. It identifies the required page, schedules a DMA transfer via the kernel driver, and updates the page table.
• Concurrency & Streams: Fault handling is designed to be asynchronous and non-blocking for other work on the GPU, allowing computation to continue on different streams while a fault is serviced.

A memory management architecture that creates a single, contiguous virtual address space shared between a CPU and GPU. This allows both processors to access the same memory pointers using a unified page table, simplifying data sharing and enabling demand paging by providing a coherent framework for page migration between host and device memory.

• Key Mechanism: A single virtual address for a data object can be accessed by code running on either processor.
• Enables Demand Paging: The unified address space is the prerequisite for the GPU's memory management unit to trigger page faults and migrate data on-demand.
• Example: NVIDIA's CUDA UVM allows cudaMallocManaged to allocate memory accessible from both CPU and GPU code.

An exception raised when a GPU thread attempts to access a virtual memory address whose corresponding physical page is not resident in GPU device memory. This hardware event is the critical trigger for demand paging.

• Fault Handler: The fault is handled by the GPU's Memory Management Unit (MMU) in conjunction with the driver, which initiates a page migration.
• Migration Source: The required page is retrieved from its backing store, which could be system memory (RAM) or even storage (NVMe SSD) in advanced systems.
• Performance Impact: While enabling overcommit, excessive page faults lead to stall cycles, harming latency and throughput.

The process of transparently moving a memory page between different tiers of a memory hierarchy in response to access patterns or capacity constraints. It is the core action executed after a page fault in a demand paging system.

• Direction: Pages migrate to the GPU on a read fault and may be migrated back to host memory if the GPU memory is under pressure (eviction).
• Granularity: Typically occurs at the system page size (e.g., 4KB, 2MB with huge pages).
• Managed by: The GPU driver and hardware MMU, often using a least-recently-used (LRU) or similar policy for eviction decisions.

A technique where the total memory allocated for active GPU workloads exceeds the physical capacity of the GPU's device memory. Demand paging is the primary mechanism that makes overcommit feasible by using system memory and storage as a backing store.

• Use Case: Allows running larger models or batch sizes than would physically fit in GPU RAM.
• Trade-off: Introduces potential latency variability due to page fault handling. Performance depends heavily on access locality.
• System Requirement: Requires a unified memory architecture (like UVM) and sufficient host memory or fast storage to act as swap space.

Host (CPU) system memory that is prevented from being paged out to disk by the operating system. This guarantees the physical memory address remains fixed, enabling high-bandwidth Direct Memory Access (DMA) transfers to/from the GPU.

• Contrast with Demand Paging: Pinned memory is used for explicit, asynchronous copies (e.g., cudaMemcpyAsync). Demand paging enables implicit, on-demand migration.
• Performance: Essential for achieving peak PCIe transfer bandwidth. Critical for prefetching strategies that work alongside demand paging.
• Drawback: Excessive pinning can reduce available system memory for other processes and increase OS memory pressure.

A technology that enables a GPU to directly access data from storage devices (e.g., NVMe SSDs), bypassing the CPU and host memory buffer. This creates a direct data path for demand paging when the backing store is fast storage.

• Architecture: Allows the GPU's DMA engines to read/write directly to NVMe drives using the PCIe bus.
• Impact on Demand Paging: Dramatically reduces I/O latency and CPU overhead when migrating pages from SSD to GPU memory, making storage-backed oversubscription more viable.
• Example: An inference server can hold a 100GB model checkpoint on SSD, with GDS enabling fast on-demand loading of active layers to GPU memory.