GPU Page Faults

A GPU page fault is an exception raised by the Memory Management Unit (MMU) when a GPU thread attempts to access a virtual memory address whose corresponding physical page is not resident in GPU memory. This occurs under Unified Virtual Memory (UVM) or demand paging schemes when the required data resides in host memory or on storage (e.g., NVMe SSD). The fault pauses the thread and initiates a page migration.

Upon a fault, the system's page fault handler is invoked. It orchestrates the migration of the required 4KB (or larger huge page) from its current location (CPU memory or storage) into GPU memory. For GPU Direct Storage (GDS), this can be a direct transfer from NVMe to GPU memory, bypassing the CPU. The resolving thread is stalled until migration completes, creating latency. This process is the foundation of memory overcommit.

Page faults introduce non-deterministic, high latency—orders of magnitude slower than a GPU memory hit (microseconds to milliseconds vs. nanoseconds). This trade-off enables oversubscription, allowing workloads to use more aggregate memory than physically available on the GPU. Performance hinges on locality: frequent faults on hot pages cause thrashing, while infrequent faults on cold data are an acceptable cost for increased capacity.

• Major Fault (Hard Fault): Occurs when the required page is not in any physical RAM (GPU or CPU) and must be read from disk/storage. This has the highest latency.
• Minor Fault (Soft Fault): Occurs when the page is resident in host (CPU) memory but not in GPU memory. Resolution involves a DMA transfer over PCIe or NVLink, which is faster than disk I/O but still significant. Understanding this distinction is key for profiling and optimizing data placement.

To mitigate fault latency, systems use prefetching. The programmer or a runtime can issue prefetch hints (e.g., cudaMemPrefetchAsync) to migrate data to the GPU before it is accessed, converting a future fault into a background transfer. Advanced systems employ access pattern predictors to automate prefetching. Stream-ordered memory allocators also integrate with this mechanism for efficient memory reuse.

Page faults operate within a broader memory hierarchy and tiering strategy. Data can reside in:

• GPU HBM/Global Memory (fastest)
• CPU DDR Memory (slower, accessed via PCIe)
• NVMe Storage (slowest, accessed via GDS) Faults trigger demotion/promotion between these tiers. Page-locked (pinned) host memory is often used as a staging area to accelerate minor faults. Effective use of huge pages can reduce fault frequency by covering larger address ranges per page table entry.

Demand paging is the virtual memory technique implemented by GPU page faults. Data resides in a slower backing store (like system RAM or an SSD) and is only transferred into fast GPU memory when a kernel instruction attempts to access it.

• The initial access attempt causes a page fault.
• The fault handler suspends the executing thread, migrates the required page, and then resumes execution.
• This enables memory overcommit, allowing workloads to use more memory than the GPU physically has, at the cost of potential latency if faulting is frequent.

Page migration is the physical action taken to resolve a GPU page fault. It involves moving a memory page (typically 4KB or 64KB) from one location in the memory hierarchy to another.

• Fault-driven migration: Triggered by an access fault (e.g., GPU needs a CPU-resident page).
• Prefetch migration: Proactively migrates pages based on predicted access patterns to hide latency.
• Eviction migration: Moves pages out of GPU memory to make space, often using an LRU policy. Efficient migration is critical for minimizing the performance penalty of faults.

Memory overcommit is the practice of allocating more virtual memory to active GPU workloads than the available physical GPU memory. It relies on:

• A unified virtual memory system.
• Demand paging and page migration to move data as needed.
• A slower backing store (system memory, NVMe SSD) to hold excess pages. While it enables larger models to run, it introduces latency variability. Performance becomes dependent on the access locality of the workload; thrashing occurs if the working set size exceeds physical capacity, causing continuous page faults.