Page migration is the process of dynamically transferring memory pages between different physical locations in a memory hierarchy, such as from GPU device memory to CPU host memory or storage, managed transparently by the system's memory management unit (MMU). It is the fundamental engine behind unified virtual memory (UVM) and demand paging, enabling efficient memory overcommit by moving data only when a processor, like a GPU, generates a page fault upon accessing a non-resident address. This allows applications to operate on datasets larger than the available physical GPU memory.
Glossary
Page Migration
Page migration is the process of transparently moving memory pages between different tiers of a memory hierarchy, such as from GPU memory to host memory or storage, in response to access patterns or capacity pressure.
GPU MEMORY OPTIMIZATION
What is Page Migration?
Page migration is a core mechanism in unified memory systems that transparently moves data between different tiers of a memory hierarchy, such as from GPU memory to host RAM or storage, in response to access patterns or capacity pressure.
The process is critical for latency reduction and throughput optimization in data-intensive workloads. When a GPU kernel accesses a page not in its local memory, a fault triggers an asynchronous migration from a slower backing store. Advanced systems employ prefetching and usage hints to proactively migrate pages, minimizing stall time. Technologies like NVIDIA's UVM and AMD's hUMA implement hardware-assisted migration, while memory tiering architectures use access frequency (heat) to automatically promote hot data to faster tiers like HBM and demote cold data to slower, larger pools.
GPU MEMORY OPTIMIZATION
Key Characteristics of Page Migration
Page migration is a core mechanism in unified memory systems, enabling transparent data movement across a memory hierarchy. Its characteristics define performance, transparency, and system behavior.
01
Transparency to Application
A defining feature of page migration is its transparency. The application uses a single, unified virtual address space. The memory management unit (MMU) and driver handle the physical location of data, moving pages between GPU device memory, host memory, and even storage without requiring code changes. This simplifies programming but introduces variable latency.
02
Triggered by Page Faults
Migration is primarily demand-driven. When a GPU thread accesses a virtual address whose page is not resident in GPU memory, a GPU page fault occurs. This fault traps to the driver, which:
- Identifies the page's current location (e.g., host memory).
- Initiates a Direct Memory Access (DMA) transfer to migrate the page to GPU memory.
- Updates the page tables.
- Resumes the faulting thread. This on-demand paging minimizes unnecessary transfers.
03
Granularity and Overhead
Migration operates at the granularity of a memory page, typically 4KB, 64KB, or 2MB (huge pages). Smaller pages reduce transfer waste for sparse access but increase page table and TLB overhead. Larger pages improve TLB coverage and reduce fault handling overhead but can transfer unused data (internal fragmentation). The choice is a critical performance trade-off.
04
Multi-Tier Memory Hierarchy
Page migration enables efficient use of a heterogeneous memory hierarchy. Hot data resides in fast, scarce GPU HBM. Warm data may be in DDR host memory. Cold, unused pages can be evicted to NVMe storage via technologies like GPU Direct Storage. Migration policies (e.g., LRU) automatically promote/demote pages between tiers based on access frequency.
05
Concurrency and Prefetching
To hide migration latency, systems employ:
- Concurrent Transfers: The GPU can continue executing other warps while a page fault is serviced.
- Prefetching: Heuristics predict future accesses (e.g., based on sequential patterns) and migrate pages before a fault occurs. Advanced systems use machine learning models to predict access patterns and optimize prefetch decisions, reducing fault rates.
06
Coherence and Consistency
In systems with Unified Virtual Memory, the hardware must maintain memory coherence between CPU and GPU caches. When a page is migrated, cached copies must be invalidated or updated. Protocols like HMM (Heterogeneous Memory Management) ensure all processors see a consistent view of data. This adds complexity but is essential for correct execution.
GPU MEMORY OPTIMIZATION
Page Migration vs. Related Memory Techniques
A comparison of page migration with other memory management techniques used to optimize data movement and capacity in GPU-accelerated systems.
| Feature / Mechanism | Page Migration | Demand Paging | Zero-Copy Transfers | Memory Overcommit (Oversubscription) |
|---|---|---|---|---|
Primary Purpose | Transparently move pages between memory tiers (e.g., GPU <-> CPU) in a unified address space. | Load data into GPU memory only upon access attempt (page fault). | Enable GPU to access data directly from host memory without a physical copy. | Allow total allocated memory to exceed physical GPU capacity, using system memory/swap as backing store. |
Triggering Event | Access pattern, prefetching hints, or capacity pressure. | GPU page fault on a non-resident address. | Kernel launch with host-memory pointer in a Unified Virtual Memory (UVM) system. | Allocation request that exceeds available GPU memory. |
Data Movement Granularity | Memory page (e.g., 4KB, 2MB, 1GB). | Memory page (e.g., 4KB, 2MB, 1GB). | Variable (byte-addressable via pointers). | Memory page (e.g., 4KB, 2MB, 1GB). |
Explicit Developer Control | Medium (APIs for hints, prefetching, advice). | Low (handled automatically by MMU). | High (requires specific pointer usage and memory types). | Low (managed by driver/runtime, often enabled by environment variable). |
Typical Latency Impact | Medium (on-demand migration adds latency; prefetch can hide it). | High (page fault handling path is expensive). | Low (no copy, but host memory access has higher latency than GPU memory). | Very High (can involve disk I/O if system memory is exhausted). |
Requires Unified Virtual Memory (UVM) | ||||
Requires Page-Locked (Pinned) Host Memory | ||||
Can Utilize GPU Direct Storage (GDS) | ||||
Primary Use Case | Optimizing data locality for iterative access patterns (e.g., training). | Enabling working sets larger than GPU RAM (e.g., large model inference). | Reducing transfer overhead for data used once or host-generated data. | Running multiple memory-intensive jobs concurrently on a single GPU. |
GPU MEMORY OPTIMIZATION
Frequently Asked Questions
Page migration is a core technique for managing memory in high-performance computing and AI inference. These questions address its mechanisms, use cases, and impact on system performance.
Page migration is the process of transparently moving a unit of memory (a page) between different tiers of a memory hierarchy, such as from GPU device memory to host RAM or storage, in response to access patterns or capacity pressure. It operates within a unified virtual memory (UVM) system, where the CPU and GPU share a single virtual address space. When a GPU attempts to access a page not resident in its physical memory, it triggers a GPU page fault. The memory management unit (MMU) then handles the fault by migrating the required page from a slower backing store (like system memory) into fast GPU memory, allowing the computation to proceed. This process is often managed by the driver and hardware, making it largely invisible to the application.
Enabling Efficiency, Speed & Accuracy
Intelligent Analysis, Decision & Execution
We build AI systems for teams that need search across company data, workflow automation across tools, or AI features inside products and internal software.
Talk to Us
Search across company data
Give teams answers from docs, tickets, runbooks, and product data with sources and permissions.
Useful when people spend too long searching or get different answers from different systems.
Enterprise searchRAGPermissions
Read more
Automate internal workflows
Use AI to route work, draft outputs, trigger actions, and keep approvals and logs in place.
Useful when repetitive work moves across multiple tools and teams.
AI agentsWorkflow automationGovernance
Read more
Add AI to products and internal tools
Build assistants, guided actions, or decision support into the software your team or customers already use.
Useful when AI needs to be part of the product, not a separate tool.
AI integrationDecision supportModel routing
Read more
About the author
Prasad Kumkar
CEO & MD, Inference Systems
Prasad Kumkar is the CEO & MD of Inference Systems and writes about AI systems architecture, LLM infrastructure, model serving, evaluation, and production deployment. Over 5+ years, he has worked across computer vision models, L5 autonomous vehicle systems, and LLM research, with a focus on taking complex AI ideas into real-world engineering systems.
His work and writing cover AI systems, large language models, AI agents, multimodal systems, autonomous systems, inference optimization, RAG, evaluation, and production AI engineering.
LinkedIn
Limited slotsGet a Free AI Consultation
Partnered with leading AI, data, and software stack.
How We Work
Custom AI workflows for your Business
One-fit-all AI don't work for modern businesses. At Inferensys, we aim to understand your business & custom requirements; which we use to define most efficient agentic workflows, the data, and the tools for your business.
01
Review the use case
We understand the task, the users, and where AI can actually help.
Read more02
Pick the right approach
We define what needs search, automation, or product integration.
Read more03
Build the first useful version
We implement the part that proves the value first.
Read more04
Improve from there
We add the checks and visibility needed to keep it useful.
Read moreThe first call is a practical review of your use case and the right next step.
Talk to Us