Memory Tiering

High Bandwidth Memory (HBM) is the fastest tier, physically stacked on the GPU package using through-silicon vias (TSVs). It provides extreme bandwidth (>1 TB/s) and low latency for active tensors and the KV cache during inference.

• Primary Role: Holds the working set—the model weights, activations, and cache data actively being computed.
• Characteristics: Very high cost per GB, limited capacity (tens of GBs).
• Example: NVIDIA H100 GPU features 80GB of HBM2e with 3.35 TB/s bandwidth.

Graphics Double Data Rate (GDDR) memory is the traditional, high-capacity DRAM soldered onto the GPU board. It acts as a secondary, larger pool for less frequently accessed data or for GPUs without HBM.

• Primary Role: Expands the total device memory capacity beyond HBM limits at a lower cost.
• Characteristics: Lower bandwidth and higher latency than HBM, but higher capacity (up to hundreds of GBs).
• Use Case: Storing less-active model parameters or serving as a spillover for large batch inference.

The server's DDR system memory (RAM) is a slower, CPU-accessible tier that serves as a backing store for GPU memory overcommit. Data is swapped via page migration triggered by GPU page faults.

• Primary Role: Holds evicted pages from GPU memory and acts as a staging area for data in transit.
• Characteristics: Accessed over PCIe, leading to significantly higher latency (microseconds vs. nanoseconds).
• Mechanism: Enabled by Unified Virtual Memory (UVM), allowing the GPU to directly address host memory.

NVMe Solid-State Drives represent a high-performance storage tier for cold data. With technologies like GPU Direct Storage (GDS), the GPU can bypass the CPU to fetch data directly, reducing I/O bottlenecks.

• Primary Role: Holds checkpointed models, massive datasets, or infrequently accessed parameters.
• Characteristics: Very high capacity (multiple TBs), but latency measured in milliseconds.
• Optimization: Demand paging can load model layers from SSD to GPU memory just before they are needed for computation.

Remote network-attached storage (NAS) or cloud object stores (e.g., S3) form the coldest, highest-latency tier. Access requires network traversal and often CPU intervention.

• Primary Role: Archival storage for model repositories, training datasets, and logs.
• Characteristics: Latency can be seconds, but cost per GB is extremely low.
• Workflow Integration: Model serving systems pre-fetch from this tier into hotter tiers (SSD or RAM) during initialization or based on predictive loading.

Intelligent software manages data movement across tiers. It uses access heat tracking (frequency, recency) and predictive policies to make promotion/demotion decisions.

• Key Metrics: Monitors page fault rates, access patterns, and tier bandwidth utilization.
• Policies: Can be LRU (Least Recently Used), LFU (Least Frequently Used), or workload-aware.
• Goal: Maximize the residency of the working set in the fastest tiers (HBM/GDDR) to minimize latency, while using cheaper tiers for capacity.

In shared GPU clusters serving diverse models (e.g., text, vision, embedding), memory tiering maximizes aggregate throughput and GPU utilization. The system acts as a global memory manager, automatically tiering models based on real-time request heat:

• Hot Models: Actively serving models remain GPU-resident.
• Idle Models: Models without recent requests are evicted to host memory. This enables dynamic model swapping, allowing the cluster to serve a larger portfolio of models than could fit simultaneously in GPU RAM. It's critical for cost optimization in SaaS inference platforms, turning GPU memory into a cache for a larger model library.

Long-running inference services suffer from memory fragmentation due to variable-sized KV caches and transient allocations. Memory tiering mitigates this by using a unified virtual address space that spans all memory tiers. When fragmentation prevents a large contiguous GPU allocation, the allocator can satisfy the request using non-contiguous pages from slower tiers, with the memory management unit handling the complexity. This increases effective memory utilization and reduces out-of-memory errors, providing more predictable performance for continuous batching workloads with dynamic request sizes.

For full-parameter fine-tuning or training of large models, memory tiering expands effective working memory. The optimizer states, gradients, and model parameters that don't fit in GPU HBM can spill to CPU RAM or NVMe. While this introduces latency, it enables fine-tuning on consumer or lower-tier professional GPUs that would otherwise be impossible. Techniques like unified virtual memory with page migration are used, where pages are promoted to GPU memory when accessed frequently during backpropagation. This trades longer training time for significantly lower hardware capital expenditure.

In multimodal pipelines (e.g., video understanding, medical imaging), the raw input data (videos, high-res images) can be terabytes in size, far exceeding GPU memory. A memory-tiered system streams data directly from NVMe storage to the GPU via GPU Direct Storage, bypassing the CPU and system RAM. The GPU's compute and decode engines process data in chunks, with the tiering manager ensuring the next data block is prefetched into a faster tier (e.g., CPU RAM) before the GPU needs it. This minimizes I/O wait time and keeps the GPU's tensor cores saturated, crucial for achieving high throughput in data-bound workloads.

The core software mechanism that enables lazy, on-demand data movement in a tiered memory system.

• Demand Paging: Data resides in a slower tier (e.g., host RAM) until the GPU attempts to access it. Only then is it paged into fast GPU memory.
• GPU Page Fault: The exception raised when a GPU kernel accesses a virtual address whose physical page is not resident in GPU memory. This fault triggers the page migration engine to fetch the data.
• Analogy: Similar to virtual memory in traditional OSes, but optimized for high-throughput GPU workloads.

A technique where the total memory allocated to active GPU workloads exceeds the physical GPU memory capacity. It relies on system memory or storage as a backing store and is managed automatically by the tiering system via paging.

• Benefit: Allows users to run larger models or batch more requests than would physically fit in GPU RAM.
• Cost: Performance penalty due to page fault latency when accessing data not in the fast tier.
• Use Case: Essential for serving very large language models (e.g., 70B+ parameters) on GPUs with limited HBM.

A memory design where access time depends on the memory location relative to the processor. In a tiered system with CPU host memory and GPU device memory, access is inherently non-uniform.

• Local vs. Remote: GPU access to its own HBM is 'local' (fastest). Access to CPU host memory over PCIe is 'remote' (slower). Access to NVMe over GDS is 'farthest' (slowest).
• System Design Implication: Tiering algorithms must minimize accesses to remote/far tiers by accurately predicting and promoting hot data.
• Contrast with UMA: In a Uniform Memory Access system, all memory is equidistant, which is not the case in CPU-GPU systems.