Memory Hierarchy

The defining trade-off of the hierarchy. Latency is the time to retrieve a single piece of data, while bandwidth is the rate of sustained data transfer.

• Registers: Sub-1 nanosecond latency, terabytes/sec of bandwidth.
• L1/L2 Cache: ~1-10 ns latency, hundreds of GB/s bandwidth.
• Shared Memory: ~10-20 ns latency, similar bandwidth to L1 (but software-managed).
• Global Memory (HBM/GDDR): ~100-300 ns latency, 1-3 TB/s bandwidth (e.g., HBM3).
• Host Memory (DDR): ~100-200 ns latency, ~50-100 GB/s bandwidth.
• Storage (NVMe SSD): ~10-100 microseconds latency, ~5-10 GB/s bandwidth.

The goal is to keep frequently accessed data (hot data) in the fastest tiers.

Capacity scales inversely with speed. Faster memory uses more transistors and advanced packaging, making it significantly more expensive.

• Registers: Kilobytes per SM. Highest cost/bit.
• On-Chip Caches (L1/L2): Megabytes per GPU. High cost/bit.
• Shared Memory: Kilobytes per Thread Block. Software-managed resource.
• High Bandwidth Memory (HBM): Tens of Gigabytes (e.g., 80GB HBM3). High cost/bit, but essential for bandwidth.
• Host DRAM: Hundreds of Gigabytes to Terabytes. Lower cost/bit.
• NVMe Storage: Multiple Terabytes. Lowest cost/bit.

This cost structure forces systems to use small amounts of fast memory as a cache for larger, slower memory.

Performance depends on exploiting two types of locality to minimize accesses to slower memory tiers:

• Temporal Locality: Recently accessed data is likely to be accessed again soon. This justifies caching. Example: The KV Cache in a transformer's attention mechanism is a form of explicit temporal locality management.
• Spatial Locality: Data near recently accessed data is likely to be accessed soon. This justifies fetching contiguous blocks (cache lines). Example: Ensuring GPU thread warps perform coalesced memory accesses to contiguous global memory addresses.

Algorithms and data structures must be designed to maximize both forms of locality.

Different hierarchy levels have distinct management models:

• Hardware-Managed Caches (L1/L2): Transparent to software. Use cache coherency protocols (like MESI) in CPU systems to keep multiple cores' views of memory consistent. GPUs often use simpler, non-coherent caches for scalability.
• Software-Managed Caches (Shared Memory): Programmer explicitly controls data movement (load/store). Offers predictable, low-latency access but requires careful design to avoid bank conflicts.
• Unified Virtual Memory (UVM): Presents a single address space across CPU and GPU memory. Hardware/OS manages page migration and demand paging via GPU page faults, simplifying programming but adding overhead.
• Peer-to-Peer (P2P): GPUs with NVLink can access each other's memory directly, creating a faster tier for multi-GPU communication than going through host memory.

A concrete example using NVIDIA's Hopper architecture:

1. Registers: Fastest, private to each thread.
2. L1 Cache / Shared Memory: 256 KB per Streaming Multiprocessor (SM). Configurable split (e.g., 128 KB L1 / 128 KB Shared).
3. L2 Cache: Unified 50 MB cache shared across all SMs, servicing all memory traffic.
4. HBM2e/HBM3 (Global Memory): 80-140 GB capacity, 2-3 TB/s bandwidth. Connected via ultra-wide memory interfaces.
5. Host Memory (DDR): Accessed via PCIe or NVLink. Acts as a swapping pool for GPU memory oversubscription.
6. NVMe Storage: Accessed via GPU Direct Storage (GDS), bypassing the CPU for direct loads into GPU memory.

Techniques like memory tiering aim to automate optimal data placement across this stack.

Understanding the hierarchy drives key optimization strategies:

• Kernel Design: Structure computations to fit working sets in L1/Shared Memory, use coalesced accesses for global memory.
• Memory Allocation: Use memory pools and stream-ordered allocators to reduce fragmentation and allocation overhead.
• Data Transfer: Minimize host-device transfers, use pinned memory for bandwidth, leverage zero-copy or UVM where appropriate.
• Model Optimization: Quantization and pruning directly reduce the model's footprint, allowing it to fit in faster memory tiers (e.g., keeping a full model in HBM vs. swapping).
• System Configuration: Enabling Huge Pages reduces TLB misses, improving address translation performance for large memory workloads.

Failure to optimize leads to the processor being memory-bound, stalling while waiting for data.

A memory management architecture that creates a single, contiguous virtual address space shared between a CPU and GPU. This allows both processors to access the same memory pointers using a common page table, simplifying data sharing and enabling demand paging and page migration between host and device memory. It abstracts the physical location of data, allowing developers to write code as if all memory is unified, while the system handles the complex movement of pages between memory tiers.

A stacked memory technology where DRAM dies are vertically integrated with a GPU or CPU using through-silicon vias (TSVs). It represents a high-performance tier in the memory hierarchy, sitting physically close to the processor core. Key characteristics include:

• Extremely high bandwidth: Significantly outperforms traditional GDDR memory.
• Improved energy efficiency: Due to shorter physical data paths.
• Smaller physical footprint: Enables more compact accelerator designs. HBM is a critical enabler for the massive parallelism and data throughput required by modern AI workloads.

A system architecture that organizes different physical memory types into a performance and capacity hierarchy. Data is automatically promoted to faster tiers or demoted to slower tiers based on usage frequency ("heat"). A typical tier for AI might be:

• Tier 1 (Fastest/Smallest): GPU HBM or SRAM caches.
• Tier 2: GPU GDDR (Global Memory).
• Tier 3: CPU DDR (System RAM).
• Tier 4 (Slowest/Largest): NVMe storage (leveraging GPU Direct Storage). Tiering, managed by the OS and hardware, is essential for implementing efficient memory overcommit strategies.

Core mechanisms that enable efficient unified virtual memory and memory overcommit.

• Page Migration: The transparent movement of memory pages between tiers (e.g., from system RAM to GPU HBM) in response to access patterns. This is often proactive, based on heuristics.
• Demand Paging: A reactive technique where data is transferred from a slower backing store into faster GPU memory only when a GPU kernel attempts to access it, triggering a GPU page fault. This ensures memory is occupied only by actively used data, maximizing utilization of scarce GPU memory.

An optimal memory access pattern crucial for achieving maximum bandwidth from GPU global memory. It occurs when consecutive threads in a GPU warp access consecutive memory addresses in a single, aligned transaction.

• Optimal Pattern: Thread 0 accesses address A, thread 1 accesses A+4, thread 2 accesses A+8, etc.
• Benefit: The memory controller can service all threads' requests in one or a minimal number of transactions.
• Non-coalesced access scatters requests across many memory transactions, drastically reducing effective bandwidth and increasing latency, becoming a major performance bottleneck.

Shared memory is a programmer-managed, on-chip cache memory on a GPU, shared by threads within a thread block. It offers ultra-low latency and high bandwidth for data reuse and inter-thread communication.

• Bank Architecture: Shared memory is divided into equally sized memory banks (e.g., 32 banks) that can be accessed simultaneously.
• Bank Conflict: A performance penalty that occurs when two or more threads in a warp attempt to access different data words within the same bank in a single cycle. This forces serialized access, reducing throughput. Careful data layout and access patterns are required to avoid conflicts.