Memory Hierarchy

Registers are the smallest, fastest storage locations, physically located within the processor core. They hold the operands for current CPU/ALU instructions.

• Speed: Access occurs within a single clock cycle (< 1 ns).
• Capacity: Extremely limited, typically 32 to 64 general-purpose registers per core.
• Management: Explicitly addressed by assembly instructions. Compilers perform register allocation to manage this scarce resource.

Caches are small, fast SRAM memories that store copies of frequently used data from main memory. They are organized in levels (L1, L2, L3) with increasing size and latency.

• L1 Cache: Split into instruction cache (I-cache) and data cache (D-cache). ~32-64 KB per core, ~1-4 cycle latency.
• L2 Cache: Often shared between a few cores. ~256 KB - 1 MB, ~10-20 cycle latency.
• L3 Cache (Last-Level Cache): Shared across all CPU cores. ~8-64 MB, ~30-50 cycle latency.
• Function: Exploits temporal and spatial locality to reduce average memory access time.

Main memory, or Random Access Memory (RAM), is the primary, volatile working memory of the system. It holds the operating system, application code, and active data.

• Technology: Dynamic RAM (DRAM), which must be refreshed periodically.
• Capacity: Ranges from gigabytes (GB) to terabytes (TB) in servers.
• Latency: ~100-300 nanoseconds (ns), orders of magnitude slower than cache.
• Bandwidth: Critical for feeding data-hungry processors; enhanced by multi-channel architectures (e.g., Dual/Quad-channel).

Accelerators like NPUs and GPUs have their own dedicated, high-performance memory, physically close to the compute cores to sustain massive parallelism.

• High Bandwidth Memory (HBM): 3D-stacked DRAM providing extreme bandwidth (~1-2 TB/s) in a compact footprint. Common in high-end AI accelerators.
• GDDR6/GDDR6X: Graphics-focused DRAM with high bandwidth, used in many GPUs.
• Role: Stores model weights, activations, and intermediate tensors. Minimizes data transfer latency to processing units.

Non-volatile storage provides persistent, high-capacity storage for the operating system, applications, and long-term data. It sits at the bottom of the active memory hierarchy.

• Solid-State Drives (SSD/NVMe): Use NAND flash memory. Access latencies in microseconds (µs) to milliseconds (ms). NVMe interfaces provide high IOPs.
• Hard Disk Drives (HDD): Magnetic storage with latencies in milliseconds (ms). Used for archival/cold storage.
• Hierarchy Role: Data is paged into main memory (DRAM) as needed by the OS's virtual memory system.

A key level in accelerator architectures (NPUs/GPUs), scratchpad memory is a small, software-managed, on-chip SRAM. It provides predictable, low-latency storage for programmer-placed data.

• vs. Cache: Unlike hardware-managed caches, the programmer or compiler explicitly controls data movement to/from scratchpad.
• Latency: Similar to L1 cache (1-2 cycles).
• Use Case: Critical for optimizing memory access patterns in AI kernels. Used for staging tiles of matrices, shared weights, or intermediate results during computation to minimize accesses to slower global memory.

AI workloads exhibit strong data locality. Temporal locality exists as filter weights and activation maps are reused across multiple operations. Spatial locality exists as computations access contiguous blocks of input data (e.g., pixels in an image patch). A well-designed hierarchy exploits this:

• Small, fast scratchpad memory or L1 caches hold the actively computed tile of data.
• Larger L2/L3 caches store entire feature maps or weight matrices.
• This organization minimizes costly accesses to off-chip DRAM, which can be 100x slower and 10x more energy-intensive than an on-chip SRAM access. Efficient prefetching algorithms further predict and load needed data in advance.

NPUs achieve performance through massive parallelism, executing thousands of operations simultaneously across hundreds of cores. The memory hierarchy must support this concurrency. Banked memory architectures within caches allow multiple cores to access data in parallel without contention. Furthermore, hierarchies enable strategic data reuse. For example, a weight matrix fetched from DRAM into a shared L2 cache can be broadcast and reused by multiple systolic arrays or tensor cores, amortizing the high cost of the initial fetch. This reuse pattern is central to optimizing operations like generalized matrix multiply (GEMM).

Each level in the hierarchy represents a specific trade-off. Registers offer near-zero latency but minimal capacity (kilobytes). Caches provide moderate capacity (megabytes) with low latency. High Bandwidth Memory (HBM) offers very high bandwidth (hundreds of GB/s) and larger capacity (gigabytes) but higher latency than on-chip memory. System DRAM (DDR) offers the largest capacity but lower bandwidth and higher latency. NPU compilers and runtime systems must explicitly manage data placement and movement across these levels—a process called tiling or blocking—to match the algorithm's needs, maximizing bandwidth for data-intensive layers and minimizing latency for critical paths.

Data movement is the dominant consumer of energy in modern AI chips. Moving a 32-bit word from DRAM can consume over 200 pJ, compared to 0.1 pJ for a register access or 10 pJ for an on-chip SRAM access. A deep, well-utilized hierarchy drastically reduces the number of long-distance, high-energy accesses to main memory. By keeping data flowing in the faster, lower-energy levels closest to the compute, the overall energy efficiency (measured in TOPS/W) of the NPU is dramatically improved. This is critical for deployment in edge devices and data centers where power budgets are constrained.

Optimal use of the memory hierarchy requires hardware-software co-design. The NPU hardware provides the hierarchy (registers, scratchpad, caches, controllers). The software stack—compilers (MLIR, LLVM), kernels, and frameworks—must be explicitly aware of it. Techniques include:

• Kernel fusion to keep intermediate results in cache.
• Memory scheduling to overlap computation with data transfer via DMA engines.
• Data layout transformations (NHWC vs. NCHW) to ensure memory access patterns are cache-friendly.
• Using pinned memory for efficient host-to-device transfers. Without this co-design, the hardware potential is left untapped.