Memory-Bound Regime

In a memory-bound regime, the arithmetic logic units (ALUs) of the GPU or TPU spend most of their time idle, waiting for data to be fetched from memory. The system's throughput is capped by the rate at which data—primarily model weights and the KV cache—can be transferred from high-bandwidth memory (HBM) to the compute cores. The roofline model is a key analytical tool for identifying this bottleneck, showing that achievable performance flattens as operational intensity decreases.

During the decode phase of autoregressive generation, the dominant computational pattern is reading the entire KV cache for every new token generated. This creates a massive, repetitive memory traffic pattern. The performance is governed by the equation: Time ≈ (Cache Size per Layer * Num Layers) / Memory Bandwidth. As context length grows, this read overhead becomes the overwhelming cost, far exceeding the FLOPs required for the actual matrix multiplications.

Arithmetic intensity, measured in FLOPs per byte of DRAM access, is very low. In the decode phase, for each new token, the system performs a relatively small number of floating-point operations (e.g., attention scoring, feedforward network) on a very large amount of cached data. This low ratio means the hardware's theoretical peak FLOPs cannot be utilized, as the compute cores are perpetually starved for data. Techniques like operator fusion aim to improve this by keeping data on-chip.

Performance scales sub-linearly or even degrades with increased batch size and context length because both increase the total size of the active KV cache. Doubling the batch size typically doubles the required memory bandwidth but does not double throughput, as the memory subsystem becomes saturated. Long contexts exacerbate the problem, making the inference of long documents or conversations particularly expensive and slow.

It is critical to distinguish this from a compute-bound regime.

• Compute-Bound: Performance is limited by ALU speed (e.g., prefill phase, model training). Adding more memory bandwidth yields little benefit.
• Memory-Bound: Performance is limited by memory bandwidth (e.g., decode phase). Adding more ALUs or increasing clock speed yields little benefit. Optimizations must focus on reducing memory traffic through methods like KV cache quantization, MQA/GQA, and PagedAttention.

Effective optimizations directly target memory access patterns and capacity:

• Reduce Cache Size: Use Multi-Query Attention (MQA) or Grouped-Query Attention (GQA).
• Increase Effective Bandwidth: Implement KV cache quantization (e.g., FP8, INT8) to move more data per cycle.
• Improve Locality: Use PagedAttention to eliminate fragmentation and enable efficient continuous batching.
• Offload/Manage: Employ KV cache offloading to CPU/NVMe or cache eviction policies for ultra-long contexts. The goal is to either shrink the working set or use the available bandwidth more efficiently.

The opposite of a memory-bound state. A system is compute-bound when its performance is limited by the speed of the Arithmetic Logic Units (ALUs) or floating-point units, not by memory bandwidth or latency. In this regime, the processor is constantly busy with calculations, and increasing its clock speed directly improves throughput. This is typical during the prefill phase of transformer inference or dense matrix multiplications where data can be kept in fast caches.

The maximum rate at which data can be read from or written to a memory system, measured in gigabytes per second (GB/s). It is the primary bottleneck in a memory-bound regime. For KV cache-heavy workloads, the required bandwidth is determined by: Batch Size * Sequence Length * Model Dimensions * Precision (bytes) / Time. When this demand approaches the hardware's peak bandwidth (e.g., ~1-2 TB/s for HBM on modern GPUs), performance becomes memory-bound. Techniques like KV cache quantization directly reduce bandwidth pressure.

The delay between a memory request and the arrival of the first piece of data, measured in nanoseconds. High latency exacerbates a memory-bound regime by stalling the processor. Accessing the KV cache from different, non-contiguous memory locations (without PagedAttention) can cause high latency. Hardware employs deep cache hierarchies (L1, L2, HBM) to mitigate this. Cache-aware scheduling aims to group requests to improve locality of reference, reducing effective latency.

An analytical performance model used to visualize whether a kernel or algorithm is compute-bound or memory-bound. It plots attainable performance (FLOPS/s) against operational intensity (FLOPS/byte).

• Below the Roofline: Performance is limited by memory bandwidth.
• At the Roofline: Performance is limited by peak compute throughput. Analyzing transformer attention and decode phase kernels with this model clearly shows they become memory-bound at moderate sequence lengths due to the low operational intensity of loading the large KV cache.

A key metric in the Roofline Model, defined as the number of floating-point operations performed per byte of data transferred from main memory (FLOPS/byte). Low arithmetic intensity leads to a memory-bound regime. The decode phase of autoregressive generation has very low arithmetic intensity because it performs few OPs (a single forward pass for one token) relative to the massive amount of KV cache data that must be read. Optimizations like FlashAttention increase arithmetic intensity by fusing operations and reducing HBM traffic.

A related but distinct regime where system performance is limited by input/output operations to persistent storage (e.g., disks, SSDs) or network interfaces. While a memory-bound regime concerns DRAM/GPU HBM, an IO-bound regime involves much slower devices. However, the concepts overlap in techniques like KV cache offloading, where parts of the cache are moved to CPU RAM or NVMe storage. In this hybrid scenario, performance can become simultaneously limited by both GPU memory bandwidth and PCIe/NVMe I/O speeds.