Memory-Bound

The memory wall is the growing performance gap between processor speed and memory access speed. In memory-bound operations, the Arithmetic Logic Units (ALUs) spend most of their time idle, waiting for data to be fetched from main memory. This is quantified by the arithmetic intensity of a kernel: the ratio of FLOPs to bytes of memory accessed. Autoregressive decoding has very low arithmetic intensity, making it fundamentally memory-bound.

Arithmetic Intensity (AI), measured in FLOPs/byte, determines if an operation is compute-bound or memory-bound. The Roofline Model is an analytical tool that plots attainable performance against AI.

• Compute-Bound: High AI. Performance limited by peak FLOPS of the hardware.
• Memory-Bound: Low AI. Performance limited by peak memory bandwidth. For example, a single-token generation step in a large transformer might have an AI of < 1 FLOP/byte, placing it far into the memory-bound region of the roofline.

The decoding phase of LLM inference is the canonical example of a memory-bound workload. After the compute-intensive prefill phase, generation proceeds token-by-token. Each step involves:

• Reading the entire KV cache from memory (size proportional to sequence length and model layers).
• Performing a relatively small amount of computation on the new token.
• Writing updated KV cache states back to memory. The massive, repeated memory access for the KV cache dominates the iteration time, not the matrix multiplies.

Compute-bound operations, like the dense matrix multiplications in the prefill phase or model training, are limited by the FLOP/s capacity of the GPU's tensor cores. Characteristics include:

• High arithmetic intensity.
• Performance scales with more/ faster compute units.
• Adding memory bandwidth provides little benefit. Memory-bound operations show the opposite: adding more compute cores does not improve performance, but faster memory (e.g., HBM3) or reducing the data moved (e.g., via KV cache compression) yields direct gains.

Memory-bound workloads stress different hardware components:

• Memory Bandwidth: The primary bottleneck. Measured in GB/s (e.g., NVIDIA H100: ~3.35 TB/s).
• Cache Hierarchy: Efficient use of L1/L2 cache is critical to reduce trips to main memory.
• Memory Latency: The delay to fetch the first byte. While bandwidth is about volume, latency affects time-to-first-token. Optimizations focus on maximizing bandwidth utilization through coalesced memory accesses, prefetching, and minimizing redundant transfers.

Techniques to alleviate memory-bound bottlenecks:

• KV Cache Quantization: Store cache in INT8 or FP8 to halve or quarter memory traffic.
• Operator Fusion: Combine multiple layers (e.g., attention + MLP) into a single kernel to keep intermediate data in fast registers/cache (kernel fusion).
• Continuous Batching: Maintains high batch size during decoding to amortize memory controller overhead across more sequences.
• Speculative Decoding: Uses a small draft model to propose multiple tokens, which the large model verifies in a single, more compute-bound pass, reducing total memory-bound steps.

A compute-bound operation is one whose execution time is limited by the speed of the processor's arithmetic logic units (ALUs), rather than by memory access speeds. This is the opposite of a memory-bound operation.

• Typical Example: The dense matrix multiplications in a transformer model's feed-forward network or attention prefill phase.
• Optimization Focus: Maximizing FLOP/s utilization through techniques like tensor core usage, operator fusion, and maximizing batch sizes.
• Hardware Indicator: High GPU SM (Streaming Multiprocessor) utilization with relatively lower memory controller activity.

The Key-Value (KV) Cache is a memory structure that stores the computed key and value vectors for previous tokens in a transformer's attention layers during autoregressive decoding.

• Primary Purpose: Avoids recomputing these vectors for every new token, trading compute for memory bandwidth.
• Memory Footprint: Grows linearly with batch size and sequence length, often becoming the dominant consumer of GPU High-Bandwidth Memory (HBM) during decoding.
• Direct Link to Memory-Bound: The iterative reading of the entire KV cache for each new token generation is the quintessential memory-bound operation in LLM inference.

Memory bandwidth is the maximum rate at which data can be read from or written to a memory subsystem, measured in bytes/second (e.g., GB/s or TB/s).

• The Limiting Factor: In a memory-bound operation, performance scales directly with available memory bandwidth, not compute power.
• Hardware Comparison: NVIDIA H100 GPU: ~2 TB/s HBM bandwidth vs. ~67 TFLOPS (FP16) compute. The ratio defines the compute/memory balance.
• Optimization Target: Techniques like continuous batching and KV cache compression aim to reduce the total bytes transferred per token, effectively increasing usable bandwidth.

Arithmetic Intensity (AI) is a key hardware/algorithm metric defined as the number of arithmetic operations performed per byte of data moved from memory (FLOPs/Byte).

• Roofline Model: This model uses AI to predict whether an operation will be compute-bound (high AI) or memory-bound (low AI).
• Autoregressive Decoding: Has very low arithmetic intensity because it performs few FLOPs (a single token forward pass) relative to the massive amount of KV cache data read.
• Design Implication: To improve performance of memory-bound kernels, you must either increase their AI (e.g., via larger effective batch sizes) or reduce the data movement required.

The decoding phase (or generation phase) is the iterative, token-by-token stage of autoregressive inference where the model produces an output sequence.

• Memory-Bound Nature: This phase is predominantly memory-bound due to the need to read the large, growing KV cache for each token generation, while the compute per token is relatively small.
• Contrast with Prefill: Differs from the compute-bound prefill phase, where the entire prompt is processed in parallel with high arithmetic intensity.
• Performance Dynamic: Latency in this phase is largely dictated by memory subsystem speed and the efficiency of caching mechanisms.

High-Bandwidth Memory (HBM) is a type of stacked, high-performance DRAM used in conjunction with GPUs and AI accelerators to provide the extreme bandwidth required for memory-bound workloads.

• Purpose-Built: Designed specifically to alleviate the memory bandwidth bottleneck for compute-intensive processors.
• Key Characteristic: Offers bandwidth an order of magnitude higher than traditional GDDR memory (e.g., 2-3 TB/s vs. ~500 GB/s).
• Capacity Constraint: HBM capacity is often limited (e.g., 80GB on H100) and expensive, making efficient management of the KV cache and model weights critical for supporting large batch sizes and long sequences.