Decode Phase

The decode phase operates autoregressively, meaning each generated token becomes the input for predicting the next. For a sequence of length L, this requires L serial forward passes. The model's output logits for the final token are used to sample the next token (e.g., via top-p or temperature sampling). This sequential dependency creates a fundamental latency bottleneck, as each step must wait for the previous one to complete, making techniques like speculative decoding critical for acceleration.

The primary optimization of the decode phase is the use of the KV cache. After the prefill phase computes keys and values for all prompt tokens, the cache is populated. During decoding:

• The query vector is computed only for the new token.
• The key and value vectors for all previous tokens are read from the cache.
• The attention scores (Query • Key^T) are computed, followed by the weighted sum of values. This avoids recomputing keys and values for the entire growing sequence, transforming the compute complexity of attention from O(n²) to O(n) per step, but creates a memory-bound workload where performance is limited by cache read speed.

Decoding is typically memory-bound, not compute-bound. The arithmetic intensity (FLOPs per byte of memory access) is very low because the primary operation is reading the large KV cache for each new token. Performance is limited by the GPU's memory bandwidth (e.g., H100: ~2 TB/s). The latency for a single decoding step can be approximated by: KV_Cache_Size / Memory_Bandwidth. Optimizations focus on:

• Reducing the size of the KV cache via Multi-Query Attention (MQA) or Grouped-Query Attention (GQA).
• Improving memory access patterns via PagedAttention.
• Using lower precision like KV cache quantization (FP8, INT8).

In production serving systems, the decode phase is executed with continuous batching (also known as iterative batching or dynamic batching). Unlike static batching, this allows:

• New requests to join the batch as others are still generating.
• Completed sequences to exit the batch, freeing resources.
• The batch size and composition to change every decoding step. This maximizes GPU utilization by keeping the hardware busy despite the variable and sequential nature of decoding. Efficient schedulers must manage KV cache memory allocation and context swapping across this dynamic request pool.

The maximum sequence length a model can process is defined by its context window, which is physically constrained by GPU memory capacity for the KV cache. For a model with n_layers, n_kv_heads, d_head, and precision b bytes, the cache size per token is: 2 * n_layers * n_kv_heads * d_head * b. Managing long contexts requires techniques like:

• KV Cache Eviction: Removing less important cached tokens (e.g., LRU).
• Sliding Window Attention: Only caching the most recent W tokens.
• KV Cache Offloading: Moving older cache blocks to CPU/NVMe.
• Attention Sinks: As in StreamingLLM, keeping initial tokens to stabilize attention for infinite generation.

Low-level kernel optimizations are essential for decode phase performance. This includes:

• Operator Fusion: Combining the layer normalization, linear projection, and activation functions into a single kernel to reduce memory reads/writes.
• FlashAttention: While primarily for training, its principles inform decode-optimized attention kernels that minimize HBM accesses.
• TensorRT-LLM / vLLM: SDKs that provide highly optimized, fused decoding kernels and efficient PagedAttention memory management.
• Mixed Precision Inference: Using FP16/BF16 for compute while potentially quantizing the KV cache to INT8, requiring efficient dequantization kernels.

The compute-intensive initial stage of transformer inference where the model processes the entire input prompt in parallel. This phase populates the initial KV cache with the key and value tensors for all prompt tokens, enabling the subsequent, efficient autoregressive decoding. It is characterized by high FLOP utilization but occurs only once per sequence.

A memory buffer that stores the computed key (K) and value (V) tensors from previous tokens during the autoregressive decode phase. Its primary function is to eliminate redundant computation: instead of recomputing K and V for all prior tokens on each new decoding step, the model simply reads them from this cache. Its size grows linearly with sequence length and batch size, making it the primary bottleneck for long-context inference.

An inference scheduling paradigm that dynamically groups incoming requests into a shared batch. It is essential for high-throughput decode phases because it allows:

• New requests to join the batch as others are still processing.
• Completed sequences to exit the batch immediately, freeing resources.
• The KV cache for each request to be managed independently within the same batch. This maximizes GPU utilization compared to static batching, where the entire batch must finish before a new one can start.

A memory management algorithm that organizes the KV cache into non-contiguous, fixed-size blocks or 'pages'. Popularized by vLLM, it solves the problem of internal fragmentation caused by variable-length sequences in continuous batching. Key benefits:

• Enables efficient sharing of physical memory across sequences.
• Allows for near-zero waste in KV cache memory.
• Supports advanced features like cache sharing for duplicated prompts and efficient cache eviction.

Attention variants designed to reduce the memory footprint of the KV cache.

• Multi-Query Attention (MQA): All query heads share a single key head and a single value head. This drastically reduces the size of the cached K and V tensors.
• Grouped-Query Attention (GQA): A hybrid where queries are grouped, and each group shares a single key and value head. It offers a tunable trade-off between the cache efficiency of MQA and the model quality of standard Multi-Head Attention. Both are critical for memory-efficient decoding of large models.

An inference acceleration technique that reduces the number of costly autoregressive steps performed by the large target model. A small, fast draft model (or a simpler method) runs ahead during the decode phase to propose a sequence of candidate tokens. The large target model then verifies these tokens in a single, parallel forward pass. If most drafts are accepted, the effective latency per output token is reduced. It directly optimizes the time-to-first-token and throughput of the decode phase.