Decoding Phase

The decoding phase operates autoregressively, meaning each generated output token becomes part of the input for generating the next token. This creates a fundamental sequential dependency where token N cannot be generated until token N-1 is produced. This serial nature prevents the full parallelization possible in the prefill phase, making it inherently slower per token. The process is defined as: P(output_token | prompt, all_previous_tokens).

Unlike the compute-heavy prefill phase, decoding is predominantly memory-bound. Each forward pass to generate a single token involves:

• Loading the model's weights from GPU memory (HBM).
• Loading the growing KV Cache for all previous tokens.
• Performing relatively small matrix operations. The time is dominated by the memory transfer of these large tensors, not the arithmetic itself. This is characterized by low arithmetic intensity (FLOPs per byte of memory access).

The Key-Value (KV) Cache is the critical state maintained during decoding. It stores the intermediate key and value vectors from the attention mechanism for all previously generated tokens, preventing their recomputation. Its size grows linearly with the batch size and sequence length, consuming significant GPU memory. Efficient management of this cache—its allocation, eviction, and memory layout—is paramount for performance. Techniques like paged attention are designed specifically to optimize KV cache usage.

Each token generation requires launching a full set of GPU kernels for a forward pass, but for a very small amount of new computation (one token per sequence). This results in high kernel launch overhead relative to useful work. The small, per-token matrix multiplications often fail to fully utilize the GPU's massive parallel compute units (SMs), leading to low hardware utilization if not carefully batched. This contrasts with the large, efficient matmuls of the prefill phase.

Continuous batching (or iteration-level scheduling) is the essential optimization for this phase. It dynamically groups multiple user requests (sequences) into a single batch for each decoding step. New requests can join the active batch, and finished sequences can exit, without waiting for the entire original batch to complete. This maximizes GPU utilization and throughput by ensuring the GPU is always working on a sufficiently large batch of token generations, amortizing kernel launch and memory access overheads.

For long generations, the time spent in the decoding phase dwarfs the initial prefill time. Latency is determined by: Total Latency = Prefill Time + (Number of Tokens * Per-Token Iteration Time). The per-token iteration time is the key metric for decoding efficiency. Furthermore, tail latency (p95, p99) is heavily influenced by variability in this phase, caused by factors like different sequence lengths in a batch leading to head-of-line blocking in naive batching schemes.