Speculative Decoding

The technique relies on a draft-target model pair. The draft model is a smaller, faster model (e.g., a distilled version of the target) that proposes a sequence of future tokens. The target model is the original, larger, and more accurate model that verifies the draft's proposals in a single, parallel forward pass. This decouples the speed of proposal from the quality of verification.

Instead of generating tokens one-by-one, the target model performs a single forward pass on the draft's proposed sequence. It uses the Jacobian (full output distribution) to check each token in parallel. For each position, it compares the draft's proposed token against the target's predicted distribution. If the draft token matches the target's most likely token, it is accepted. This parallel check is the core mechanism for latency reduction.

A probabilistic algorithm determines which draft tokens to accept.

• Acceptance: If the draft token matches the target's sampled token (based on the target's distribution), it is kept.
• Rejection: On the first mismatch, the draft token is rejected. The target model's distribution at that position is then used to sample a corrected token. All subsequent draft tokens in that sequence are discarded, and generation continues from the corrected point. This ensures the output distribution is identical to the target model's autoregressive output.

The primary benefit is a reduction in time-to-first-token (TTFT) and time-per-output-token (TPOT). Speedup occurs because:

• The draft model's rapid proposals hide the latency of the target model's larger forward pass.
• Multiple tokens are verified in parallel, amortizing the cost of the target model's single pass.
• In practice, speculative decoding can achieve 2x-3x latency reduction for models like Llama 2 and GPT-3, with the exact gain depending on the draft model's quality and the task's predictability.

A critical guarantee of the algorithm is that the final generated text has an identical probability distribution to text generated autoregressively by the target model alone. This is not an approximation; it is mathematically proven by the acceptance algorithm. The draft model only influences the speed of generation, not the quality or statistical properties of the output. This makes it a lossless acceleration technique.

Effective deployment requires careful engineering:

• Draft Model Selection: The draft model must be small (e.g., 7B parameter model drafting for a 70B target) and have a high token acceptance rate. A poor draft model leads to frequent rejections, negating the speedup.
• Memory Footprint: Both models must be loaded into GPU memory, increasing static memory requirements.
• Kernel Optimization: Custom kernels are needed to efficiently manage the two-model workflow and the parallel verification step. Frameworks like vLLM and TensorRT-LLM have integrated support for speculative decoding.

A fundamental optimization that stores the computed Key and Value tensors for previously processed tokens during autoregressive generation. This eliminates the need to recompute attention for the entire prompt and prior context on each new token generation step.

• Direct Relationship to Speculative Decoding: Both the draft and target models in speculative decoding heavily rely on efficient KV caching to minimize per-token latency. The target model's verification step is parallelized partly due to cached keys and values.
• Performance Impact: Can reduce inference latency by 10x or more for long generation sequences.
• Memory Challenge: The KV cache is often the memory bottleneck for long-context models, leading to techniques like PagedAttention.

An inference optimization where 'exit classifiers' are placed at intermediate layers of a neural network. If the classifier at an early layer is confident enough in its prediction, the inference halts, and the result is returned without traversing the full model depth.

• Contrast with Speculative Decoding: Both are 'adaptive computation' methods. Early exiting adapts depth (number of layers computed), while speculative decoding adapts width (using a draft model to propose multiple tokens for parallel verification).
• Best For: Classification tasks or generations where difficulty can be assessed early (e.g., simple factual queries).
• Limitation: Less effective for open-ended, creative text generation where later layers are crucial for coherence.

A neural network architecture where a routing network dynamically selects a small subset of specialized 'expert' sub-networks for each input token. This enables massive model capacity (e.g., trillion parameters) with a manageable computational cost per token.

• Shared Principle of Sparse Activation: Both MoE and speculative decoding are forms of conditional computation. MoE activates different parameters per token; speculative decoding conditionally uses a draft model.
• Performance Characteristic: MoE reduces FLOPs per token, while speculative decoding reduces latency per sequence by generating multiple tokens in parallel.
• Infrastructure Need: Both techniques add complexity to inference serving systems (e.g., managing expert routing or draft/target model coordination).