Speculative Decoding

The core mechanism relies on a dual-model setup: a small, fast draft model (often 10-50x smaller) generates a sequence of K candidate tokens sequentially. The larger target model then processes these K tokens in a single parallel forward pass to verify their correctness against its own probability distribution. This replaces the standard sequential token-by-token generation of the large model, converting a memory-bandwidth-bound process into a compute-bound one.

Verification uses rejection sampling to guarantee the output distribution is identical to the target model's. The target model computes probabilities for each draft token. A token is accepted if its draft probability exceeds the target probability, or via a random acceptance criterion when the target probability is higher. Upon the first rejection, the target model generates a corrected token, and all subsequent draft tokens are discarded. This ensures zero statistical degradation in output quality.

Standard autoregressive decoding is memory-bandwidth-bound: each token generation requires loading the full model weights from GPU HBM to SRAM, but performs minimal computation. Speculative decoding amortizes this cost by performing a single weight load for K tokens. This directly addresses the primary bottleneck in LLM inference, achieving speedups of 2x-3x in typical deployments without any model compression or quantization.

Optimal draft models balance three factors:

• High acceptance rates: The draft must closely approximate the target's distribution. Models from the same family (e.g., a 0.5B parameter draft for a 7B target) work best.
• Low inference latency: The draft must generate K tokens faster than the target would generate 1 token.
• Speculative lookahead (K): Typical K values range from 3-8 tokens. Larger K increases potential speedup but reduces acceptance probability for later tokens.

Advanced implementations extend beyond linear speculation using tree attention. Instead of a single sequence of K tokens, the draft model proposes a tree of multiple candidate continuations at each step. The target model verifies all branches simultaneously using a specialized attention mask that prevents cross-branch contamination. This increases the expected number of accepted tokens per forward pass, pushing speedups beyond 3.5x.

Efficient KV cache handling is critical. During verification, the target model computes keys and values for all K draft tokens in parallel. Accepted tokens' KV entries are appended to the persistent cache. Rejected tokens' entries are discarded. PagedAttention implementations like vLLM integrate speculative decoding natively, managing memory at the block level to avoid fragmentation from the frequent cache extensions and truncations inherent in the rejection sampling process.

A small, fast language model that generates candidate token sequences for verification. The draft model must be significantly smaller than the target model—typically 10-100x fewer parameters—to provide a net speedup. Common choices include distilled models or shallow transformer variants trained on the same tokenizer. The draft model's predictions don't need to be perfectly accurate; even modest acceptance rates yield substantial wall-clock improvements because verification is parallelized.

The large, high-quality model that verifies draft tokens in a single forward pass. Unlike autoregressive generation where each token requires a separate pass, the target model processes all draft tokens simultaneously using masked self-attention. It computes log probabilities for each position and compares them against the draft's predictions using a strict acceptance criterion—typically rejecting tokens where the target's probability diverges from the draft's.

The fraction of draft tokens that survive verification by the target model. A higher acceptance rate directly correlates with greater speedup. Key factors influencing acceptance:

• Distributional alignment between draft and target models
• Temperature settings during sampling (lower temperatures increase acceptance)
• Draft length K—longer drafts risk more rejections Typical acceptance rates range from 60-85% in production deployments, yielding 2-3x latency reductions.

Speculative decoding introduces complexity to key-value cache handling. The draft model maintains its own KV cache during autoregressive generation, but upon verification, only the accepted tokens' KV entries are transferred to the target model's cache. Rejected tokens require cache rollback—discarding invalidated entries. Efficient implementations use tree-structured attention masks to verify multiple candidate sequences simultaneously without duplicating KV storage.

An alternative to using a separate draft model. Medusa attaches multiple lightweight prediction heads directly to the target model's final hidden states, enabling parallel prediction of several future tokens in a single forward pass. Unlike traditional speculative decoding, Medusa requires no separate draft model, simplifying deployment. The technique uses tree attention to verify multiple candidate continuations simultaneously, achieving comparable speedups with reduced architectural complexity.

A parallel decoding strategy that generates multiple tokens per step using Jacobi iteration rather than a draft model. The method exploits the observation that transformer outputs converge to fixed points under repeated application. By feeding model outputs back as inputs and extracting converged tokens, lookahead decoding achieves 1.5-2x speedups without training auxiliary models. It integrates naturally with existing sampling strategies and requires no architectural changes.