Speculative Decoding

A small, lightweight language model that generates candidate token sequences autoregressively at high speed. The draft model typically has 10-100x fewer parameters than the target model and runs on the same or adjacent hardware. Key characteristics:

• Optimized for low latency per token rather than output quality
• Often a distilled version of the target model or a shallower Transformer
• Generates K tokens (typically 3-10) before verification
• Quality gap between draft and target determines acceptance rate

The draft model's predictions must share the same tokenizer vocabulary as the target model to enable direct token-level verification.

The large, high-quality model that verifies draft tokens in a single parallel forward pass. Instead of processing one token at a time, the target model:

• Feeds all K draft tokens through the model simultaneously
• Computes output probabilities for each position
• Accepts draft tokens where the draft and target distributions agree
• Rejects at the first mismatch and generates a corrected token

The verification step guarantees that the output distribution is identical to standard autoregressive decoding from the target model alone. This is the key property that distinguishes speculative decoding from approximation methods.

The fraction of draft tokens that survive target model verification. This metric directly determines the speedup multiplier achieved by speculative decoding.

Factors affecting acceptance rate:

• Model alignment: How closely the draft model's distribution matches the target
• Temperature: Lower sampling temperatures increase acceptance rates
• Domain specificity: Draft models fine-tuned on the target domain perform better
• Draft length K: Longer drafts risk lower cumulative acceptance

Typical acceptance rates range from 60-85% in production deployments. A rate below 50% may negate the overhead benefit of running the draft model.

An alternative to the separate draft model approach where multiple prediction heads are attached directly to the target model's final hidden layer. Each head predicts a token at a future position:

• Head 1 predicts the next token (t+1)
• Head 2 predicts the token after that (t+2)
• And so on for N heads

Advantages over separate draft models:

• No separate model to load or maintain in memory
• Heads are lightweight (single linear layers)
• Can be trained with a simple distillation loss
• Tree attention enables branching verification of multiple candidate sequences

Medusa achieves comparable speedups to traditional speculative decoding while simplifying the serving architecture.

Speculative decoding introduces unique key-value cache management challenges. The KV-cache stores pre-computed attention tensors to avoid redundant computation.

Key considerations:

• The draft model maintains its own KV-cache during autoregressive generation
• The target model's KV-cache must be updated to reflect only accepted tokens
• Rejected tokens require cache rollback to the last accepted position
• Tree attention in Medusa-style approaches requires branching KV-cache structures

Efficient KV-cache management is critical to realizing the latency benefits of speculative decoding. Poor cache handling can consume the speedup gains through memory operations.

A serving optimization that combines speculative decoding with dynamic request scheduling. Instead of processing one sequence at a time, continuous batching:

• Groups multiple concurrent requests into a single forward pass
• Dynamically adds new requests and removes completed ones mid-batch
• Applies speculative decoding independently per sequence
• Maximizes GPU utilization by filling compute capacity

The combination of speculative decoding and continuous batching is the foundation of production systems like vLLM and TensorRT-LLM, enabling throughput improvements of 10-20x over naive serving while maintaining low per-token latency.