Speculative Decoding

Speculative decoding relies on a dual-model setup where a small, fast draft model (often 10-100x smaller than the target) generates k candidate tokens autoregressively. The large target model then processes these candidates in a single parallel forward pass, accepting or rejecting each token based on its own probability distribution. This shifts the workload from sequential to parallel computation, exploiting the fact that verifying multiple tokens is nearly as fast as generating one on modern accelerators.

The verification step uses modified rejection sampling to guarantee that the output distribution exactly matches what the target model would have produced on its own. For each draft token, the target computes its probability and compares it to the draft's probability. If the draft token is too improbable under the target distribution, it is rejected and the target resamples from an adjusted distribution. This ensures zero degradation in output quality while still achieving speedups.

Practical speedups range from 2x to 3x in typical deployments, with theoretical maximums determined by the acceptance rate of the draft model. Key factors influencing performance include:

• Draft model alignment: Higher agreement between draft and target distributions yields higher acceptance rates
• Draft latency ratio: The draft must generate tokens significantly faster than the target's per-token latency
• Speculation length (k): More candidate tokens increase potential speedup but risk wasted computation on rejected sequences

Advanced implementations use tree-structured speculation where the draft model proposes multiple branching sequences simultaneously. The target model verifies all branches in parallel using a specialized tree attention mask that prevents cross-branch contamination. This increases the expected acceptance length by exploring alternative token paths, particularly useful at decision points where the draft model has low confidence.

Some architectures eliminate the need for a separate draft model by skipping layers of the target model itself. Early layers act as the draft, generating candidates using a subset of the full computation, while all layers participate in verification. This approach simplifies deployment by removing the need to maintain and align two distinct models, though it typically yields more modest speedups than a dedicated draft model.

Speculative decoding composes effectively with continuous batching in production serving systems. While one request undergoes draft generation, other requests can utilize the GPU for verification or their own draft phases. This overlapping of draft latency with useful computation from other sequences maximizes overall throughput. Serving frameworks like vLLM have begun incorporating speculative decoding into their scheduling algorithms.

A lightweight, low-latency language model that generates candidate token sequences rapidly. In speculative decoding, the draft model proposes multiple tokens autoregressively, which are then verified in parallel by the larger target model. The draft model is typically 10-100x smaller than the target model, often using the same tokenizer.

• Key property: High acceptance rate by the target model
• Common choices: A shallower version of the target model or a distilled n-gram model
• Trade-off: Faster proposals vs. lower acceptance rates

A memory buffer that stores the computed Key and Value tensors from all previous tokens during autoregressive generation. Without caching, each new token would require recomputing attention over the entire sequence. Speculative decoding leverages the KV cache to verify multiple draft tokens in a single forward pass.

• Eliminates quadratic recomputation of attention
• Memory footprint grows linearly with sequence length
• PagedAttention optimizes KV cache memory management

A serving technique that dynamically appends new generation requests to a running batch rather than waiting for all sequences in the batch to complete. This maximizes GPU utilization by filling idle compute capacity. When combined with speculative decoding, continuous batching ensures the verifier model's parallel forward passes are efficiently packed.

• Eliminates static batch granularity
• Reduces Time To First Token (TTFT) under load
• Implemented in serving engines like vLLM

The large, high-quality language model that performs verification in speculative decoding. It evaluates all draft tokens in a single parallel forward pass, accepting those that match its own distribution and rejecting the first divergent token. The target model guarantees that the final output distribution is identical to standard autoregressive decoding.

• No approximation in output quality
• Forward pass processes multiple draft tokens simultaneously
• Typically a dense transformer with full precision weights

The fraction of draft tokens that the target model accepts during verification. A higher acceptance rate directly translates to greater speedup. Acceptance rates depend on the alignment between the draft and target model distributions.

• Typical range: 70-90% for well-matched draft models
• Rejection mechanism: The first divergent token is resampled from the target's corrected distribution
• Tree-based verification can increase acceptance by exploring multiple candidate sequences

An alternative to using a separate draft model. Medusa attaches multiple lightweight prediction heads directly to the frozen target model, each predicting a token at a future position. This eliminates the need for a second model while enabling parallel candidate generation.

• No separate draft model required
• Heads are fine-tuned while the base model remains frozen
• Can be combined with tree attention for multi-candidate verification