Acceptance Rate

The acceptance rate is fundamentally limited by the distributional alignment between the draft and target models. A draft model trained via distillation on the target model's outputs typically achieves higher acceptance than a generic small model. Key aspects include:

• Architectural Similarity: Draft models with the same tokenizer and vocabulary alignment prevent mismatches.
• Training Objective: Distillation loss (e.g., KL divergence) directly optimizes for matching the target's next-token probabilities.
• Domain Specificity: A draft model fine-tuned on the target's expected data domain (e.g., code, medical text) will have higher contextual acceptance.

The speculative factor (γ) is the fixed number of draft tokens generated per verification step. It creates a direct trade-off:

• Low γ (e.g., 3-5): Higher probability all tokens are accepted, but less work is offloaded from the target model per step. Optimal for low-confidence contexts.
• High γ (e.g., 8-12): Potential for greater speedup, but the probability of full acceptance drops exponentially. A single rejection wastes the verification cost for all subsequent tokens in the candidate sequence. The optimal γ is found where the expected accepted length maximizes the speedup factor, balancing verification cost against autoregressive cost.

The target model's per-token entropy directly impacts achievable acceptance rate. Drafting is highly effective in low-entropy (predictable) contexts and challenging in high-entropy (creative) ones.

• Low-Entropy Regions: Grammar tokens, common phrases, and deterministic code syntax have high acceptance.
• High-Entropy Regions: Creative writing, reasoning steps, or novel problem-solving have lower acceptance. Techniques like confidence thresholding can adaptively skip drafting when the draft model's probability for a token is below a threshold, preventing low-probability proposals that are likely to be rejected.

The algorithm for verifying the candidate sequence affects the efficiency of accepted tokens.

• Independent Token Acceptance: The standard algorithm where each draft token is accepted if its probability under the target model is greater than a random threshold. Rejection triggers a rollback.
• Early Stopping: The verification forward pass can halt immediately upon encountering the first rejected token, avoiding wasted computation on the rest of the sequence. This reduces verification cost for rejected batches.
• Tree-Based Verification: Advanced methods using tree attention allow verifying multiple candidate branches in parallel, increasing the chance of finding an acceptable continuation.

The realized speedup depends on how well the speculative decoding workflow maps to hardware. Key bottlenecks include:

• Memory Bandwidth: The speculative KV cache must be loaded for verification. If the candidate sequence is long (high γ), memory bandwidth can limit the verification speed.
• Parallel Compute: The target model's verification forward pass must process a batch of γ+1 tokens. Hardware with high parallel throughput (e.g., large GPU matrices) benefits more.
• Kernel Fusion: Optimized kernels for the joint scoring and sampling operation during verification reduce overhead. Hardware-aware speculation tunes γ based on these characteristics.

Static small-big model pairs are suboptimal for varying contexts. Adaptive methods dynamically adjust to maximize acceptance:

• Multiple Draft Models: A system can choose between several draft models (e.g., a 1B parameter model for high-confidence stretches, a 100M parameter model for ultra-fast, simple drafts).
• Self-Speculation: Techniques like Medusa heads or n-gram drafting use the target model itself to draft, ensuring perfect distributional alignment for the draft tokens, often boosting acceptance rate.
• Predictive Scaling: Algorithms can predict contextual entropy and adjust the speculative factor γ dynamically per decoding step.

A draft model is a smaller, faster language model used in speculative decoding to generate a sequence of candidate tokens for verification by a larger target model. Its primary purpose is to propose plausible continuations at a lower computational cost.

• Key Role: Acts as a low-latency proposer to 'guess' the target model's output.
• Design Goal: Maximize the acceptance rate of its proposed tokens to achieve net speedup.
• Common Types: Can be a distilled version of the target model, a smaller general model, or a domain-specific model.

The target model is the primary, larger, and more accurate language model in speculative decoding. It verifies the sequence of tokens proposed by the draft model in a single, parallel forward pass.

• Key Role: Acts as the arbiter of correctness, accepting or rejecting draft tokens.
• Verification Process: Computes its own probability distribution for each position in the candidate sequence.
• Cost Constraint: The time for this verification forward pass must be less than the time to generate the same number of tokens autoregressively for a net speedup.

Token verification is the deterministic process where the target model checks the correctness of a sequence of draft tokens. It is the computational step where the acceptance rate is determined.

• Mechanism: The target model performs a single forward pass on the entire candidate sequence.
• Decision Rule: A draft token is accepted if it matches the token sampled from the target model's probability distribution at that position.
• Parallelism: The core speedup comes from verifying k tokens in the time it would take to generate 1 token autoregressively.

A verification forward pass is the single, batched inference step through the target model that scores all tokens in a speculative candidate sequence against the model's own predictions.

• Input: The shared context plus the k-token candidate sequence from the draft model.
• Output: Probability distributions for each token position in the sequence.
• Performance Critical: This pass's latency directly offsets the gains from drafting. Its efficiency is enabled by operator fusion and a speculative KV cache.

The rollback mechanism is the process triggered when the target model rejects a draft token. Generation reverts to the last accepted position and continues autoregressively from the target model's corrected token.

• Trigger: The first token position where the draft and target model outputs diverge.
• Action: All subsequent tokens in the candidate sequence are discarded.
• Impact on Acceptance Rate: A low acceptance rate triggers frequent rollbacks, diminishing the overall speedup factor. The rollback cost is a key part of the verification cost.

The speedup factor is the ratio of wall-clock time for standard autoregressive decoding to the time for speculative decoding. It is the ultimate metric quantifying inference latency reduction, and it is a direct function of the acceptance rate.

• Formula: Speedup ≈ (Number of Accepted Tokens + 1) / (Time for Draft + Time for Verification).
• Dependency: Increases with higher acceptance rates and lower relative verification costs.
• Theoretical Limit: Bounded by the inverse of the time for a single target model autoregressive step. In practice, speedups of 2-3x are common for well-tuned systems.