Speedup Factor

The acceptance rate is the single most critical determinant of speedup. It is the probability that a token proposed by the draft model is accepted by the target model. The expected speedup is approximately 1 / (1 - α). For example, an acceptance rate of 0.7 yields a theoretical speedup of up to ~3.3x. This rate depends heavily on the draft model's alignment with the target model's distribution.

The speculative factor is the fixed number of draft tokens generated per verification step. It defines the length of the candidate sequence. A larger γ increases potential speedup but also raises the verification cost and the risk of a low-acceptance sequence causing a rollback. The optimal γ is hardware-dependent, balancing parallel compute capability against memory bandwidth for the KV cache.

Verification cost is the computational overhead of the target model's parallel forward pass to score γ candidate tokens. Speedup is only achieved if the time saved by avoiding γ autoregressive steps exceeds C_ver. This cost is influenced by:

• Model size: Larger target models have higher per-token verification latency.
• Hardware parallelism: Efficiently batching γ tokens is crucial.
• KV cache management: Accessing and updating the speculative KV cache.

The time taken by the draft model to generate the candidate sequence. T_draft must be significantly less than the time the target model would take to generate the same sequence autoregressively (γ * T_target). This is why draft models are typically 10x-100x smaller and faster. Techniques like draft model distillation are used to minimize T_draft while maximizing alignment with the target.

The physical hardware dictates the achievable speedup. Key factors include:

• Parallel compute units: GPUs/TPUs must efficiently execute the batched verification pass.
• Memory bandwidth: The speculative KV cache for γ tokens must be loaded and stored quickly; bandwidth limits can bottleneck verification.
• Kernel fusion: Optimized operator fusion for the verification pass reduces overhead. This leads to hardware-aware speculation, where parameters like γ are tuned for specific accelerators.

Real-world speedup is reduced by system-level latencies not captured in pure computational models:

• Rollback mechanism: Time spent reverting to the last accepted token after a rejection.
• Scheduling overhead: Cost of coordinating draft and target model execution, especially in a model serving architecture with continuous batching.
• Dynamic draft selection: Logic to choose between multiple draft strategies adds minor latency but can improve the effective acceptance rate.

The speculative factor is the fixed number of tokens the draft model generates ahead of the target model in a single iteration of the speculative decoding algorithm. It is a critical hyperparameter that balances potential speedup against the risk of wasted computation from low acceptance rates.

• A higher gamma creates longer candidate sequences, offering greater potential speedup but increasing verification cost.
• The optimal gamma is hardware-dependent, influenced by memory bandwidth and parallel compute capabilities (hardware-aware speculation).
• If the acceptance rate is too low, a high gamma can actually degrade performance due to excessive rollback.

The acceptance rate is the percentage of tokens proposed by the draft model that are accepted by the target model during the verification forward pass. It is the primary determinant of the effective speedup factor.

• A high acceptance rate means the draft model's predictions closely align with the target model's, minimizing rollback events.
• Acceptance rate can be improved via draft model distillation or confidence thresholding, where only high-probability draft tokens are proposed.
• The relationship is non-linear: Speedup ≈ (Acceptance Rate * Speculative Factor) / (Verification Cost).

Verification cost is the computational overhead incurred when the target model scores the entire candidate sequence in a single, batched forward pass. For speculative decoding to provide a net speedup, this cost must be less than the time it would take the target model to generate the same tokens autoregressively.

• This cost is primarily the time for the verification forward pass, which processes the candidate sequence in parallel.
• Efficient speculative KV cache management is crucial to minimize this overhead.
• The cost is compared against the autoregressive cost of generating gamma tokens to calculate the theoretical speedup limit.

The latency-accuracy tradeoff in speculative decoding refers to the balance between the inference speedup achieved and any potential deviation from the target model's exact autoregressive output distribution.

• The goal is to achieve the target model's accuracy with reduced latency. A perfect draft model would produce zero deviation.
• In practice, minor distribution shifts may occur, but the token verification step ensures the final output is mathematically identical to a slow, full autoregressive sample from the target model.
• This tradeoff is managed by the choice of draft model and the acceptance rate; a poor draft forces more frequent corrections from the target model, preserving accuracy but reducing speedup.

The rollback mechanism is the corrective process triggered when the target model rejects a token in the candidate sequence. Generation reverts to the last accepted position, and the target model generates the correct token autoregressively before drafting resumes.

• This mechanism guarantees output equivalence to standard autoregressive decoding from the target model.
• Early stopping during verification halts scoring as soon as a rejection is found, avoiding unnecessary computation on the invalid suffix of the candidate sequence.
• Frequent rollbacks degrade the speedup factor, highlighting the importance of a high acceptance rate.

Throughput improvement measures the increase in tokens generated per second (TPS) achieved by speculative decoding, typically evaluated in a batched inference setting. While the speedup factor often refers to per-request latency reduction, throughput improvement accounts for system-level efficiency gains.

• Speculative decoding enables more efficient batch verification, where candidate sequences from multiple requests are verified in a single forward pass, improving GPU utilization.
• Throughput gains are often more pronounced than latency gains in production serving systems (model serving architectures).
• This metric is central to inference cost optimization, as higher throughput directly reduces the compute cost per token.