Throughput Improvement

The acceptance rate is the percentage of tokens proposed by the draft model that are accepted by the target model during the verification forward pass. It is the primary determinant of speculative decoding's success.

• Calculation: (Number of accepted draft tokens / Total draft tokens proposed) * 100%
• Impact: A higher acceptance rate directly increases the speedup factor by reducing the frequency of costly rollbacks to autoregressive generation.
• Typical Range: Effective speculative decoding systems often achieve acceptance rates between 70-85% for well-aligned small-big model pairs.

The speedup factor is the ratio of the time taken for standard autoregressive decoding to the time taken for speculative decoding, providing a direct measure of latency reduction.

• Formula: Speedup = T_autoregressive / T_speculative
• Dependencies: It is a function of the acceptance rate, the speculative factor (γ), and the relative computational costs of the draft and target models.
• Theoretical Maximum: With a perfect draft model, the speedup is bounded by γ + 1. In practice, speedups of 2x-3x are common for large language models like Llama 2 70B.

Tokens per second (TPS) is the absolute throughput metric, measuring the total number of tokens generated by the system per unit time. This is the ultimate business metric for inference cost.

• Measurement: Should be measured in a batched inference setting under realistic load to account for system overhead.
• Components: TPS is influenced by the speedup factor, the target model's native latency, and the efficiency of the batch verification process.
• Use Case: This is the key metric for CTOs and engineering managers calculating inference cost optimization and infrastructure scaling requirements.

Verification efficiency measures the computational cost of the target model's parallel verification pass relative to the cost of generating the same number of tokens autoregressively.

• Core Principle: For a net speedup, the cost of verifying γ tokens must be less than the cost of generating γ tokens autoregressively.
• Factors: Efficiency is determined by the implementation of tree attention, speculative KV cache reuse, and operator/kernel fusion in the verification step.
• Optimization: High verification efficiency minimizes the verification cost, making speculative decoding viable even with moderate acceptance rates.

Effective batch utilization quantifies how well the system uses parallel compute resources (e.g., GPUs) during the batch verification of candidate sequences from multiple requests.

• Importance: Speculative decoding's throughput gains are maximized in a model serving context with continuous, dynamic request batching.
• Metric: It measures the ratio of active computational units during the verification forward pass versus idle time.
• Relationship: High utilization is enabled by continuous batching systems that can dynamically group requests with varying speculative factors and sequence lengths.

Rollback frequency measures how often the speculative process fails and must revert to standard autoregressive generation from the target model, triggered by a token rejection.

• Inverse Metric: It is directly related to the acceptance rate. A low acceptance rate leads to high rollback frequency.
• Cost: Each rollback incurs a penalty, as the work to draft the rejected token sequence is wasted, and generation continues at the slower autoregressive pace.
• Mitigation: Techniques like confidence thresholding and dynamic draft selection aim to reduce rollback frequency by making the draft model more conservative.

The core mechanism enabling throughput gains. Instead of generating one token at a time (autoregressively), the target model verifies a sequence of candidate tokens in a single, batched forward pass. This amortizes the fixed cost of loading the model's weights and attention states over multiple tokens.

• Key Benefit: Reduces the number of serial model executions.
• Implementation: Requires batching the candidate sequence and using a modified attention mask to allow the target model to score all positions simultaneously.

The percentage of tokens proposed by the draft model that are accepted by the target model. This is the primary determinant of effective throughput improvement.

• High Acceptance Rate (>80%): Means most draft tokens are correct, leading to a near-ideal speedup proportional to the speculative factor.
• Low Acceptance Rate: Causes frequent rollbacks, wasting the compute spent on drafting and verification. The net speedup is calculated as Speedup = 1 / ( (1/γ) + (1 - α) ), where γ is the speculative factor and α is the acceptance rate.

The quantitative measure of throughput improvement. It's the ratio of wall-clock time for standard autoregressive decoding to the time for speculative decoding.

• Theoretical Maximum: Bounded by the speculative factor (γ). If every draft token is accepted, the speedup approaches γ.
• Real-World Range: Typically between 1.5x and 3.5x for well-tuned small-big model pairs, as it must offset the verification cost of the target model's parallel forward pass.
• Measurement: Must be measured in a batched inference setting under realistic loads to account for system overhead.

The computational overhead incurred when the target model scores the candidate sequence. This cost must be less than the time saved by not generating those tokens autoregressively for a net speedup.

• Components: Includes the FLOPs for the extra forward pass and the memory bandwidth cost of handling the speculative KV cache.
• Optimization: Techniques like early stopping (halting verification upon first rejection) and efficient batch verification across multiple requests are used to minimize this cost.

The process of verifying candidate sequences from multiple independent requests or beams in a single forward pass of the target model. This is essential for achieving high aggregate throughput in a production serving system.

• System-Level Throughput: Maximizes GPU utilization by maintaining a high batch size, even when individual requests are at different stages of the speculative decoding loop.
• Challenge: Requires sophisticated scheduling to align the drafting and verification phases across heterogeneous requests without introducing excessive latency.

Tuning speculative decoding parameters based on the underlying hardware's characteristics to maximize throughput.

• Key Parameters: The speculative factor (γ) is often chosen based on memory bandwidth constraints and the target model's parallel compute efficiency.
• Memory Bound vs. Compute Bound: On memory-bound systems, a larger γ may not yield gains due to KV cache overhead. On compute-bound systems, a larger γ can better utilize parallel cores.
• Goal: To find the optimal operating point on the latency-throughput tradeoff curve for a specific deployment hardware (e.g., specific GPU architecture).