Verification Forward Pass

Unlike standard autoregressive decoding, which requires N sequential forward passes for N tokens, the verification forward pass is a single, batched operation. The target model processes the entire candidate sequence (e.g., 3-5 tokens) in one go, scoring each position against its own probability distribution. This batch parallelism is the primary source of latency reduction, provided the cost of this single pass is less than the cost of multiple sequential ones.

The forward pass computes logits for every token position in the candidate sequence simultaneously. For a candidate sequence of length γ (the speculative factor), the model outputs a probability distribution over its vocabulary for positions 1 through γ. This allows for independent acceptance decisions:

• A draft token is accepted if its probability under the target model is greater than a random threshold.
• This parallel scoring replaces γ sequential sampling steps, dramatically reducing computation time.

The target model evaluates tokens conditioned on the full, correct prefix. The input to the verification pass is the sequence: [accepted_context, draft_token_1, draft_token_2, ..., draft_token_γ]. The model computes P(target_token_i | context, draft_1, ..., draft_{i-1}). Crucially, if an early token is rejected, the conditional probabilities for subsequent tokens are invalidated, as they were computed using an incorrect prefix. This necessitates the rollback mechanism to the first rejection.

A major optimization is the reuse of the Key-Value (KV) cache from the draft model's forward pass. The attention keys and values for the candidate sequence can be computed once by the draft model and efficiently transferred to the target model's attention mechanism. This avoids recomputing these expensive tensors, significantly reducing the verification cost. Efficient speculative KV cache management is critical for achieving net latency gains, especially for long sequences.

The verification uses a deterministic, greedy acceptance rule. For each position i, given the target model's probability p for the draft token and a random number r ~ Uniform(0,1), the token is accepted if r < p. If p is high (the draft model was confident and correct), acceptance is likely. This rule ensures the output distribution is identical to standard autoregressive sampling from the target model, preserving quality. The process halts at the first rejection (early stopping).

The speedup factor of speculative decoding is directly governed by the efficiency of this pass. The theoretical speedup is approximately γ / (1 + (C_verification / C_autoregressive)), where C_verification is the cost of the verification forward pass and C_autoregressive is the cost of a single autoregressive step. The goal is to minimize C_verification through batching, KV cache reuse, and hardware-aware implementation so that verifying γ tokens costs less than generating γ tokens sequentially.

An inference acceleration technique where a smaller, faster draft model proposes a sequence of future tokens, which are then validated in parallel by a larger target model in a single verification forward pass. This reduces latency by replacing multiple sequential autoregressive steps with parallel computation.

• Core Mechanism: Parallel verification of a candidate sequence.
• Primary Goal: Reduce wall-clock time per generated token.
• Key Requirement: The cost of verification must be less than the cost of autoregressively generating the same number of tokens.

A smaller, faster language model used to generate a candidate sequence of tokens for speculative decoding. Its purpose is to cheaply predict likely continuations that the target model can verify quickly.

• Characteristics: Typically 10x-100x faster than the target model, with fewer parameters.
• Training: Often distilled from the target model to improve prediction alignment.
• Output: Generates k tokens (the speculative factor) ahead of the current position.

The primary, larger, and more accurate language model that performs the verification forward pass in speculative decoding. It scores the draft model's proposed tokens against its own probability distributions.

• Role: Acts as the authoritative source of truth for token acceptance.
• Operation: Processes the entire candidate sequence in a single, batched forward pass.
• Output: Decides to accept draft tokens or generate its own correction, triggering a rollback mechanism.

The ordered list of k tokens generated by the draft model for verification. This sequence is the input to the target model's verification forward pass.

• Length: Defined by the speculative factor (γ).
• Structure: A linear continuation from the current generation position.
• Verification: The target model evaluates the sequence using tree attention or a similar batched scoring mechanism.

The percentage of tokens proposed by the draft model that are accepted by the target model. This is the primary determinant of the speedup factor achieved by speculative decoding.

• Calculation: (Number of Accepted Tokens) / (Total Proposed Tokens).
• Impact: A low acceptance rate increases verification cost waste. A high rate (e.g., >80%) enables near-ideal speedups.
• Optimization: Improved via draft model distillation or confidence thresholding.

The process triggered when the target model rejects a draft token. Generation reverts to the last accepted position, and the target model generates the correct token autoregressively before speculation resumes.

• Trigger: First rejected token in the candidate sequence.
• Effect: Limits wasted computation via early stopping of the verification pass for the remaining candidate tokens.
• Importance: Ensures the final output distribution is identical to the target model's standard autoregressive output.