Token-Level Acceptance

Unlike sequence-level acceptance, token-level acceptance evaluates each candidate token individually. The target model computes the probability distribution for the token at position t given the accepted prefix. The draft token is accepted if a random sample from this distribution matches the draft token. This allows for partial acceptance of a candidate sequence, where only the first k tokens are kept before a rejection occurs.

Acceptance is not deterministic. For each candidate token x_t, the target model produces a probability distribution P_target(x_t | prefix). A token is accepted if a sample drawn from this distribution equals the draft token x_draft. This preserves the original output distribution of the target model, ensuring no statistical deviation from standard autoregressive generation, which is a critical guarantee for correctness.

• Key Metric: The probability of acceptance for a token is P_target(x_draft | prefix).
• Guarantee: The final output sequence is statistically identical to what the target model would have generated alone.

When a token is rejected, the system rolls back to the last accepted position. The target model then resamples a new token from its own distribution at that position, replacing the rejected draft token. This resampled token becomes the new prefix, and generation continues. This mechanism ensures correctness but introduces a rollback cost, as the computation spent on verifying tokens after the first rejection is wasted.

The effectiveness of token-level acceptance is measured by the acceptance rate. A high rate means fewer rollbacks and a greater speedup factor. The theoretical maximum speedup is limited by the speculative factor (γ) and the acceptance rate (α). The net speedup occurs because verifying γ tokens in one parallel forward pass is faster than generating γ tokens autoregressively, provided the verification cost is less than γ times the cost of a single autoregressive step.

Token-level acceptance is distinct from sequence-level acceptance, where an entire candidate sequence is either fully accepted or fully rejected based on a joint probability score. Token-level acceptance is more fine-grained and typically leads to higher overall efficiency because it can salvage correct prefixes from partially incorrect drafts. It is the standard method used in foundational speculative decoding papers like Fast Inference from Transformers via Speculative Decoding.

Efficient token-level acceptance requires careful KV Cache management. During the draft phase, KV states for the candidate sequence are computed. During the target model's verification forward pass, these speculative KV states can be reused to score the draft tokens in parallel. Upon rejection, the cache must be rolled back to the state at the last accepted token. Optimizing this cache lifecycle is crucial for minimizing the verification cost overhead.

An inference acceleration technique where a smaller, faster draft model proposes a sequence of future tokens, which are then verified in parallel by a larger target model in a single forward pass. This allows the target model to accept multiple tokens per step, bypassing its slower autoregressive generation.

• Core Mechanism: Parallel verification of a candidate sequence.
• Primary Goal: Reduce wall-clock latency of text generation.
• Key Requirement: The verification cost must be less than the cost of generating the same tokens autoregressively.

A smaller, faster language model used to generate a candidate sequence of tokens for speculative verification. Its sole purpose is to predict likely continuations, trading some accuracy for significantly lower latency.

• Characteristics: 10-100x faster than the target model, often a distilled or quantized version.
• Output: A fixed-length sequence of tokens (the speculative factor, γ).
• Optimization Goal: Maximize acceptance rate with the target model to achieve net speedup.

The primary, larger, and more accurate language model whose output distribution is the gold standard. In speculative decoding, it performs a verification forward pass to score the draft model's proposals.

• Role: Acts as the verifier and final arbiter for each token.
• Process: Computes probabilities for the draft tokens and the correct next token in a single, batched forward pass.
• Output: The final, accepted token sequence, which is guaranteed to match the target model's own autoregressive distribution.

The process where the target model checks the correctness of the draft model's proposed sequence. It runs the entire candidate sequence through the target model in parallel, comparing the draft tokens against the target's own predicted probability distribution.

• Mechanism: A single, batched forward pass through the target model.
• Comparison: For each position, check if the draft token matches the target model's most likely token.
• Result: A binary accept/reject decision for each token in the candidate sequence.

The percentage of tokens proposed by the draft model that are accepted by the target model. This is the critical metric determining the efficacy of speculative decoding.

• Direct Impact: Determines the speedup factor. A low acceptance rate leads to frequent rollbacks and diminished gains.
• Typical Range: Varies widely (e.g., 60-85%) based on draft model quality and task.
• Optimization Target: Improved via draft model distillation or confidence thresholding.

The corrective process triggered when the target model rejects a draft token. Generation reverts to the last accepted position, and the target model generates the correct token autoregressively before drafting resumes.

• Trigger: First rejected token in the candidate sequence.
• Action: Discard all subsequent draft tokens, even if they would have been correct.
• Consequence: Introduces overhead, making high acceptance rates essential for net speedup.