Candidate Sequence

A candidate sequence is generated with a predetermined length, known as the speculative factor (γ). This is a key hyperparameter, typically ranging from 3 to 10 tokens. The draft model generates exactly γ tokens in a single, fast autoregressive pass. This fixed length allows the target model's verification forward pass to be perfectly batched, enabling parallel scoring of all γ candidates against the model's own predictions.

The sequence is generated autoregressively by the draft model, meaning each token is predicted based on the original context and all previously generated candidate tokens. Formally, for a context x, the candidate sequence (y1, y2, ..., yγ) is sampled from the draft model's distribution: P_draft(y1 | x) * P_draft(y2 | x, y1) * ... * P_draft(yγ | x, y1, ..., yγ-1). This chain rule of probability is fundamental to its structure.

The core purpose of the candidate sequence is to be verified in parallel by the target model. Instead of running γ separate forward passes, the target model performs one batched forward pass. It takes the context x and the entire candidate sequence as input, using a tree attention or similar mechanism to efficiently compute the target model's probability for each candidate token in its correct position: P_target(yi | x, y1, ..., yi-1) for all i from 1 to γ.

Verification is not token-by-token independent. The target model accepts a prefix of the candidate sequence. It compares its own probability P_target(yi | ...) to the draft's probability P_draft(yi | ...) for each position i, starting from the first token. Acceptance continues sequentially until the first token where P_target(yi) is less than or equal to P_draft(yi). All tokens before this point are accepted; the first mismatched token and all following are rejected. This leads to the rollback mechanism.

The sequence creates computational leverage. The cost of generating γ tokens with the small draft model plus the cost of one parallel verification pass with the large target model must be less than the cost of generating γ tokens autoregressively with the large target model alone. This is the fundamental equation for speedup: Cost(draft γ tokens) + Cost(verify γ tokens) < γ * Cost(target 1 token). The candidate sequence is the vehicle for this batched computation.

The sequence embodies the latency-accuracy tradeoff of speculative decoding. A longer sequence (higher γ) offers greater potential speedup but risks a lower acceptance rate, as the draft model's predictions diverge further from the target's over longer horizons. The sequence's content is a direct product of the draft model's capability; a poorly aligned draft model generates low-quality candidates that are frequently rejected, negating the speedup benefit. Techniques like draft model distillation aim to improve candidate sequence quality.

A draft model is the smaller, faster language model responsible for generating the candidate sequence. Its sole purpose is to propose a short series of likely future tokens (the speculative factor, γ) to accelerate the larger target model. Key characteristics include:

• Speed: Must be significantly faster than the target model to offset verification overhead.
• Alignment: Often distilled from the target model to maximize token acceptance rate.
• Architecture: Can be a smaller transformer, an n-gram model, or auxiliary heads (like Medusa) attached to the target model itself.

The target model is the primary, larger, and more accurate language model that verifies the candidate sequence. It performs a single, batched forward pass to score all draft tokens against its own predictions. Its role is twofold:

• Verifier: Accepts draft tokens that match its predicted distribution.
• Corrector: Rejects incorrect draft tokens and generates the correct token autoregressively from that point. The computational verification cost of this parallel pass must be less than the cost of generating the same number of tokens autoregressively for a net speedup.

Token verification is the deterministic process where the target model validates the candidate sequence. In a single verification forward pass, the target model computes probability distributions for all positions in the sequence. The standard algorithm is:

• For each position i, if the target model's probability for the draft token is greater than a random threshold, the token is accepted.
• The first token where this condition fails is rejected.
• All subsequent tokens in the candidate sequence are discarded, triggering a rollback mechanism where generation continues autoregressively from the target model's corrected output.

The acceptance rate is the critical metric determining the efficacy of speculative decoding. It measures the percentage of tokens proposed by the draft model that are accepted by the target model. Its impact is direct:

• A high acceptance rate (e.g., >80%) means most draft tokens are correct, leading to a high speedup factor.
• A low acceptance rate increases rollbacks, diminishing returns as verification cost is wasted on incorrect candidates. The rate depends on draft model quality, task difficulty, and the chosen speculative factor (γ).

The speculative factor, often denoted as gamma (γ), is the fixed length of the candidate sequence generated by the draft model. It represents a fundamental trade-off:

• Small γ (e.g., 3-5): Lower risk, higher likelihood of acceptance, but limited potential speedup per verification pass.
• Large γ (e.g., 5-10): Higher potential speedup, but acceptance rate typically drops, increasing wasted verification compute. Optimal γ is determined empirically, balancing the draft model's speed, the target model's verification cost, and the latency-accuracy tradeoff.

The rollback mechanism is the failure-handling protocol in speculative decoding. When the target model rejects a token in the candidate sequence, the system must recover correctly. The process is:

1. Generation reverts to the last accepted token position.
2. The target model's own predicted token (sampled from its distribution at the rejection point) is emitted.
3. Standard autoregressive decoding continues from this new point, or a new speculative cycle begins. This mechanism guarantees that the final output is mathematically identical to what the target model would have produced alone, preserving output distribution.