Small-Big Model Pair

The draft model is a smaller, faster language model responsible for generating a sequence of candidate future tokens. Its primary objective is speed, not ultimate accuracy. Key characteristics include:

• Architecture: Often a distilled or quantized version of the target model, or a separate, highly optimized model like a small language model (SLM).
• Function: Autoregressively proposes a short sequence (e.g., 3-5 tokens) called the candidate sequence.
• Optimization Goal: Minimize single-token generation latency to keep the drafting phase cheap. Acceptance rate is the critical metric of its alignment with the target model.

The target model is the primary, larger, and more accurate language model whose output distribution the system aims to preserve. It verifies the draft's work. Key characteristics include:

• Architecture: Typically a large foundation model (e.g., Llama 3 70B, GPT-4) where standard autoregressive inference is costly.
• Function: Performs a single, batched verification forward pass to score all tokens in the candidate sequence against its own predictions.
• Optimization Goal: Execute the parallel verification pass faster than it would take to generate the same number of tokens autoregressively. Its computational profile defines the viability of the speedup.

Acceptance rate is the percentage of tokens proposed by the draft model that are accepted by the target model. It is the single most important determinant of the speedup factor.

• Calculation: If the draft proposes 4 tokens and the target accepts the first 3, the acceptance rate for that step is 75%.
• Impact: A high acceptance rate means more draft tokens are used per verification pass, amortizing the target model's fixed verification cost. A low rate triggers frequent rollbacks, eroding the speedup.
• Targets: Practical systems often require acceptance rates above 70-80% to achieve meaningful latency reduction.

The verification forward pass is the parallelized inference step where the target model validates the draft's candidate sequence. This is the core computational trick of speculative decoding.

• Mechanism: The target model takes the shared prompt context and the candidate sequence as input, processing all candidate tokens in parallel using a modified tree attention mechanism.
• Output: For each position, the model outputs probability distributions. A draft token is accepted if its probability under the target model is greater than a random threshold.
• Cost: This single batched pass must be cheaper than the multiple sequential passes required for standard autoregressive generation of the same number of tokens.

The rollback mechanism is the corrective procedure triggered when the target model rejects a draft token. It ensures the final output distribution matches the target model's autoregressive output.

• Process: Upon rejection at position k, all subsequent tokens in the candidate sequence (positions k+1...n) are discarded.
• Continuation: The system outputs all accepted tokens (up to k-1), then uses the target model's own sampled token for position k as the next token in the sequence.
• Implication: A rollback resets the drafting process, meaning the speculative factor's potential speedup is only realized over sequences of accepted tokens.

A speculative KV cache is an optimized memory management strategy for the key-value (KV) pairs generated during the drafting and verification phases.

• Purpose: To avoid redundant computation by reusing the KV cache from the draft model's forward pass during the target model's verification pass.
• Challenge: The draft and target models often have different architectures, making their KV cache dimensions incompatible.
• Solution: Common techniques include projecting the draft model's KV cache to match the target model's dimensions or using a shared backbone architecture for the pair. Efficient cache management is critical for minimizing the verification cost.