Target Model

The target model's core function is to act as the final arbiter of token correctness. It performs a verification forward pass, scoring a batch of candidate tokens proposed by the draft model against its own autoregressive predictions. This process ensures the final output distribution remains faithful to the target model's original capabilities, preventing quality degradation from the draft model's approximations.

A target model is typically a large-scale transformer (e.g., Llama 3 70B, GPT-4) with significantly more parameters and layers than its paired draft model. This scale provides:

• Higher reasoning accuracy and factual grounding.
• Broader world knowledge and linguistic capability.
• The computational weight that makes autoregressive decoding slow, which speculative decoding aims to mitigate. Its architecture must support efficient parallel scoring of multiple candidate tokens in a single batch.

For speculative decoding to provide a net speedup, the target model must verify tokens faster than generating them autoregressively. Key efficiency factors include:

• Parallelizable Forward Pass: The model must efficiently score a sequence of k draft tokens in roughly the same time as scoring one token.
• Optimized KV Cache Management: Reusing or efficiently computing the key-value (KV) cache for the candidate sequence is critical to avoid redundant computation.
• Hardware Utilization: The verification step must saturate GPU compute and memory bandwidth to offset its cost.

The target model defines the latency-accuracy tradeoff for the entire speculative decoding system. Its presence guarantees that the output quality has a known upper bound—its own native performance. The system's speedup is measured against the target model's baseline autoregressive latency. Any deviation from the target model's exact output sequence (e.g., due to early stopping) is a deliberate engineering tradeoff, not a model capability issue.

Effective speculative decoding requires architectural and distributional alignment between the target and draft models. Key compatibility aspects include:

• Vocabulary Alignment: Both models must use the same tokenizer.
• Output Space Consistency: The draft model's predictions should approximate the target model's probability distributions to achieve a high acceptance rate.
• Draft Model Distillation: Often, the draft model is specifically trained (distilled) from the target model to improve this alignment.

The target model is the central component integrated into the broader inference serving stack. This involves:

• Serving Infrastructure: It must be deployed within a model serving architecture (e.g., using vLLM, TGI) that supports dynamic batching for verification passes.
• Orchestration Logic: The serving system manages the rollback mechanism, switching to autoregressive generation from the target model when draft tokens are rejected.
• Performance Telemetry: Its latency and acceptance rates are key metrics for inference performance benchmarking and cost optimization.

The draft model is the smaller, faster language model that proposes candidate token sequences for the target model to verify. Its primary design goal is low-latency generation, often achieved through architectural simplicity, fewer parameters, or aggressive quantization.

• Purpose: Generate a speculative future sequence (e.g., 3-5 tokens) rapidly.
• Relationship: Acts as a proxy for the target model's next-token distribution.
• Optimization: Often distilled from the target model to maximize acceptance rate.

Token verification is the deterministic process where the target model scores the draft model's proposed sequence. In a single, batched forward pass, the target model computes probability distributions for each position in the candidate sequence.

• Mechanism: The target model checks if its own predicted token matches the draft's proposal at each position.
• Acceptance Rule: A draft token is accepted if it matches the target's argmax token or, in stochastic variants, if sampled from the target's adjusted distribution.
• Output: Produces a verified subsequence and triggers a rollback on the first rejection.

The acceptance rate is the critical efficiency metric for speculative decoding, defined as the percentage of draft tokens accepted by the target model. It directly determines the theoretical speedup.

• Calculation: (Number of Accepted Tokens) / (Total Draft Tokens Proposed).
• Impact: A higher acceptance rate reduces the frequency of costly rollbacks to standard autoregressive generation.
• Typical Range: For a well-tuned draft model, rates of 70-85% are common, but this varies by domain and model pair.

A verification forward pass is the single, parallelized inference step through the target model that validates a candidate sequence. This is the core computational trade-off of the technique.

• Efficiency: The cost of this batched pass must be less than the cost of generating the same number of tokens autoregressively.
• Implementation: Uses a modified attention mechanism (like tree attention) to score the sequence in parallel.
• KV Cache: Efficiently reuses the speculative KV cache from the draft phase to avoid recomputation.

The rollback mechanism is the corrective procedure invoked when the target model rejects a draft token. It ensures the final output distribution is identical to the target model's native autoregressive output.

• Trigger: The first position where the draft token is rejected.
• Action: Generation reverts to that position. The target model's own predicted token is emitted, and decoding continues autoregressively or with a new speculation.
• Guarantee: This mechanism preserves the latency-accuracy tradeoff, ensuring no degradation in output quality.

A small-big model pair is the specific combination of models deployed for speculative decoding. The pairing is a system-level optimization problem.

• Selection Criteria: The draft model's speed and the pair's acceptance rate.
• Architectures: Often a distilled version (e.g., a 7B parameter model drafting for a 70B model) or a heavily quantized variant of the same model family.
• Deployment: Both models must be hosted concurrently, impacting total GPU memory footprint and inference cost.