Speculative Decoding

The choice of draft model is the primary engineering decision. It must be:

• Significantly faster than the target model (often 3-10x) to offset verification overhead.
• Architecturally aligned (e.g., same tokenizer) to ensure token compatibility.
• Sufficiently accurate to maintain a high acceptance rate. Common strategies include using a smaller version of the same model family (e.g., Llama 3 8B drafting for Llama 3 70B) or a heavily distilled model. The draft model's size directly impacts memory footprint and cost.

The core of speculative decoding is the parallel verification step. The target model processes the proposed token sequence in a single forward pass. The system then performs a token-by-token comparison between the draft and target model outputs.

• Acceptance: Tokens are accepted until the first mismatch.
• Rollback: Generation rolls back to the first incorrect token, which is replaced by the target model's corrected output. This deterministic correction ensures output quality is identical to standard autoregressive generation. Efficient implementation requires careful tensor masking and index management.

The actual speedup is not guaranteed and depends on several variables:

• Draft Acceptance Rate: The probability the draft's next token matches the target's. Higher rates (e.g., >80%) yield better speedups. This rate is task and data-dependent.
• Draft Length (k): The number of tokens proposed per step. There's a trade-off: longer drafts increase potential speedup but risk lower acceptance rates for later tokens. Optimal k is typically 3-5.
• Hardware Parallelism: Speedup is maximized when the verification forward pass time is less than generating k tokens autoregressively. This requires sufficient parallel compute (e.g., large batch size on GPU). The theoretical max speedup is k.

Speculative decoding introduces a cost-memory-latency trade-off:

• Compute Cost: While it reduces time-to-first-token and increases tokens/second, it requires running two models. The cost efficiency gain depends on the draft model's cheapness relative to the speedup achieved.
• Memory Overhead: The draft model and the parallel verification batch increase GPU memory consumption. The draft's KV cache must also be managed.
• Infrastructure Complexity: Requires hosting and loading two models, complicating serving systems. The benefit is greatest for large, slow target models where the latency reduction provides significant user experience or cost-per-token improvements.

Ideal for:

• Chat applications with long, interactive sessions where low latency is critical.
• Batch inference jobs (e.g., summarization) where high throughput reduces cost.
• Larger models (70B+ parameters) where the relative cost of the draft is small.

Less effective for:

• Very small target models where the draft offers little relative speed advantage.
• Tasks with highly unpredictable outputs (e.g., creative writing) where draft acceptance rate plummets.
• Memory-constrained environments where loading two models is prohibitive.

Also known as in-flight batching, this is a serving optimization that dynamically adds new requests to a running batch as previous sequences finish generation. Unlike static batching, it eliminates idle GPU time caused by variable-length sequences, maximizing hardware utilization and throughput. It is a foundational technique often used in conjunction with speculative decoding to improve overall server efficiency.

• Key Mechanism: Manages a pool of active requests, scheduling token generation in waves.
• Benefit: Dramatically increases tokens per second (TPS) in multi-user serving scenarios.
• Example: A request finishing early doesn't stall the batch; a new request is immediately slotted in.

The Key-Value (KV) Cache is a memory optimization for transformer decoder blocks. It stores computed key and value tensors for all previously generated tokens during autoregressive decoding. This prevents the recomputation of these tensors for every new token, which is the primary computational bottleneck. Speculative decoding relies heavily on efficient KV cache management, as both the draft and target models use their own caches.

• Purpose: Trade memory for compute, reducing latency per token.
• Challenge: Memory footprint grows linearly with sequence length and batch size.
• Optimization: Techniques like PagedAttention (from vLLM) manage this cache in non-contiguous blocks to reduce fragmentation.

A model compression technique that reduces the numerical precision of a model's weights and activations (e.g., from 32-bit floating-point to 8-bit integers). This decreases memory footprint and increases compute speed on hardware with optimized integer units. Quantization is often applied to the draft model in speculative decoding to make it extremely fast and cheap to run.

• INT8/FP8: Common formats for inference, offering 4x memory reduction vs. FP32.
• GPTQ/AWQ: Advanced algorithms for quantizing to very low precision (e.g., 4-bit) with minimal accuracy loss.
• Use Case: A 4-bit quantized, small model serves as an efficient draft model, while the target model may run in higher precision.

PagedAttention is an algorithm that manages the KV cache in fixed-size, non-contiguous blocks (pages), analogous to virtual memory in operating systems. It is the core innovation behind the vLLM inference serving engine. This eliminates memory fragmentation caused by variable-length sequences, allowing for near-zero waste in KV cache memory. This efficient memory management is critical for serving systems implementing speculative decoding at high concurrency.

• Solves: External and internal fragmentation of KV cache.
• Result: Enables higher batch sizes and better GPU utilization.
• Ecosystem: vLLM is a production-grade serving system that natively supports speculative decoding.

An inference acceleration technique where certain input samples exit the neural network through intermediate layers rather than passing through all layers. The heuristic is that 'easier' tokens or sentences can be accurately predicted with less computation. This is conceptually related to speculative decoding as both are adaptive computation methods, but early exiting modifies the depth of computation for a single model, while speculative decoding uses a separate draft model.

• Mechanism: A confidence metric (e.g., entropy) at intermediate layers determines if inference can halt early.
• Contrast: Speculative decoding modifies the breadth of token exploration per step using a separate model.
• Goal: Reduce average latency, particularly for non-uniform difficulty inputs.

The core business metrics that speculative decoding directly optimizes. Inference Cost is the total financial expenditure of generating tokens, driven by GPU time and memory. Tokens Per Second (TPS) is the primary throughput metric for LLM serving. Speculative decoding aims to increase TPS for a given model (reducing latency) and/or maintain TPS with a lower effective compute cost, thereby driving down the cost per token.

• Direct Impact: A successful speculative decoding system can increase TPS by 2-3x for the same hardware.
• Metric: Performance is measured by speedup (wall-time time reduction) and acceptance rate (how many draft tokens are verified).
• Trade-off: Must account for the added cost of running the draft model versus the time saved.