Iteration Time

Iteration time is the inverse of token generation throughput for a fixed batch size. If the iteration time is 50 milliseconds, the system generates 20 tokens per second per sequence in the batch. This relationship is linear: halving the iteration time doubles the token output rate, making it the most direct metric for evaluating decoding speed.

• Throughput Formula: Tokens/sec/sequence = 1000 ms / Iteration Time (ms)
• Batch Impact: Total system throughput = (Batch Size * 1000) / Iteration Time (ms)

During autoregressive decoding, iteration time is dominated by memory bandwidth, not compute. Each step performs a small matrix-vector multiplication to generate the next token, but must load the entire key-value (KV) cache from high-bandwidth memory (HBM). The time is largely spent waiting for data.

• Bottleneck: Speed of reading the KV cache and model weights.
• Implication: Optimizations like KV cache quantization or improved memory access patterns have a greater impact on iteration time than increasing FLOPs.

Iteration time increases linearly with the total context length of all sequences in the batch due to the size of the KV cache. Processing a batch with 10 sequences of length 1000 requires accessing 10x more cached data than a batch with 10 sequences of length 100.

• Cache Size: KV cache memory ~= 2 * batch_size * num_layers * num_heads * d_head * seq_len
• Performance Impact: Longer sequences increase memory traffic per iteration, slowing down each step. Techniques like paged attention help mitigate this by organizing memory more efficiently.

The iteration time for a batch is set by the critical path—the slowest sequence or operation in that forward pass. If one sequence requires a specialized kernel or hits a memory bottleneck, it stalls the entire batch. This is a key challenge for variable-length batching.

• Synchronization Point: GPU execution is synchronous within a batch; all sequences must finish the forward pass before the next iteration begins.
• Optimization Target: Minimizing variance in per-sequence processing time through efficient kernels and memory layouts is essential for low, predictable iteration time.

Iteration time is a property of the decoding loop, separate from the initial prefill phase. Prefill is a single, compute-intensive parallel processing of the prompt. Decoding is many sequential, memory-bound steps. They are optimized differently.

• Prefill Phase: High FLOP utilization, processes the entire prompt at once.
• Decoding Phase: Characterized by iteration time; generates tokens auto-regressively.
• System Design: High-performance servers often use separate hardware queues or scheduling policies for prefill vs. decode requests.

For any request, the total time-to-first-token (TTFT) and time-per-output-token (TPOT) can be derived from iteration time and scheduling behavior.

• TTFT: = Prefill Latency + Queueing Delay.
• TPOT: ≈ Iteration Time (when batch composition is stable).
• Total Latency: = TTFT + (Output Token Count - 1) * TPOT.

Engineers use iteration time to model and guarantee service-level objectives (SLOs) for interactive applications like chatbots, where TPOT directly impacts user perception.

Tail latency refers to the high-percentile response times (e.g., p95, p99) of inference requests, which are critical for user experience in interactive applications. While average latency may be low, a few slow requests can dominate user perception.

• Causes: Variability in sequence length, system noise, garbage collection, and head-of-line blocking in batches.
• Impact: Directly affects service-level agreements (SLAs) for real-time applications like chatbots.
• Mitigation: Techniques like request interleaving and intelligent scheduling aim to reduce tail latency variance.

A memory-bound operation is one where execution time is limited by the speed of reading from or writing to memory (e.g., GPU HBM or DRAM), not by the processor's computational speed. The decoding phase of autoregressive generation is typically memory-bound.

• Primary Bottleneck: The time to load the model's weights and the KV cache for each generated token.
• Contrast with Compute-Bound: The prefill phase is usually compute-bound, dominated by matrix multiplications.
• Optimization Focus: Techniques aim to reduce memory bandwidth pressure, such as KV cache quantization and optimizing data layout.

The decoding phase is the iterative, sequential stage of autoregressive model inference where output tokens are generated one at a time. Each step (iteration) conditions on the previously generated tokens.

• Characteristics: Inherently sequential, memory-bandwidth-bound, and latency-sensitive.
• Key Component: Relies heavily on the KV (Key-Value) Cache to avoid recomputing previous attention states.
• Performance Driver: Iteration time is the fundamental latency of this phase. Throughput is determined by iteration time and effective batch size.

The prefill (or prompt processing) phase is the initial, parallel computation stage where the model processes the entire input prompt to generate the first output token and populate the initial KV cache.

• Characteristics: Compute-intensive, parallelizable across the prompt length, and typically compute-bound.
• Contrast with Decoding: The prefill phase has high latency for long prompts but is amortized over many generated tokens.
• Scheduling Impact: In continuous batching, a new request's prefill can cause a temporary spike in iteration time for the active batch.

Throughput is the rate at which a system processes work, measured in tokens per second or requests per second. For inference, it represents the total capacity of the system.

• Relationship to Iteration Time: System throughput = (Effective Batch Size) / (Iteration Time). Lower iteration time enables higher throughput.
• Trade-off with Latency: Increasing batch size generally improves throughput but can increase per-request latency (iteration time may stay constant, but queueing delay increases).
• Measurement: Requires steady-state load and is distinct from peak or burst performance metrics.

Kernel launch overhead is the latency incurred when dispatching a computational kernel (a GPU function) for execution. This includes costs from the driver, runtime, and hardware scheduler.

• Impact on Iteration Time: For very small operations, this overhead can dominate the actual computation time. In decoding, each iteration involves launching multiple kernels.
• Optimization: Operator/Kernel fusion combines multiple small operations into a single kernel to amortize launch overhead and improve memory access patterns.
• Significance: Becomes a critical bottleneck for low-latency, single-token generation scenarios.