Decoding Latency

The time required for the model to perform a single, full forward pass on the static input prompt and context. This phase generates the initial Key-Value (KV) cache for the attention mechanism, which is then used during autoregressive generation. It is a one-time, upfront cost before the first output token is produced. Factors influencing prefilling latency:

• Length of the input context/prompt.
• Model size (parameter count).
• Hardware compute and memory bandwidth.
• Batch size of concurrent requests.

The average latency incurred for generating each subsequent token after the first in an autoregressive sequence. This is the core iterative cost of decoding. TPOT is primarily driven by the small, sequential forward pass needed to produce the next token, which reads from and updates the KV cache. Key determinants of TPOT:

• Model architecture and per-token FLOPs.
• Efficiency of KV cache memory access (bandwidth-bound).
• GPU kernel launch overhead for small operations.
• Continuous batching efficiency, which can amortize overhead across requests.

The latency overhead associated with storing and retrieving the attention key and value tensors for all previous tokens. This cache grows linearly with sequence length and is critical for avoiding recomputation. Inefficient management is a major source of decoding slowdown. Critical aspects include:

• Memory allocation and fragmentation for variable-length sequences.
• Memory bandwidth saturation as caches exceed GPU SRAM (L2 cache).
• PagedAttention (as in vLLM), which virtualizes the KV cache to eliminate fragmentation and waste, significantly improving throughput and latency under high concurrency.

The latency introduced or saved by the inference scheduler's strategy for grouping and executing requests. Continuous batching (or in-flight batching) dynamically adds new requests to a running batch as others finish, maximizing GPU utilization. Scheduling impacts include:

• Request queuing delay: Time a request waits before execution begins.
• GPU utilization vs. latency trade-off: Larger batches increase throughput but can raise TPOT for individual requests.
• Memory contention from multiple concurrent sequences sharing GPU resources.

The latency from the low-level execution of neural network operations on the GPU. This involves the launch and execution of many small computational kernels. Optimizations here directly reduce TPOT:

• Operator Fusion: Combining multiple sequential ops (e.g., Linear, Bias, Activation) into a single kernel to reduce memory accesses and launch overhead.
• Kernel Auto-Tuning: Selecting the most efficient GPU kernel implementation for specific input sizes and hardware.
• Use of optimized model execution graphs from compilers like TensorRT or ONNX Runtime, which apply these optimizations statically.

The ancillary latency not from the model's computation itself, but from the serving framework and system stack. This can become a significant portion of total latency, especially for short sequences or high QPS. Components include:

• GPU-CPU Synchronization: Overhead from device-host memory transfers and synchronization points.
• Python GIL Contention: In Python-based servers, the Global Interpreter Lock can block concurrent request handling.
• gRPC/HTTP Latency: Network stack overhead for remote procedure calls, including serialization/deserialization of protocol buffers.
• Token Sampling Logic: The computational cost of applying top-p, top-k, or temperature scaling to logits.

Time to First Token (TTFT), also known as First Token Latency, is the duration from the start of an inference request to when the first token of the output is generated or delivered to the client. This metric is crucial for perceived responsiveness in streaming applications.

• Primary Driver: The prefilling latency—the forward pass through the model with the input prompt—dominates TTFT.
• Key Consideration: Users perceive a system as "fast" or "slow" based on TTFT, making it a primary target for optimization.

Time Per Output Token (TPOT) is the average latency incurred for generating each subsequent token after the first in an autoregressive model. It directly dictates the speed of streaming completions.

• Direct Relationship: TPOT is the inverse of a system's token generation throughput.
• Optimization Target: Techniques like continuous batching, PagedAttention, and speculative decoding aim to minimize TPOT by improving GPU utilization and reducing memory bottlenecks during the decoding phase.

Prefilling latency is the time required for a language model to process the static input prompt and context through its initial forward pass. This phase generates the first Key-Value (KV) cache before autoregressive token generation begins.

• Impact on TTFT: Prefilling is the major component of Time to First Token (TTFT).
• Computational Profile: This phase is compute-bound, as the entire prompt context must be processed in parallel, making it sensitive to prompt length and GPU FLOPs.

Continuous batching, also known as dynamic or in-flight batching, is an inference optimization technique where new requests are dynamically added to a running batch as previous requests finish generation.

• Core Benefit: Maximizes GPU utilization and throughput by eliminating idle time where only part of a batch is active.
• Latency Trade-off: While improving aggregate throughput, it can introduce slight request queuing delay as new requests wait for the next scheduling cycle. Engines like vLLM implement this efficiently.

PagedAttention is an algorithm, introduced by the vLLM serving engine, that manages the Key-Value (KV) cache using virtual memory paging concepts.

• Solves Fragmentation: It eliminates memory waste caused by variable-length sequences, allowing non-contiguous storage of attention keys and values.
• Direct Impact: This optimization dramatically increases the number of concurrent requests a GPU can handle, thereby improving throughput and reducing decoding latency under load.

Speculative decoding is an inference acceleration technique where a small, fast 'draft' model proposes a sequence of tokens that are then verified in parallel by a larger, more accurate 'target' model.

• Mechanism: If the target model accepts the draft tokens, multiple tokens are generated in a single, costly verification step.
• Latency Reduction: It effectively reduces the number of slow autoregressive steps the large model must perform, lowering the average Time Per Output Token (TPOT).