Inter-Token Latency

The fundamental design and parameter count of the LLM impose a computational lower bound. Key architectural factors include:

• Parameter Count: Larger models (e.g., 70B+ parameters) require more floating-point operations (FLOPs) per token, increasing compute time.
• Attention Mechanism: The complexity of the attention calculation, especially in models with large context windows, directly impacts the time per decoding step. Techniques like grouped-query attention (GQA) or sliding window attention are used to reduce this cost.
• Model Family: Architectures like Mixture of Experts (MoE) can have variable latency depending on which experts are activated for a given token.

The physical layer of computation is a primary determinant. Critical elements are:

• GPU Memory Bandwidth: Autoregressive decoding is often memory-bandwidth bound. The speed at which model weights and the KV Cache can be read from GPU VRAM (e.g., HBM on NVIDIA H100/A100) sets a hard limit.
• Compute Capability: The FLOP/s (Floating-Point Operations per second) of the accelerator (e.g., Tensor Cores on NVIDIA GPUs) determines how quickly the dense matrix multiplications for each layer can be processed.
• Host-to-Device Latency: The time to transfer prompts and receive tokens between the CPU (host) and GPU (device) can add overhead, especially for small batches.

Software and algorithmic optimizations applied during model serving can drastically reduce latency.

• KV Cache Management: Efficiently storing and retrieving the Key-Value cache for the attention mechanism avoids recomputing past token states. Cache misses or inefficient eviction policies can cause significant delays.
• Continuous Batching: Dynamically batching requests as they finish (iteration-level batching) maximizes GPU utilization and amortizes overhead, improving overall Tokens per Second (TPS) and reducing average inter-token latency.
• Quantization: Using lower precision (e.g., FP8, INT8) for weights and activations reduces memory bandwidth requirements and increases compute throughput.
• Operator Fusion & Kernel Optimization: Custom, low-level CUDA kernels that fuse multiple operations (like attention or layer normalization) reduce kernel launch overhead and memory transfers.

The algorithm used to select the next token introduces variable computational cost.

• Greedy Decoding: Selecting the token with the highest probability is deterministic and fast, as it requires only a argmax operation over the vocabulary logits.
• Sampling Methods: Techniques like top-k, top-p (nucleus), or temperature sampling require more computation: sorting logits, calculating cumulative probabilities, and performing random sampling.
• Beam Search: Maintaining multiple candidate sequences (beams) requires computing and tracking probabilities for k paths, significantly increasing latency compared to greedy decoding. It is rarely used for streaming chat due to this high cost.

The runtime state of the serving system introduces variable overhead.

• Request Concurrency & Queueing: High traffic can lead to requests waiting in a queue for available batch slots or GPU resources, increasing the observed end-to-end latency between tokens.
• Multi-Tenancy Noise: In shared GPU clusters, other co-located workloads can cause contention for memory bandwidth, cache, and compute cycles, leading to unpredictable latency spikes.
• Cold Starts: The first request after a period of inactivity may experience higher latency due to model loading, kernel compilation, or cache warming.

Properties of the specific input and output sequence being processed affect generation time.

• Output Token Position: Early tokens in a sequence often have slightly higher latency as the KV Cache is being populated. Later tokens benefit from a stable cache state.
• Vocabulary Size & Logit Processing: Models with larger vocabularies require a bigger final linear layer (the lm_head) to produce logits, adding a small but consistent cost per token.
• Prompt Complexity: While the prefill stage affects Time to First Token (TTFT), a very complex prompt can result in a large initial KV Cache, which may slightly impact the memory bandwidth available for the first few generated tokens.

Also known as iterative batching or dynamic batching, this technique dynamically adds new inference requests to a running batch as previous requests finish generation. Unlike static batching, which waits for the entire batch to complete, continuous batching improves GPU utilization and overall system throughput, directly reducing average inter-token latency by keeping hardware consistently occupied. It is a foundational optimization in high-performance inference servers like vLLM and NVIDIA Triton.

The Key-Value (KV) Cache stores computed key and value vectors from the transformer's attention mechanism for previously generated tokens. Optimizing its management is crucial for latency.

• Memory Layout: Using paged attention (as in vLLM) to manage cache in non-contiguous blocks reduces memory fragmentation and waste.
• Quantization: Applying INT8 or FP4 quantization to the KV cache reduces memory bandwidth pressure, allowing faster reads/writes.
• Cache Sharing: For multi-user scenarios, techniques like prefix caching can share computed KV states for common prompt prefixes across requests.

This advanced technique uses a smaller, faster draft model to predict a sequence of several future tokens. These predictions are then verified in a single forward pass by the larger target model. If verified, multiple tokens are accepted at once, dramatically reducing the number of costly large-model decoding steps. This can improve inter-token latency by 2-3x for compatible workloads, though it requires maintaining two models and benefits from high-prediction acceptance rates.

The attention computation is a primary bottleneck. Several optimizations target it:

• FlashAttention: An I/O-aware algorithm that reduces memory reads/writes between GPU SRAM and HBM, speeding up the attention computation itself.
• Multi-Query Attention (MQA) & Grouped-Query Attention (GQA): Reduce the size of the KV cache by sharing key and value heads across query heads, decreasing memory footprint and bandwidth requirements during decoding.
• Sparse Attention: For very long contexts, limiting the attention window to a relevant subset of past tokens reduces computational complexity.

Reducing the computational load of the model directly lowers latency.

• Weight Quantization: Converting model weights from FP16 to lower precision formats like INT8 or INT4 reduces memory traffic and can accelerate compute on supported hardware (e.g., NVIDIA Hopper GPUs with FP8).
• Weight Pruning: Removing less important weights creates a sparser model that can leverage specialized kernels for faster inference.
• Distillation: Training a smaller, faster student model to mimic a larger teacher model often results in a model with lower inherent latency.

Maximizing low-level hardware efficiency.

• Custom Kernels: Using hand-optimized CUDA kernels (e.g., for fused operations like layer normalization) minimizes kernel launch overhead and improves instruction-level parallelism.
• Tensor Parallelism: Splitting model layers across multiple GPUs reduces the per-device workload, though it introduces communication overhead that must be managed.
• GPU Architecture Targeting: Compiling models for specific architectures (e.g., NVIDIA's TensorRT-LLM for Ampere/Hopper) enables the use of the latest hardware features like tensor cores and TMA units.