Inference Latency

The time required to process the static input prompt and context through the model's forward pass, generating the initial Key-Value (KV) cache before token generation begins. This phase is compute-bound and scales with prompt length and model size.

• Primary Driver: Complexity of the initial encoder/forward pass.
• Optimization Target: Operator fusion, efficient attention computation for long contexts.

The time consumed during the autoregressive token generation phase, where each new output token is produced conditioned on all previously generated tokens. This is typically the dominant latency component for long outputs.

• Primary Driver: Sequential nature of autoregressive generation.
• Key Metric: Time Per Output Token (TPOT).
• Optimization Target: Speculative decoding, improved memory bandwidth utilization for KV cache reads.

The time an inference request spends waiting in a scheduler's queue before GPU execution begins. This is a major component of end-to-end latency under load and directly impacts tail latency (P95, P99).

• Primary Driver: Number of concurrent requests exceeding immediate compute capacity.
• Mitigation: Advanced schedulers with continuous batching to maximize GPU utilization and minimize idle time.

The additional delay incurred when servicing the first request(s) to a model that is not loaded in GPU memory. This includes time to load weights from disk, initialize the runtime, and warm up caches.

• Primary Driver: Model size and storage I/O bandwidth.
• Impact: Critical for serverless or auto-scaling environments where instances spin up/down dynamically.
• Mitigation: Pre-warmed pods, model keeping policies, and optimized serialization formats (e.g., Safetensors).

The latency of core mathematical operations on the accelerator (GPU/TPU) and the time spent moving data between host (CPU) and device memory. Includes GPU kernel launch overhead.

• Primary Drivers: GPU compute capability, memory bandwidth, and PCIe bus saturation.
• Key Bottlenecks: Small, inefficient kernels; excessive H2D/D2H transfers for pre/post-processing.
• Optimization Target: Operator fusion, using optimized execution graphs (TensorRT, ONNX Runtime), and keeping data on-device.

The delay introduced by transmitting the request and response over the network and serializing/deserializing data structures. This is captured in end-to-end latency.

• Primary Drivers: Payload size (input + output tokens), network round-trip time (RTT), and serialization efficiency.
• Common Frameworks: gRPC latency (protobuf serialization, HTTP/2), REST API overhead.
• Mitigation: Efficient binary protocols, compression, and colocating clients with inference endpoints.

The total elapsed time from client request initiation to receipt of the complete final response. This is the user-perceived latency and includes all sub-components:

• Network transmission (client to server and back)
• Request queuing and scheduling
• Server-side processing (prefill, decode)
• Serialization/deserialization overhead

Key Insight: While inference latency focuses on model execution, end-to-end latency provides the complete service-level picture, essential for defining user-facing Service Level Objectives (SLOs).

The duration from request submission to the generation or delivery of the first output token. This metric is critical for streaming applications where perceived responsiveness is paramount.

Mechanism: TTFT is dominated by the prefilling latency—the single forward pass through the model with the input prompt—and initial system overhead. A low TTFT ensures the user feels the model has begun thinking immediately, even if total generation time is longer.

The average latency incurred to generate each subsequent token after the first in an autoregressive model. This directly controls the speed of streaming completions.

Drivers: TPOT is determined by the efficiency of the decoding phase, where each new token is produced conditioned on all previous ones. It is heavily influenced by:

• GPU memory bandwidth for reading the KV cache
• Autoregressive computational dependency
• Optimization techniques like speculative decoding

The high-percentile response times (e.g., the 95th or 99th percentile) that represent the slowest requests in a distribution. While average latency is important, tail latency defines worst-case user experience and system stability.

Causes: Tail latency spikes are often caused by:

• Resource contention (e.g., noisy neighbors in multi-tenant clouds)
• Garbage collection pauses
• Request queuing delays during traffic bursts
• Cold starts for infrequently used models

Managing P99 latency is a core challenge for production AI services.

An inference optimization technique, also known as in-flight or dynamic batching, where new requests are added to a running batch as previous requests finish generation. This contrasts with static batching, which waits for all requests in a batch to complete before starting a new one.

Impact on Latency:

• Dramatically improves GPU utilization and throughput, amortizing fixed costs.
• Can increase individual request latency if a request must wait for others in its batch.
• Requires sophisticated scheduling to balance throughput gains with latency SLOs.

It is a foundational technique in high-performance servers like vLLM and TGI.

The two primary computational phases of autoregressive language model inference, each with distinct performance characteristics.

Prefilling Latency:

• Processes the entire input prompt in one parallel forward pass.
• Computationally intensive but highly parallelizable.
• Generates the initial Key-Value (KV) cache.

Decoding Latency:

• The autoregressive loop generating tokens one-by-one.
• Memory-bound: dominated by reading the growing KV cache.
• Has limited parallelism per step.

Optimization Focus: Prefilling is optimized via operator fusion and efficient attention. Decoding is optimized via KV cache management (e.g., PagedAttention) and speculative execution.