Model Inference Latency

The initial stage where raw input data (e.g., text, images) is converted into the numerical format (tensors) required by the model. This includes:

• Tokenization: Splitting text into sub-word units.
• Normalization/Resizing: Standardizing image pixel values or audio samples.
• Batching: Grouping multiple requests for parallel processing, which introduces a queuing delay but improves throughput. Latency here is often CPU-bound.

The core computational phase where input tensors are propagated through the neural network's layers. This is typically the most GPU/accelerator-intensive component. Key factors influencing its duration include:

• Model Size & Architecture: Larger models (more parameters, layers) require more FLOPs.
• Sequence Length: For transformers, compute scales quadratically with input token length in attention layers.
• Hardware: Performance is dictated by accelerator memory bandwidth (for loading weights) and compute throughput (for matrix multiplications).

A critical latency metric for autoregressive text generation models (e.g., LLMs). TTFT measures the delay from request start until the first output token is generated. This period includes the full prefill stage, where the entire input prompt is processed in one forward pass to initialize the model's internal state (KV cache). High TTFT directly impacts user-perceived responsiveness.

The throughput metric following TTFT. TPOT measures the average time to generate each subsequent token. This occurs during the decode stage, where the model performs a much smaller forward pass for each new token, conditioned on the cached previous outputs. TPOT determines the speed of streaming responses and is heavily influenced by memory bandwidth constraints.

The final stage where the model's raw numerical output is converted into a usable format for the client. This may include:

• Detokenization: Converting token IDs back into human-readable text.
• Formatting: Applying output templates (JSON, XML).
• Filtering: Applying logit processors for tasks like top-p (nucleus) sampling or beam search.
• Streaming: Chunking and sending tokens as they are generated.

Latency introduced by the surrounding infrastructure, often outside the direct model execution. This includes:

• Network Round-Trip Time (RTT): Between client, load balancer, and inference server.
• Inter-Process Communication (IPC): If pre/post-processing runs on separate CPUs from the GPU.
• Queueing Delay: Time a request spends waiting if the system is at capacity (saturation).
• Cold Starts: Initialization latency when loading a model into GPU memory after a period of inactivity.

A Service Level Objective (SLO) is a quantitative target for the reliability or performance of a service, expressed as a percentage of requests that must meet a specific Service Level Indicator (SLI) over a defined time window. For model inference latency, an SLO might be "99% of inference requests must complete within 100ms over a 30-day window." SLOs are internal goals that drive engineering priorities and error budget management.

A Service Level Indicator (SLI) is a directly measurable metric that quantifies a specific aspect of a service's performance. Model inference latency is a primary SLI for AI services. Other common AI SLIs include:

• Throughput (queries per second)
• Error Rate (e.g., failed inference requests)
• Quality Metrics (e.g., hallucination rate, retrieval precision) SLIs provide the raw data against which Service Level Objectives (SLOs) are evaluated.

Percentile latency is a statistical measure of request processing time, where a given percentile indicates the maximum latency experienced by that percentage of requests. It is essential for understanding user experience beyond average latency.

• p50 (Median): The latency at which 50% of requests are faster and 50% are slower.
• p95: 95% of requests are at or below this latency. A common target for SLOs.
• p99: 99% of requests are at or below this latency, representing the tail latency experienced by the worst-case requests. p99 is highly sensitive to system bottlenecks and dependency chains.

For autoregressive models (like LLMs), latency is decomposed into two key SLIs:

• Time To First Token (TTFT): The duration from request submission to the generation of the first output token. This determines perceived responsiveness.
• Time Per Output Token (TPOT): The average latency to generate each subsequent token after the first. This determines the speed of streaming responses. Optimizations often target TTFT (via improved scheduling and prefill) and TPOT (via better decoding kernels) separately.

Continuous batching is a core inference optimization technique that dynamically groups requests of varying lengths and processing states to maximize GPU utilization. Unlike static batching, it allows new requests to join a batch as others finish, dramatically improving throughput SLIs and reducing tail latency. It is a foundational method in high-performance inference servers like vLLM and TGI.

An error budget is the allowable amount of service unreliability, calculated as 100% - SLO. If the SLO is 99.9%, the error budget is 0.1%. It defines the risk capacity for deployments and changes. Burn rate is the speed at which this budget is consumed (e.g., "we are burning error budget 5x faster than allowed"). It is a key metric for multi-window alerting, triggering alerts based on the risk of an imminent SLO violation rather than on single metric thresholds.