Latency

Time to First Token measures the duration from request submission until the first output token is generated by an autoregressive model. This initial delay encompasses the entire prefill stage, including:

• Prompt encoding and processing
• Loading the model's KV cache
• Executing the initial forward pass through all layers

TTFT is highly sensitive to prompt length and model size. For long-context models, this can be the dominant component of total latency.

Time per Output Token is the average latency to generate each subsequent token after the first. This metric represents the incremental cost of the decoding stage in autoregressive generation. Key factors influencing TPOT include:

• Model architecture and size
• Efficiency of the KV cache lookup
• Hardware memory bandwidth (often the bottleneck)

Optimizations like speculative decoding target TPOT directly, using smaller draft models to propose multiple tokens for verification in a single pass of the large model.

Tail latency refers to the high-percentile latencies (e.g., P95, P99) in a request distribution. While average latency is useful, tail latency determines user-perceived performance. The slowest 1% of requests (P99) often dictates Service Level Objectives (SLOs).

Causes of high tail latency include:

• Resource contention from concurrent requests
• Garbage collection pauses or other system noise
• Cold starts for infrequently used models
• Variable input/output lengths causing uneven processing times

End-to-End Latency is the total elapsed time from the client's perspective, encompassing all system components beyond just model execution. This holistic metric includes:

• Network transmission (client to server and back)
• Request queuing and scheduling time
• Pre/post-processing (tokenization, detokenization, formatting)
• Model inference time (TTFT + TPOT * output length)

For real-time applications like chatbots, end-to-end latency must typically be under 200-300 milliseconds to feel instantaneous.

Inter-Token Latency measures the time gap between consecutive tokens as they are streamed to a client. In streaming response scenarios, consistent inter-token latency is critical for user experience. High variance can make output feel "jerky."

This metric is influenced by:

• The underlying TPOT of the model
• Network buffering and transmission protocols
• Server-side batching strategies, where a single batch processes tokens for multiple requests, potentially delaying individual token delivery

Latency is measured in two distinct operational phases:

Cold Start Latency: The significant delay on the first request to a newly initialized service or model. This includes:

• Loading model weights from disk into GPU memory
• Just-in-Time (JIT) compilation of kernels (e.g., with PyTorch or TensorRT)
• Warm-up inference passes to stabilize performance

Steady-State Latency: The consistent performance achieved after caches are populated, kernels are compiled, and the system has reached equilibrium. Performance benchmarks and SLOs are typically defined for steady-state operation.

Throughput measures the rate at which a system processes work, typically expressed in Queries Per Second (QPS) or Tokens Per Second (TPS). It represents the system's overall capacity, while latency measures the experience of a single request.

• Key Relationship: Systems often face a trade-off between latency and throughput. Increasing load to maximize throughput typically increases average latency due to queuing.
• Example: A GPU server might achieve 1000 TPS but have a P99 latency of 500ms under that load. Reducing the load to 500 TPS might lower P99 latency to 100ms.

Time to First Token is a critical latency metric for autoregressive language models (like LLMs). It measures the delay from request submission until the first output token is generated.

• What it Captures: This metric includes the initial processing overhead: prompt encoding, loading the model's KV cache, and executing the first forward pass through the entire model.
• User Impact: TTFT directly affects perceived responsiveness. A high TTFT makes a chat interface feel sluggish, even if subsequent tokens arrive quickly.
• Optimization Target: Techniques like continuous batching and prefetching aim to reduce TTFT.

Time per Output Token measures the average latency for generating each subsequent token after the first in an autoregressive sequence. It represents the incremental cost of generation.

• Mechanism: After TTFT, generating the next token primarily involves a much smaller forward pass focused on the new token and updating the KV cache. TPOT is typically much lower than TTFT.
• Determinants: TPOT is influenced by model size, memory bandwidth (for cache access), and the efficiency of the sampling algorithm.
• System Design: For long, streaming responses (e.g., story generation), optimizing TPOT is crucial for a smooth user experience.

Tail latency refers to the high-percentile latencies in a request distribution, such as the P90 (90th percentile) or P99 (99th percentile). It measures the experience of the slowest requests.

• Why it Matters: User perception and Service Level Objectives (SLOs) are often defined by tail latency, not averages. A few very slow requests can degrade the overall experience.
• Common Causes: Garbage collection, resource contention, network variability, cold starts, and unpredictable OS scheduling can cause tail latency spikes.
• Mitigation: Techniques include efficient load balancing, request hedging, and over-provisioning resources to handle variance.

A Service Level Objective is a target for system reliability or performance that is agreed upon with users or stakeholders. For latency, an SLO is typically a percentile-based target.

• Structure: A latency SLO is often defined as "P99 latency shall be less than 200ms for 99.9% of requests over a 30-day window."
• Purpose: SLOs provide a clear, measurable goal for engineering teams, guiding capacity planning, optimization efforts, and release criteria.
• Related Concept: Service Level Indicators (SLIs) are the actual measured metrics (like P99 latency) used to evaluate compliance with the SLO.

Hardware utilization measures how effectively computational resources (GPU, CPU, memory bandwidth) are used during inference. It is a key efficiency metric that indirectly impacts latency and cost.

• Compute-Bound vs. Memory-Bound: A compute-bound workload saturates the processor's FLOPs, while a memory-bound workload is limited by data movement. Inference is often memory-bound due to attention mechanisms.
• The Roofline Model: This analytical model plots attainable performance against operational intensity, showing whether a kernel is limited by compute peak or memory bandwidth.
• Latency Connection: Low utilization can indicate inefficiency but may also allow for lower latency due to lack of contention. High utilization is cost-effective but can increase queuing delay.