Latency Percentiles (P50, P90, P99)

The P50 latency, or the median, is the value below which 50% of all observed response times fall. It represents the typical user experience. For example, if an LLM's P50 latency is 250ms, half of all requests complete in 250ms or less. This metric is useful for understanding the central tendency of your system's performance but can mask poor experiences from outliers. It is often compared to the arithmetic mean, but unlike the mean, the median is not skewed by extremely high-latency requests.

The P90 latency is the maximum response time experienced by the fastest 90% of requests. It is a critical benchmark for user satisfaction, as it captures the experience for the vast majority of users, excluding the worst 10%. A high P90 indicates that a significant portion of your user base is experiencing noticeable delays. For LLM services, optimizing P90 often involves addressing common bottlenecks like database queries, external API calls, or moderate queueing in the inference pipeline. Monitoring P90 helps ensure consistent quality of service.

The P99 latency measures the maximum latency for the fastest 99% of requests, exposing the experience of the slowest 1%. This 'tail latency' is where the most severe bottlenecks and system pathologies appear. Causes can include:

• Garbage collection pauses in the runtime environment.
• Cold starts for newly scaled inference instances.
• Resource contention on shared hardware (e.g., noisy neighbors in cloud environments).
• Extreme input/output sequence lengths that stress computational limits. Engineering for low P99 is essential for high-performance, reliable LLM services, as these outliers often correspond to critical user requests.

Using simple average (mean) latency is misleading for LLM performance because it can be dramatically skewed by a small number of very slow requests. A system with a 100ms average could still have a P99 of 5 seconds, meaning 1% of users have a terrible experience. Percentiles provide a complete picture of the latency distribution. They allow engineers to set meaningful Service Level Objectives (SLOs) based on user experience (e.g., '95% of requests under 500ms') rather than infrastructure averages. This percentile-based thinking is fundamental to site reliability engineering.

Accurate latency percentile calculation requires capturing high-resolution timing data for every request. This is typically done via:

• Structured logging with nanosecond-precision timestamps.
• Distributed tracing using frameworks like OpenTelemetry to track requests across services.
• Metrics systems like Prometheus with histograms (e.g., http_request_duration_seconds_bucket). These tools aggregate request durations into buckets, enabling the calculation of any percentile (e.g., via Prometheus's histogram_quantile function). The data is then visualized on Grafana dashboards with percentile overlays to monitor trends and SLO compliance over time.

Latency percentiles provide the overall picture, but specific LLM inference stages have their own critical timing metrics:

• Time to First Token (TTFT): The latency from request start to the first token streamed back. Crucial for perceived responsiveness.
• Inter-Token Latency: The average time between subsequent tokens during streaming. Impacts the fluency of the output.
• End-to-End Latency: The total time for a complete, non-streamed response. Effective monitoring involves tracking percentiles (P50, P90, P99) for each of these distinct metrics to pinpoint bottlenecks—whether in the initial prefill computation (affecting TTFT) or the autoregressive decoding loop (affecting inter-token latency).

P99 latency is the primary signal for infrastructure scaling decisions. A rising P99 indicates the system is approaching a saturation point, even if P50 remains stable.

• Autoscaling triggers are often based on P90 or P99 thresholds. For example, scale out GPU instances when P99 exceeds 2 seconds.
• Resource provisioning for burst traffic is sized to handle the expected P99 load, not the median. This prevents tail latency spikes during peak usage.
• Analyzing the gap between P50 and P99 helps identify optimization opportunities. A large gap suggests irregular bottlenecks (e.g., long prompts, cold starts) rather than uniformly slow hardware.

Different percentiles point to different classes of problems. Segmenting latency by percentile is a first-step in root cause analysis (RCA).

• A high P50 indicates a systemic, widespread slowdown affecting most requests (e.g., overloaded GPU, inefficient model kernel).
• A high P99 with normal P50/P90 is a classic tail latency problem. This points to specific, irregular issues:
  • Cold starts for new model replicas.
  • Long-context prompts triggering expensive attention computations.
  • Noisy neighbor effects in multi-tenant clusters.
  • Downstream API timeouts or retries.

The efficacy of inference optimizations is measured by their impact across the latency distribution. Some techniques improve median latency but worsen the tail.

• Continuous batching dramatically improves P50 and P90 throughput but requires careful management to avoid increasing P99 latency for requests that get stuck behind very long sequences.
• KV Cache optimization and attention slicing directly target the computational bottlenecks that cause high P99 for long sequences.
• Model quantization (e.g., FP16 to INT8) typically improves P50 latency uniformly but must be validated to ensure it doesn't introduce outliers (degraded P99) on edge-case inputs.

When rolling out a new model version or infrastructure change, percentile latency is a key comparative metric. Canary and shadow deployments rely on this analysis.

• In a canary deployment, the new version's P90/P99 latency is compared in real-time against the baseline. A regression triggers an automatic rollback.
• Shadow deployments allow full comparison of latency distributions without user impact. The question isn't just "is the median faster?" but "did we introduce more latency outliers (worse P99)?"
• Cohort analysis segments latency by user tier, model size, or prompt type to understand differential performance impacts.

Engineering leaders use latency percentiles to make informed decisions about the cost-performance Pareto frontier.

• Running a model on more powerful (expensive) hardware may significantly improve P99 but only marginally improve P50. The business case depends on the SLO.
• Using a smaller, distilled model might increase P50 latency slightly but provide a more predictable, lower P99, leading to better overall user experience and simpler capacity planning.
• Multi-model routing systems can use real-time P95 latency estimates to route requests to the optimal backend (e.g., fast-track simple queries to a smaller model, reserve large models for complex tasks).