P95 Latency

P95 Latency, or the 95th Percentile Latency, is the value below which 95% of all observed request latencies fall. It is calculated by:

1. Collecting latency measurements for all tool calls over a period.
2. Sorting these latencies from fastest to slowest.
3. Selecting the value at the 95th percentile of this sorted list.

For example, if you have 1000 latencies sorted, the P95 is the 950th slowest value. This metric is inherently non-linear and is heavily influenced by the slowest 5% of requests, making it sensitive to tail-end performance degradation.

Unlike average (mean) latency, which can be skewed by outliers, or median (P50) latency, which shows the typical case, P95 specifically measures the tail of the distribution. This is crucial because:

• User Experience: The slowest requests are often the most memorable and frustrating for end-users.
• System Health: Consistently high P95 latency can indicate underlying systemic issues like resource contention, garbage collection pauses, or network congestion that aren't visible in averages.
• SLO Definition: Service Level Objectives (SLOs) for reliability are often defined using P95 or P99 latency to guarantee a quality experience for the vast majority of requests.

P95 is part of a family of percentile metrics that paint a complete picture of latency distribution:

• P50 (Median): The middle value. 50% of requests are faster, 50% are slower. Represents the 'typical' experience.
• P90: 90% of requests are at or below this latency. A less strict measure than P95.
• P95: The standard for measuring performance outliers in many production systems.
• P99: 99% of requests are at or below this latency. Measures the extreme tail, critical for high-performance applications.

Monitoring the spread between P50 and P95 (e.g., P95 is 10x P50) is often more informative than any single metric, revealing latency variability.

Elevated P95 latency in tool call instrumentation typically stems from issues that affect a subset of requests:

• Noisy Neighbors: Contention for shared resources (CPU, network, database) on the host or in the cloud.
• External API Variability: Inconsistent response times from third-party services, which may have their own performance tails.
• Garbage Collection: Major garbage collection events in managed runtimes (e.g., JVM Full GC) that pause all threads.
• Cache Misses: Requests that bypass hot caches and require expensive computation or data fetching.
• Retries and Timeouts: The Exponential Backoff from a Retry Policy on failed calls directly adds to latency for affected requests.
• Serialization/Deserialization: Large or complex payloads can cause sporadic slowdowns.

Effective observability requires tracking P95 latency with context:

• Time-Series Dashboards: Graph P95 latency alongside P50 and P99 to see distribution changes.
• Breakdown by Dimension: Slice P95 by Span Attributes like tool.name, http.status_code, or user.id to identify problematic services or users.
• Alerting on SLO Violations: Set alerts based on Service Level Objectives (SLOs) defined for P95 latency. Use the Error Budget to determine alert thresholds.
• Correlation with Traces: When P95 spikes, use Distributed Tracing to sample slow Trace records and inspect the detailed Span timeline to pinpoint the root cause.

Using average latency for capacity planning can lead to under-provisioning. Because P95 latency is higher and more variable, it is a better metric for determining the resources needed to handle peak load while maintaining performance.

• Queueing Theory: As system utilization increases, latency increases non-linearly. The P95 will rise much faster than the mean.
• Provisioning Target: Systems should be provisioned to keep P95 latency within SLO bounds at expected load, not just to keep the average low.
• Cost Implications: Ignoring P95 can result in a system that meets average latency targets but fails frequently under real-world load, leading to user churn and increased operational burden.