Service Level Objective (SLO)

An SLO is defined by a specific, measurable metric and a threshold. It is not a vague aspiration but a hard number that leaves no room for interpretation.

• Metric: The specific indicator being measured (e.g., P99 latency, availability, error rate).
• Threshold: The boundary value (e.g., ≤ 200ms, ≥ 99.9%).
• Measurement Window: The rolling period over which compliance is calculated (e.g., 28 days).

Example: '99% of all search queries over the last 30 days will complete in under 150 milliseconds.'

The SLO directly creates an error budget, which is the maximum amount of time a system can fail or degrade without breaching the contract. This budget is 1 minus the SLO target.

• Formula: Error Budget = 1 - SLO (e.g., a 99.9% availability SLO allows for 0.1% downtime).
• Release Valve: If the error budget is exhausted, all feature launches are frozen, and engineering effort pivots entirely to reliability work.
• Innovation Token: A remaining budget is a quantified token that allows teams to take risks and push new code to production.

An effective SLO is defined from the user's perspective, not the server's. It measures the experience at the edge, not just internal health checks.

• Key Principle: Measure what the user sees. If the server is up but the database is slow, the user experiences an error.
• Black-Box Monitoring: Use probes and metrics that mimic a real user's journey rather than relying solely on white-box metrics like CPU usage.
• Example: An SLO for a chat application should measure Time-to-First-Token (TTFT) and end-to-end response latency, not just the uptime of the inference server.

An SLO is an internal engineering target, while a Service Level Agreement (SLA) is an external business contract with customers that includes financial penalties for breach.

• SLO: Aggressive, internal goal. 'We aim for 99.95% availability.'
• SLA: Conservative, external promise. 'We guarantee 99.9% availability or you receive a service credit.'
• Safety Buffer: The SLO should always be more stringent than the SLA. This gap provides a crucial early-warning buffer, allowing engineering teams to fix issues before customers notice a breach of the contractual SLA.

An SLO is only useful if it is continuously measured and surfaced. This requires a robust observability pipeline that calculates compliance over the defined rolling window.

• SLI is the Input: The Service Level Indicator (SLI) is the raw measurement (e.g., the actual P99 latency measured over 1 minute). The SLO is the target applied to that SLI.
• Burn Rate Alerts: Configure alerts on how quickly the error budget is being consumed, not on a simple threshold breach. A burn rate of 1 means the budget will be exhausted exactly by the end of the window.
• Multi-Window Burn Alerts: Use a short window (e.g., 1 hour) for rapid burn detection and a long window (e.g., 24 hours) to filter out noise, preventing alert fatigue.

An SLO must be simple enough for every engineer to understand and defend. Overly complex SLOs become ignored and unactionable.

• The 3-5 Rule: Start with no more than 3 to 5 critical SLOs per service. Any more dilutes focus.
• SLI Menu: Provide application teams with a standard menu of SLIs (latency, availability, freshness, correctness) to choose from, rather than letting them invent bespoke metrics.
• Defensibility Test: Can an engineer explain in 30 seconds why the SLO is set at this specific number and how it protects the user experience? If not, it's too complex.

The 99th percentile of request latency—the response time threshold below which 99% of all requests fall. This metric captures the worst-case user experience that still falls within the SLO's defined acceptable range.

• P99 is preferred over averages because averages mask tail latency
• An SLO might specify: "P99 retrieval latency < 200ms over a 30-day rolling window"
• P99.9 and P99.99 are used for systems with extreme consistency requirements
• Monitoring P99 directly informs whether the error budget is being consumed

The high-latency outliers in a distribution of service response times, typically measured at the 90th, 99th, or 99.9th percentile. Tail latency is the primary adversary of SLO compliance in retrieval systems because a single slow sub-query can stall the entire answer generation pipeline.

• Caused by resource contention, garbage collection pauses, or degraded hardware
• Amplified in fan-out architectures where the slowest shard dictates total response time
• Mitigation strategies include hedged requests and speculative retrieval

The elapsed time between a user submitting a query and the language model emitting the first token of the response. TTFT is a critical perceived performance metric for SLOs in generative retrieval systems.

• Users perceive a system as responsive when TTFT is under 200ms
• TTFT encompasses embedding generation, ANN search, re-ranking, and prompt construction
• Distinct from total generation time, which includes token-by-token streaming
• SLOs often specify separate targets for TTFT and tokens-per-second throughput

The percentage of data requests successfully served from a caching layer rather than the primary data store. This metric is a primary lever for meeting latency SLOs because cache hits bypass expensive retrieval and inference.

• A semantic cache can serve paraphrased queries with near-identical embeddings
• High cache hit ratios (>90%) dramatically reduce P99 tail latency
• Cache hit ratio degradation directly threatens SLO compliance
• Monitoring this ratio provides early warning before error budgets are consumed