Service Level Objective (SLO)

An SLI is the specific, measurable metric that quantifies an aspect of service reliability. For LLMs, common SLIs include:

• Latency Percentiles (P50, P90, P99) for request completion or Time to First Token (TTFT).
• Availability, measured as the proportion of successful requests (non-5xx HTTP status codes).
• Throughput, such as Tokens per Second (TPS).
• Quality, using metrics like output correctness scores or low hallucination rates. The SLI must be precisely defined, including its measurement method and aggregation window.

This defines the numerical goal and the time period over which compliance is evaluated.

• Target: A specific value or range (e.g., "99.9% of requests have latency < 500ms", "Availability >= 99.95%").
• Measurement Window: The rolling period for calculating the SLI, such as 28 or 30 days. This window size balances responsiveness to issues with statistical significance, preventing transient blips from violating the SLO. The SLO is considered "met" if the SLI's value over the entire window meets the target.

The error budget is the allowable amount of unreliability, derived directly from the SLO. It is calculated as 1 - SLO_target. For a 99.9% monthly availability SLO, the error budget is 0.1%, or approximately 43.2 minutes of downtime per month. This budget:

• Quantifies risk, providing a clear, shared resource for the engineering team.
• Governs velocity, allowing teams to spend the budget on risky changes (like model deployments) or must conserve it after an incident.
• Drives prioritization, making reliability work data-driven by tracking budget consumption.

To proactively manage the error budget, SLOs require alerting on the burn rate—how quickly the budget is being consumed.

• Fast Burn Alerts trigger when a high error rate consumes a significant portion (e.g., 5%) of the budget in a short period (e.g., 1 hour), indicating a severe, urgent incident.
• Slow Burn Alerts trigger when a moderate error rate consumes the budget over a longer period (e.g., days), signaling a chronic degradation that requires attention. This approach focuses alerts on user-impacting reliability, reducing alert fatigue from non-SLO-related metric noise.

Defining SLIs for LLM services involves unique challenges beyond traditional APIs:

• Multi-Stage Latency: Differentiating Time to First Token (TTFT) (perceived latency) from inter-token latency (streaming fluency).
• Quality vs. Speed: Balancing latency SLOs with quality SLIs like output correctness or low hallucination rates, which may require Human-in-the-Loop (HITL) sampling or automated scoring against a golden dataset.
• Non-Functional Errors: Defining failures to include not just HTTP 5xx codes, but also safety filter violations, excessive output truncation, or severe output drift from a baseline.

Effective SLOs require robust instrumentation and observability tooling.

• Measurement: SLIs are computed from high-cardinality metrics and distributed traces collected via frameworks like OpenTelemetry (OTel).
• Aggregation & Storage: Time-series databases like Prometheus store raw metrics for SLI calculation.
• Visualization & Dashboards: Grafana dashboards display SLO status, burn rate, and remaining error budget.
• Integration: SLO status informs canary deployment decisions and root cause analysis (RCA) processes, linking reliability directly to operational workflows.

A Service Level Indicator is a quantitatively measured aspect of an LLM service's performance that directly informs an SLO. An SLI is the raw measurement, while an SLO is the target for that measurement.

• Examples for LLMs: Request latency (P99), successful request rate (non-5XX errors), token throughput (Tokens per Second), or output quality score.
• Implementation: SLIs are derived from metrics collected via observability tools like Prometheus or OpenTelemetry. They must be specific, measurable, and representative of user happiness.

An Error Budget is the calculated, allowable amount of unreliability an LLM service can incur over a defined period (e.g., one month) before violating its SLO. It quantifies risk and drives engineering priorities.

• Calculation: If an SLO is 99.9% availability monthly, the error budget is 0.1% unreliability, or approximately 43 minutes of downtime.
• Usage: Teams consume the budget during incidents or degradations. A depleted budget should halt feature deployments and trigger a focus on stability and reliability work.

A Service Level Agreement is a formal, contractual commitment between a service provider and a customer that specifies guaranteed levels of service, often backed by financial penalties or service credits for violation.

• Relationship to SLO: An SLO is an internal, engineering-focused target. An SLA is the external, business-facing promise. Teams typically set SLOs more aggressively than SLAs to provide a safety buffer.
• LLM Context: An SLA might guarantee 99.5% uptime for an LLM API, while the internal SLO is set at 99.8% to ensure the contract is consistently met.

Latency Percentiles are statistical measures describing the distribution of response times. They are critical for defining SLOs for LLM latency, as user experience is often dictated by worst-case (tail) performance.

• P50 (Median): The latency below which 50% of requests fall. Often close to the average.
• P90 / P99 (Tail Latency): The maximum latency experienced by 90% or 99% of requests. P99 is crucial for SLOs as it captures the slowest 1% of user experiences, which can be caused by model cold starts, long sequences, or resource contention.
• SLO Example: "99% of LLM chat completions must return a first token within 2 seconds (P99 TTFT < 2s)."

Mean Time to Recovery is a key reliability metric measuring the average time taken to restore an LLM service to normal operation after a failure or SLO-violating degradation is detected.

• Components: MTTR includes time to detection, diagnosis (see Root Cause Analysis), mitigation, and full remediation.
• SLO Context: While SLOs define how reliable a service should be, MTTR defines how quickly it should be fixed when it becomes unreliable. A robust monitoring system aims to minimize MTTR through effective alerting and runbooks.

Canary and Shadow Deployments are release strategies used to validate new LLM models or application versions against performance SLOs with minimal user risk.

• Canary Deployment: A new version serves a small percentage of live traffic. Its SLIs (latency, error rate) are closely monitored and compared to the baseline. If SLOs are met, traffic is gradually increased.
• Shadow Deployment: The new version processes all requests in parallel with the primary version, but its outputs are discarded. This allows for full-scale performance and correctness testing (e.g., checking for output drift) with zero user impact before a cutover.