Service Level Objective (SLO)

An SLI is the specific, measurable metric used to quantify a service's reliability. It is the raw measurement upon which an SLO is based.

• Examples: Request latency (p99), error rate (successful requests / total requests), throughput (requests per second), availability (uptime).
• For LLMs: Common SLIs include token generation latency, end-to-end request success rate (factoring in context window limits and timeouts), and hallucination rate as measured by an evaluation pipeline.
• Key Property: Must be a direct, user-centric measure of service quality, not an internal system metric like CPU utilization.

The SLO target is the numerical goal for the SLI, expressed as a percentage or threshold over a compliance period (e.g., 30 days). The error budget is the derived, allowable amount of unreliability.

• Calculation: If an SLO is 99.9% availability, the error budget is 0.1% failure. Over 30 days (43,200 minutes), the budget is 43.2 minutes of downtime.
• Primary Use: The error budget is a resource for innovation. It quantifies how much risk a team can take with new releases. Exhausting the budget triggers a focus on stability over new features.
• LLM Consideration: Targets must account for non-deterministic behavior. A 99.5% success rate for a complex agentic workflow may be ambitious, whereas 99.95% for a simple classification endpoint is standard.

The compliance period is the rolling time window over which the SLO is evaluated (e.g., 28 days). Burn rate measures how quickly the error budget is being consumed.

• Critical Insight: A fast burn rate indicates an imminent breach. A burn rate of 10 means the error budget is being used 10x faster than if failures were evenly distributed over the period.
• Alerting Strategy: SLO-based alerting often uses burn rates. For example, alert if the 6-hour burn rate exceeds 5, signaling a rapid degradation requiring immediate investigation, rather than alerting on a momentary SLI dip.
• LLM Context: Useful for detecting gradual degradation, like a creeping increase in latency due to prompt chain complexity or a slow rise in 'refusal' rates post-safety fine-tuning.

SLOs should reflect the reliability of critical user journeys, not just isolated endpoints. This requires careful aggregation of SLIs across services.

• Example: An LLM-powered customer support chat's SLO might be defined for the journey: "User query → Intent classification → Knowledge retrieval → Response generation → Sentiment-positive output." Failure at any step fails the journey.
• Aggregation Methods: SLIs can be aggregated by weighting different endpoints (e.g., login API is more critical than avatar upload) or by defining SLOs for specific API pathways.
• Avoiding Pitfalls: A service with 99.9% uptime on each of 10 dependent services does not yield a 99.9% user journey success rate due to compound probability.

Defining SLIs for LLM services requires metrics beyond traditional infrastructure health, capturing the unique failure modes of generative AI.

• Quality SLIs: Hallucination Rate (percentage of outputs with unsupported facts), Task Success Rate (measured via automated evaluation or human review sampling), Output Compliance Rate (adherence to formatting, safety, and policy rules).
• Performance SLIs: Time-to-First-Token (TTFT) and Inter-Token Latency for streaming responses. These are critical for user-perceived latency.
• Resource SLIs: Context Window Utilization and Cost-Per-Token can be used for internal efficiency SLOs, though they are not direct user-reliability metrics.

SLOs are not static documents; they are dynamic tools integrated into the deployment pipeline and observability stack.

• Progressive Delivery: SLO burn rates are the primary gating metric for canary deployments and traffic splitting. If the canary's error budget burn exceeds a threshold, the rollout is automatically halted or rolled back.
• Observability: SLO status must be a prominent dashboard visualization. Tools like Prometheus and Grafana (with SLI/SLO plugins) or commercial APM platforms are used to compute and display burn rates.
• LLM Observability: Requires integration with specialized LLM evaluation and monitoring platforms that can compute quality SLIs (e.g., hallucination rate) in near-real-time for SLO calculation.

For Large Language Model applications, reliability includes the quality of generated content. This SLO measures the factual accuracy or adherence to formatting rules.

• SLI Formula: (Number of responses passing a quality check) / (Total evaluable responses).
• Implementation: Uses a small, sampled traffic routed through a validation pipeline (e.g., a more powerful LLM judge, heuristic rules, or human evaluation).
• Example: A customer support chatbot may have an SLO that 95% of its answers are factually correct when evaluated against a known knowledge base.
• Key Challenge: Requires a robust, automated evaluation strategy to measure at scale without human-in-the-loop for every request.

Critical for public-facing generative AI, this SLO sets a target for filtering harmful, biased, or non-compliant content before it reaches users.

• SLI Formula: (Number of unsafe responses detected) / (Total responses).
• Common Target: < 0.1% of responses contain undetected harmful content over a rolling week.
• Implementation: Relies on a dedicated moderation layer (a separate classifier or model) that screens all outputs before delivery.
• Example: A creative writing assistant has an SLO that 99.97% of its generated text passes its safety filter, minimizing the risk of producing violent or explicit content.

A Service Level Indicator (SLI) is the specific, quantitative metric used to measure a service's performance against a Service Level Objective (SLO). An SLO is the target, while the SLI is the actual measurement.

• Examples for an LLM API: Request latency (p95, p99), successful response rate (non-error HTTP status codes), token generation throughput.
• Key Property: Must be a direct, measurable reflection of user experience. For instance, using error rate is more user-centric than internal server CPU utilization.

A Service Level Agreement (SLA) is a formal contract between a service provider and a customer that specifies the guaranteed level of service, including consequences (like financial penalties) if the SLOs are not met. The SLO is the internal target; the SLA is the external promise.

• Relationship to SLO: Internal SLOs are typically set more aggressively than the SLA (e.g., SLO at 99.9% uptime, SLA at 99.5%) to create a reliability buffer.
• Focus: SLAs are business and legal documents, while SLOs are engineering tools for guiding development and operational priorities.

An Error Budget is the calculated amount of unreliability a service can tolerate over a period (e.g., a month) before violating its SLO. It is derived directly from the SLO.

• Calculation: If the monthly SLO is 99.9% availability, the error budget is 0.1% of the time, or approximately 43.2 minutes of downtime per month.
• Primary Use: It quantifies risk and enables data-driven decisions. Spending the budget on a risky release is a conscious trade-off. Preserving the budget may halt new feature work in favor of stability improvements.

Canary Deployment is a release strategy where a new version of a service is deployed to a small, controlled subset of production traffic. Its performance (monitored via SLIs) is compared against the stable version before a full rollout.

• SLO Integration: The canary's SLI metrics (latency, error rate) are monitored in real-time. If they degrade and threaten the error budget, the rollout is automatically halted and rolled back.
• Purpose: This strategy directly uses SLOs as the gating mechanism for releases, preventing bad deployments from consuming the entire error budget.

Multi-Region Deployment is an architectural pattern where an application and its data are replicated across geographically dispersed cloud regions. This is a primary strategy for achieving high-availability SLOs and providing disaster recovery.

• Impact on SLOs: It protects against region-wide failures. SLIs like availability and latency are often measured per region and globally.
• Complexity: Introduces challenges for data consistency, synchronization latency, and global traffic management, all of which must be factored into SLO definitions and measurements.

Chaos Engineering is the discipline of proactively injecting failures into a production system to test its resilience and validate the effectiveness of monitoring, alerting, and recovery procedures.

• Relationship to SLOs: Experiments are designed to test hypotheses about how the system behaves under stress and whether SLO violations are detected and responded to appropriately.
• Proactive Budget Management: By discovering weaknesses in a controlled manner, chaos engineering helps prevent unexpected failures that would consume the error budget.