Service Level Objective (SLO)

A Service Level Indicator is the precise, quantitative measurement of a service's performance upon which an SLO is based. It is the raw metric.

• Examples: Request latency (p99), error rate (5xx responses / total requests), throughput (requests per second), availability (successful requests / total requests).
• Key Property: Must be measurable, well-defined, and directly tied to user experience. An SLI answers the question: "What exactly are we measuring?"

An SLO combines an SLI with a target value over a defined time window. This creates the formal objective.

• Target: The desired performance level, expressed as a percentage or threshold (e.g., "99.9%", "< 200ms p95 latency").
• Time Window: The rolling period over which compliance is measured (e.g., 28 days, 30 days). This prevents short-term spikes from masking long-term trends and aligns with typical business cycles.
• Example: "The proportion of successful HTTP requests, measured over a rolling 28-day window, must be at least 99.95%."

An Error Budget is the explicit, calculated amount of unreliability a service team is allowed within an SLO's time window. It is derived directly from the SLO.

• Calculation: Error Budget = 1 - SLO Target. For a 99.9% SLO, the error budget is 0.1% of the total possible measurement units in the time window.
• Purpose: It quantifies risk and drives prioritization. Spending the budget on releases or experiments is acceptable; exhausting it triggers a focus on stability and reliability work.
• Core Concept: It transforms reliability from an abstract goal into a consumable resource for managing innovation velocity.

Burn Rate measures how quickly a service is consuming its error budget. It is a critical metric for understanding the urgency of a reliability issue.

• Definition: The speed at which errors are accumulating relative to the total budget for the time window. A burn rate of 1.0 means the budget will be exhausted exactly at the end of the window.
• High Burn Rate: A burn rate > 1.0 (e.g., 5.0, 10.0) indicates a severe incident that will exhaust the budget in hours or days, requiring immediate action.
• Use Case: It enables alerting on SLOs based on the time-to-exhaustion of the budget, rather than on static thresholds, leading to more actionable and user-impact-focused alerts.

Effective SLO implementation requires alerting based on the rate of budget consumption, not on momentary SLI violations. This prevents alert fatigue and focuses attention on user-impacting trends.

• Multi-Window, Multi-Burn-Rate Alerts: A common pattern uses two alerts:
  • Warning Alert: Triggered by a moderate burn rate (e.g., 3.0) over a shorter window (e.g., 1 hour). Signals investigation.
  • Critical Alert: Triggered by a high burn rate (e.g., 10.0) over a longer window (e.g., 6 hours). Signals imminent budget exhaustion and requires immediate remediation.
• Philosophy: "Alert on symptoms, not causes." The symptom is the rapid consumption of the error budget allocated for user happiness.

In a microservices architecture, SLOs are not isolated. They form a hierarchy based on service dependencies, which is crucial for understanding system-wide reliability.

• Composite SLOs: User-facing SLOs (e.g., for an API endpoint) are often dependent on the SLOs of underlying microservices, databases, and third-party APIs. The composite reliability is a function of all dependent components.
• Dependency Analysis: Identifying critical dependencies allows teams to set appropriate SLOs for internal services and negotiate SLAs with external providers.
• Implication: A failure in a low-level service with a tight SLO can rapidly exhaust the error budget of many upstream, user-facing services.

A Service Level Indicator is a direct, quantitative measure of a specific aspect of a service's performance or reliability. It is the raw metric used to evaluate compliance with an SLO.

• Examples: Request latency (p99), error rate (successful requests / total requests), throughput (requests per second), availability (uptime percentage).
• Relationship to SLO: An SLO is a target value or range for an SLI. For instance, an SLO might state "p99 latency < 200ms," where the SLI is the actual measured p99 latency.

A Service Level Agreement is a formal contract between a service provider and a customer that defines the guaranteed level of service, including consequences (like financial penalties) if the guarantees are not met.

• Key Difference from SLO: An SLO is an internal, aspirational goal for reliability. An SLA is an external, contractual promise. SLOs are typically set more aggressively than SLAs to provide a buffer (an error budget) before violating the SLA.
• Purpose: SLAs manage business risk and customer expectations, while SLOs guide internal engineering priorities.

An Error Budget is the explicit, quantified amount of unreliability a service team can tolerate over a specific period, derived directly from its SLOs.

• Calculation: If an SLO is 99.9% availability per month, the error budget is 0.1% of that time, or approximately 43.2 minutes of downtime.
• Engineering Function: The error budget frames outages and SLO misses not as pure failures, but as a resource to be spent. It creates a shared, objective metric for balancing the pace of innovation (releasing new features) against reliability (avoiding errors). Spending the budget may trigger a freeze on new feature releases to focus on stability.

The Circuit Breaker pattern is a fault-tolerance design pattern that prevents an application from repeatedly attempting to execute an operation that is likely to fail. It acts as a proxy for operations, monitoring for failures and "opening the circuit" to fail fast after a threshold is breached.

• States: Closed (requests pass through), Open (requests fail immediately), Half-Open (a trial request is allowed to test if the underlying service has recovered).
• SLO Connection: Circuit breakers are a tactical implementation to protect SLOs. By failing fast on a dependent service failure, they prevent cascading failures and resource exhaustion that could violate the system's own latency or error rate SLOs.

Chaos Engineering is the disciplined practice of proactively injecting failures into a system in production to build confidence in the system's capability to withstand turbulent conditions.

• Methodology: Hypothesize about steady state (often defined by SLIs), introduce real-world failure events (e.g., terminate instances, inject latency, corrupt packets), and observe if the system's SLIs deviate from the SLO.
• SLO Relationship: Chaos experiments are a rigorous method of validating that SLOs are meaningful and that the system's resiliency mechanisms (like circuit breakers and retries) actually work as designed under failure conditions.

A Canary Deployment is a release strategy where a new version of an application is deployed to a small, representative subset of users or servers first. Its performance is monitored against key SLIs before a full rollout.

• Risk Mitigation: Limits the "blast radius" of a bad release. If the canary's error rate or latency SLIs degrade, violating SLO expectations, the rollout can be halted and rolled back, minimizing user impact.
• SLO as Gate: SLO compliance often serves as the automated gate for a canary promotion. If the canary's SLIs remain within SLO bounds for a defined period, the release is considered safe to proceed.