Service Level Objective (SLO)

A Service Level Indicator (SLI) is the specific, quantifiable measurement of a service's behavior that an SLO targets. It is the raw metric from which reliability is calculated. An SLI must be precisely defined, consistently measurable, and directly tied to user experience.

• Examples: Availability (successful requests / total requests), Latency (p95 response time), Throughput (requests per second), Error Rate (failed requests / total requests).
• Key Property: An SLI is often expressed as a ratio, rate, or percentile over a defined measurement window.

Every SLO combines a numerical target with a measurement window. The target defines the acceptable level of service (e.g., 99.9%), while the window defines the period over which compliance is evaluated (e.g., 30 days). This pairing creates a concrete, time-bound commitment.

• Target Examples: "99.95% availability" or "p95 latency < 200ms."
• Window Examples: Rolling 28-day window, calendar month, or weekly sprint. The choice of window impacts how quickly an SLO reflects changes in service health.

An Error Budget is the calculated, allowable amount of unreliability derived from an SLO. It is defined as 1 - SLO target. If an SLO is 99.9% availability over a month, the error budget is 0.1% of that month's total time—approximately 43.2 minutes. This budget is a powerful management tool.

• Purpose: It quantifies the risk a team can take. Consuming the budget on planned changes (like deployments) is acceptable; burning it on unplanned outages is not.
• Management: Teams can trade error budget for development velocity, using it to decide when to prioritize reliability work over new features.

The Burn Rate measures how quickly the error budget is being consumed. It is critical for moving from passive monitoring to proactive alerting. A fast burn rate indicates an urgent problem, while a slow burn rate might only require investigation.

• Calculation: Error Budget Consumed / Time Period.
• Alerting Strategy: Implement multi-window, multi-burn-rate alerts (e.g., alert if burning budget 10x faster for 1 hour OR 2x faster for 6 hours). This prevents alert fatigue while ensuring timely response to significant incidents before the budget is exhausted.

Effective SLOs are defined from the user's perspective, measuring what matters to their experience, not just internal system health. This requires identifying critical user journeys and instrumenting SLIs that reflect their success or failure.

• Example: For an e-commerce service, an SLO might target the "checkout" journey's availability, not just the uptime of the payment service's API endpoint. A user can't complete a purchase if the cart service is down, even if the payment API is healthy.
• Benefit: This focus aligns engineering efforts with business outcomes and customer satisfaction.

SLOs must be clearly documented and subject to a regular review cadence. Documentation should include the rationale, measurement methodology, and responsible parties. Regular reviews ensure SLOs remain relevant as the service and user expectations evolve.

• Documentation Elements: SLI query/calculation, SLO target and window, error budget policy, alerting rules, and escalation paths.
• Review Cadence: Typically aligned with product planning cycles (e.g., quarterly). Reviews assess if SLOs are too loose (masking problems), too tight (stifling innovation), or misaligned with current user behavior.

An error budget is the explicit, quantified amount of unreliability a service team is allowed within a defined period, calculated directly from its Service Level Objectives (SLOs). It is the cornerstone of a data-driven approach to balancing reliability with development velocity.

• Purpose: Provides a clear, shared metric for deciding when to prioritize stability work (e.g., bug fixes, infrastructure upgrades) over feature development.
• Calculation: If an SLO is 99.9% availability per month, the error budget is 0.1% of the time, or approximately 43.2 minutes of allowed downtime.
• Management: Teams can "spend" their budget on deployments and changes. Exhausting the budget triggers a blameless post-mortem and often a freeze on new feature releases until reliability is restored.

A Service Level Indicator (SLI) is a direct, quantitative measure of a specific aspect of a service's performance or reliability. An SLO is a target value or range for an SLI.

• Relationship to SLO: An SLI is the measurement, an SLO is the goal. For example:
  • SLI: The proportion of successful HTTP requests (success rate).
  • SLO: The success rate must be ≥ 99.95% over a 30-day rolling window.
• Common SLIs: Include availability (uptime), latency (response time), throughput (requests per second), and quality (error rate).
• Implementation: SLIs are derived from metrics collected via monitoring systems like Prometheus, Datadog, or OpenTelemetry, and must be defined with precise query logic.

A Service Level Agreement (SLA) is a formal, often contractual, commitment between a service provider and its customers that specifies the consequences (typically financial penalties or service credits) if the published Service Level Objectives (SLOs) are not met.

• Key Distinction: An SLO is an internal, engineering-focused reliability target. An SLA is the external, business-facing promise with associated liabilities.
• Typical Structure: An SLA will reference one or more SLOs (e.g., "Monthly Uptime Percentage of 99.9%") and define the remedy (e.g., "10% service credit") for missing it.
• Strategic Buffer: Engineering teams often set internal SLOs more strictly than the published SLA (e.g., SLO at 99.95% vs. SLA at 99.9%) to create a reliability margin and avoid triggering penalties.

Mean Time To Recovery (MTTR) is a critical reliability metric that measures the average time taken to restore a service to normal operation after a failure or incident is detected. It directly impacts error budget consumption.

• Components: MTTR encompasses the time to detect, respond, diagnose, mitigate, and resolve an incident.
• SLO Impact: A high MTTR means failures consume the error budget faster, making SLO violations more likely. Improving MTTR is often a primary reliability initiative.
• Reduction Strategies: Effective monitoring, on-call playbooks, automated rollbacks, and canary deployments are all designed to reduce MTTR.

The circuit breaker pattern is a resilience design pattern that prevents an application from repeatedly attempting to call a failing dependency (e.g., a downstream API), allowing it time to recover and preventing cascading failures.

• Mechanism: It operates like an electrical circuit breaker with three states:
  • Closed: Requests flow normally.
  • Open: Requests fail immediately without calling the dependency.
  • Half-Open: A limited number of test requests are allowed to probe for recovery.
• SLO Protection: By failing fast on calls to unhealthy dependencies, circuit breakers protect the calling service's own latency and availability SLIs, helping it meet its SLOs even when dependencies are unstable.
• Implementation: Commonly implemented via libraries like Resilience4j (Java) or Polly (.NET).

Chaos engineering is the disciplined practice of proactively injecting failures into a system in a controlled, production-like environment to test and improve its resilience and validate its ability to meet Service Level Objectives (SLOs) under stress.

• Purpose: To uncover systemic weaknesses and validate reliability hypotheses before they cause customer-impacting incidents and consume the error budget.
• Common Experiments: Injecting latency, killing processes (chaos monkey), shutting down network zones, or filling up disks.
• Process: The cycle is: 1) Define a steady state (often measured by SLIs), 2) Formulate a hypothesis (e.g., "The system will maintain latency SLO if Database-A fails"), 3) Run the experiment, 4) Analyze the impact on SLIs and SLOs.