Error Budget

The Service Level Objective (SLO) is the foundation. An error budget is calculated as 100% - SLO. For example, a 99.9% availability SLO over a 30-day quarter creates a budget of 0.1% allowable unreliability, equating to approximately 43.2 minutes of total downtime the service can incur. This quantifies the explicit risk the business accepts.

The burn rate measures how quickly the error budget is being consumed. It's expressed as a multiple of the budget consumption rate needed to exhaust it within a specific timeframe.

• A burn rate of 1.0 consumes the budget evenly over the SLO period.
• Multi-window alerting uses different burn rates (e.g., fast burn over 1 hour, slow burn over 6 hours) to distinguish between brief incidents and sustained degradation, reducing alert fatigue while protecting the SLO.

The error budget must be governed by clear policies that dictate how it is spent and what happens when it is exhausted. Key policies include:

• Budget Spend Decisions: Defining what constitutes valid consumption (e.g., planned risk for feature launches, unplanned incidents).
• Exhaustion Protocols: Actions triggered when the budget is depleted, such as a feature freeze where only reliability-focused work is permitted until the budget is replenished in the next period.
• Escalation & Review: Mandating post-mortems for significant budget consumption.

For AI services, the error budget is consumed by failures against quality SLIs, not just infrastructure uptime. Critical AI SLIs include:

• Model Inference Latency (p95/p99)
• Hallucination Rate (proportion of factually incorrect outputs)
• Retrieval Precision@K (for RAG systems)
• Answer Faithfulness (to source context)
• Agent Task Success Rate Violations of these quality SLIs directly consume the service's error budget, linking model performance to operational risk.

Effective error budget management requires real-time, transparent tracking. This is typically implemented via dashboards that show:

• Remaining Budget: As a percentage and absolute time.
• Burn Rate: Current and historical consumption speed.
• Incident Attribution: Which events or deployments consumed budget. Tools like Google's SLO Toolkit, Sloth, or OpenSLO-compatible platforms automate this tracking, integrating with monitoring systems to calculate burn rates and forecast exhaustion.

The error budget is a key input to engineering prioritization and release processes:

• Risk Assessment: Before a deployment, teams estimate its potential budget consumption.
• Canary Deployments: Used to validate SLO compliance with a small traffic slice before full rollout, minimizing potential budget burn.
• Blame-Free Post-Mortems: Focused on systemic fixes rather than individual blame, using budget consumption data to justify reliability investments. This creates a feedback loop where the budget directly informs the trade-off between innovation velocity and system stability.

A Service Level Objective (SLO) is the quantitative reliability target that defines an error budget. It is expressed as a percentage over a time window (e.g., 99.9% availability per month). The error budget is calculated as 100% - SLO. An SLO is the threshold the team commits to upholding, making it the foundation for calculating how much 'bad' reliability (errors, downtime) is permissible.

A Service Level Indicator (SLI) is the specific, measured metric that informs the SLO. It is the raw measurement of service health. Common SLIs for AI services include:

• Model inference latency (p95, p99)
• Request success rate
• Hallucination rate
• Retrieval precision@K The SLI is continuously measured and compared against the SLO to determine error budget consumption.

Burn rate quantifies how quickly the error budget is being consumed. It's calculated as the percentage of the total budget used per unit of time. A burn rate of 1.0 means the budget will be exhausted in the SLO's time window. Critical alerts are often triggered by high burn rates (e.g., 5.0 or 10.0), signaling that degradation must be addressed immediately to prevent an SLO breach. Multi-window alerting uses burn rates across short (e.g., 1 hour) and long (e.g., 30-day) windows to reduce noise.

Graceful degradation is a design principle directly enabled by error budget thinking. When a service experiences high load or partial failure, it intentionally reduces non-essential functionality to protect its core SLOs. For AI services, this could mean:

• Switching from a high-quality, slow model to a faster, lighter one.
• Disabling complex feature extraction to maintain baseline response SLIs.
• Serving cached responses when live inference is degraded. This strategy allows a service to remain operational within its error budget during incidents.

A canary deployment is a release strategy that leverages the error budget to manage risk. A new model or service version is deployed to a small, controlled percentage of production traffic. Its performance (SLIs) is closely monitored. If the canary performs within SLOs and does not consume error budget excessively, the rollout proceeds. If it degrades performance, it is rolled back, having only consumed a small, acceptable portion of the budget. This makes deployments a deliberate, measured risk.

A composite SLO represents the overall reliability of a service composed of multiple dependent components, each with its own SLO. The error budget for the composite service is derived from the mathematical combination (often the product) of the SLOs of its critical dependencies. This is crucial for AI services, which often depend on:

• Model inference endpoints
• Vector databases for retrieval
• External APIs for tool calling Managing the composite error budget requires understanding and budgeting for failure modes across this entire stack.