Error Budget

A product team wishes to launch a new, high-risk feature. The Site Reliability Engineering (SRE) team calculates that the deployment could consume 30% of the quarterly error budget based on historical failure rates of similar changes.

• Decision Gate: The launch is approved, but with the condition that it is rolled out using a canary deployment to a small percentage of traffic first.
• Budget Tracking: Real-time monitoring compares the canary's error rate against the baseline. If the burn rate exceeds projections, the rollout is automatically paused.
• Outcome: This gates innovation with quantifiable risk, preventing a single release from jeopardizing the service's overall SLO.

A monitoring alert shows database latency is degrading, threatening the latency SLO (e.g., p95 < 300ms). The team has 20% of its error budget remaining for the month.

• Quantifying the Risk: Engineers estimate that without intervention, the trend will burn 15% of the remaining budget per week.
• Trade-off Analysis: The team has also planned new feature work. Using the error budget as data, they make an objective decision: postpone the new feature sprint and allocate engineering resources to database index optimization and query refactoring.
• Result: The budget acts as an unbiased arbiter, ensuring reliability work is prioritized based on measurable impact to user experience.

A major incident causes a service outage for 45 minutes, consuming 60% of the monthly error budget in a single event.

• Triggering the Policy: The team's error budget policy states that consuming >50% of the budget in a week triggers a focus period or release moratorium.
• Action: All non-essential feature deployments and risky changes are frozen. The engineering team enters a dedicated blameless postmortem and remediation sprint.
• Goal: The moratorium is not punishment; it's a cooling-off period to improve system stability, pay down technical debt, and ensure the error budget can be rebuilt before resuming normal velocity.

A startup needs to move extremely fast. They define a lenient SLO of 95% availability (uptime) for their MVP, accepting more downtime in exchange for speed.

• Budget Calculation: Error Budget = 1 - 0.95 = 0.05 (5%). Over a 30-day month, this allows for 30 days * 24 hours * 0.05 = 36 hours of acceptable downtime.
• Usage: This large budget explicitly permits the team to take significant risks, perform frequent major deployments, and experiment aggressively.
• Evolution: As the service matures and user base grows, the SLO is tightened (e.g., to 99.5%), automatically reducing the error budget and forcing a more disciplined engineering process.

A user request flows through four microservices: A → B → C → D. The product's end-to-end SLO is 99.9% availability.

• Budget Allocation: The SRE team uses the budget decomposition method. They cannot simply give each service a 99.9% SLO, as failures compound. Using probability, they allocate stricter individual SLOs (e.g., 99.97% each) so their combined theoretical availability meets the 99.9% target.
• Dependency Management: Service B's SLO depends on its own code and the health of its database. Its error budget must account for both internal failures and dependency failures.
• Benefit: This creates clear, quantified reliability targets for each team, aligning incentives across a complex dependency graph.

To move beyond spreadsheets, teams integrate error budgets into their CI/CD and observability platforms.

• Burn Rate Alerts: Tools like Sloth or custom Prometheus alerts fire when the error budget is being consumed too quickly (e.g., "budget will be exhausted in 3 days if current error rate continues").
• Deployment Gates: A pipeline check queries the remaining error budget. If it's below a threshold (e.g., <5%), the deployment is blocked unless explicitly overridden by an engineering manager.
• Dashboarding: Public dashboards show real-time budget status, making reliability a transparent, shared metric for product and engineering leadership.