Error Budget

The error budget explicitly defines the acceptable failure threshold. It is calculated as 1 minus the Service Level Objective (SLO).

• Formula: If the SLO is 99.9% availability, the error budget is 0.1%.
• Measurement Window: Typically tracked over a rolling 4-week period to balance responsiveness against statistical noise.
• Non-Negotiable: Once exhausted, it triggers a mandatory freeze on all feature launches and model updates.

The burn rate measures how quickly the error budget is being consumed relative to the measurement window. It is the critical signal for preemptive incident response.

• Fast Burn: Consuming 2% of the budget in 1 hour signals an urgent incident requiring immediate paging.
• Slow Burn: Consuming 10% of the budget over 3 days indicates a chronic degradation that needs scheduled engineering work.
• Multi-Window Tracking: Sophisticated policies use multiple windows (e.g., 1h, 6h, 3d) to detect both acute spikes and gradual drift.

The error budget policy acts as an automated governance mechanism, not just a dashboard metric. It gates the CI/CD pipeline based on budget status.

• Green State: Budget remaining > 50%. Standard deployment velocity is permitted.
• Yellow State: Budget remaining between 20% and 50%. New deployments require explicit risk-acceptance approval.
• Red State: Budget exhausted. The deployment pipeline is frozen. All engineering effort must pivot to reliability improvements until the budget replenishes.

Every error event must be attributed to a specific cause to prevent the budget from being wasted on unactionable noise. This drives precise remediation.

• Attributable to Change: Failures caused by a specific model update or infrastructure change are debited against the team that initiated the change.
• Background Noise: A small, pre-allocated portion of the budget covers unavoidable infrastructure flakiness.
• Excluded Events: Planned maintenance windows and external dependencies (e.g., third-party API outages) are typically excluded from the calculation to maintain internal accountability.

The error budget policy must be calibrated against the system's Recovery Time Objective to ensure the budget is not structurally impossible to maintain.

• Budget Floor: The error budget must be larger than the failure time introduced by a single worst-case recovery event.
• Example: If the RTO for a critical model is 4 hours, a 99.9% SLO (43 minutes of downtime/month) is mathematically unachievable. The SLO must be relaxed or the RTO must be improved.
• Architectural Constraint: This forces a hard conversation about whether the system architecture can actually support the desired reliability target.

The error budget is a formal agreement between product management and engineering, replacing subjective arguments about speed versus stability with objective data.

• Product's Lever: As long as the budget is positive, product managers have the authority to prioritize new feature velocity.
• Engineering's Lever: The moment the budget is exhausted, engineering has the unilateral authority to halt all feature work and dedicate resources to hardening the system.
• Cultural Shift: This depersonalizes the conflict between 'moving fast' and 'keeping the lights on,' turning it into a mathematical negotiation.