Burn Rate

Burn rate is the speed at which a service consumes its error budget, expressed as a percentage of the total budget used per unit of time. It is calculated by measuring the rate of bad events (requests failing the SLI) relative to the total event rate over a specific window.

• Formula (Simplified): Burn Rate = (Error Budget Spent / Total Error Budget) / Time Period.
• Example: If a service with a 30-day error budget consumes 3 days' worth of budget in 6 hours, its burn rate is (3 days / 30 days) / (0.25 days) = 0.4 or 40% per day.
• It directly answers: "How fast are we approaching an SLO breach?"

Burn rate is the foundational metric for multi-window alerting, a core SRE practice that reduces alert noise. Alerts are configured to fire based on burn rate thresholds across different time windows.

• Short, Fast-Burn Alert: Triggers if the error budget is being consumed very rapidly over a short window (e.g., burn rate > 1000% per day for 5 minutes). This catches sudden, severe outages.
• Long, Slow-Burn Alert: Triggers if the budget is being consumed steadily over a longer period (e.g., burn rate > 100% per day for 1 hour). This catches sustained degradation that might not trigger a short-window alert.
• This strategy distinguishes between a brief spike and a serious, ongoing problem.

Burn rate cannot be understood in isolation; it is intrinsically linked to the error budget. The error budget defines the "fuel" that burn rate consumes.

• Error Budget Formula: Error Budget = (100% - SLO%) * Measurement Window.
• A high burn rate (e.g., 500%/day) means the budget is being depleted quickly, indicating high immediate risk. The team must prioritize firefighting.
• A low burn rate (e.g., 10%/day) indicates the service is well within its reliability target, granting the team budget to spend on risky deployments, experiments, or feature launches.
• Monitoring burn rate over time provides a trend of risk exposure.

For AI-powered services, burn rate is calculated using AI-specific Service Level Indicators (SLIs). The concept remains identical, but the "bad events" are defined by model performance metrics.

• Example SLIs for Burn Rate Calculation:
  • Requests where model inference latency > 500ms (p95).
  • Requests where the hallucination detection score exceeds a threshold.
  • Retrieval Precision@5 for a RAG system falling below 80%.
  • Agent task success rate dropping below 95%.
• A rising burn rate on an SLO for answer faithfulness signals that model outputs are becoming increasingly ungrounded, requiring immediate investigation into the retrieval system or prompt context.

Burn rate is a more sophisticated metric than a simple error percentage or error rate. It incorporates time and budget context, providing a sense of urgency and trajectory.

• Error Rate: "5% of requests are failing right now."
• Burn Rate: "At the current failure rate, we will exhaust our monthly error budget in 36 hours."
• The latter is an actionable risk forecast. A 5% error rate might be acceptable for a service with a 99% SLO (1% budget) but catastrophic for a service with a 99.9% SLO (0.1% budget). Burn rate automatically accounts for this by measuring consumption against the specific budget.

The burn rate value dictates the operational response priority and strategy. It is a key input for incident management and blameless postmortems.

• Critical Burn Rate (>500%/day): Immediate incident declaration, all-hands response, and potential rollback of recent changes. The focus is on restoration.
• Elevated Burn Rate (100%-500%/day): High-priority investigation by the on-call engineer. Deployment freeze may be enacted.
• Normal Burn Rate (<100%/day): Standard operations. The team has budget to deploy and experiment.
• By tying alerts to burn rate, teams avoid alert fatigue and ensure they are only paged for situations that genuinely threaten their SLO commitments.

An error budget is the allowable amount of service unreliability, calculated as 100% - SLO. It quantifies the risk a team can accept for changes, deployments, or incidents before violating the Service Level Objective.

• Purpose: Defines a finite resource for reliability trade-offs.
• Calculation: If an SLO is 99.9% monthly availability, the error budget is 0.1% (or ~43 minutes of downtime).
• Usage: Burn rate directly consumes this budget. A high burn rate indicates the budget is being spent quickly, signaling high risk.

Multi-window alerting is a strategy that triggers alerts based on SLO burn rate violations evaluated over multiple, overlapping time windows (e.g., 1-hour and 6-hour windows).

• Purpose: Reduces alert noise and distinguishes between brief, transient spikes and sustained, serious degradation.
• Common Pattern: Alert on a fast burn (e.g., consuming 5% of the error budget in 1 hour) and a slow burn (e.g., consuming 25% of the budget in 6 hours).
• Benefit: Prevents over-alerting on short-lived issues while ensuring prolonged problems are caught before the error budget is exhausted.

A Service Level Indicator (SLI) is a directly measurable metric that quantifies a specific aspect of a service's performance or quality. It is the raw measurement that an SLO targets.

• Examples for AI: Model inference latency, error rate, task success rate, hallucination rate, retrieval precision.
• Relationship to Burn Rate: Burn rate calculations are based on the SLI's performance relative to the SLO target. The SLI provides the real-time data stream that determines if the SLO is being met or if the error budget is being consumed.

A Service Level Objective (SLO) is a quantitative target for the reliability or performance of a service, expressed as a percentage of requests that must meet a specific SLI over a defined time window.

• Foundation for Burn Rate: The SLO defines the "good" threshold. Burn rate measures how quickly the service is deviating from this target.
• Example: "99.9% of inference requests must have a latency under 100ms over a 30-day window."
• Criticality: Without a well-defined SLO, calculating a meaningful burn rate is impossible, as there is no budget to burn.

Tail latency amplification is a phenomenon in distributed systems and AI inference pipelines where the slowest percentile of requests (e.g., p99, p99.9) becomes disproportionately slower due to dependencies, queuing, and resource contention.

• Impact on SLOs: User-facing SLOs are often defined on tail latencies (e.g., p99 < X ms). Amplification can cause sudden, severe burn rate spikes.
• Causes in AI: Variable-length generations, cold starts, GPU memory contention, and downstream API calls can all amplify tail latency.
• Mitigation: Techniques like continuous batching, intelligent load shedding, and dependency isolation are used to control tail latency and stabilize burn rate.

Graceful degradation is a system design principle where a service maintains partial or reduced functionality when components fail or experience high load.

• Relationship to Burn Rate: A key strategy for managing burn rate during incidents. By degrading non-essential features, the core service can protect its primary SLOs and slow the consumption of the error budget.
• AI Examples: A RAG system might disable complex re-ranking and fall back to simple semantic search, or a generative model might switch to a faster, smaller model to preserve latency SLOs.
• Proactive Measure: Designed ahead of time to provide levers to pull when burn rate alerts fire.