Burn Rate

Burn rate is the ratio of the actual error rate to the allowed error budget rate over a specific time window.

• Formula: Burn Rate = (Actual Error Rate) / (Error Budget / SLO Period)
• Baseline: A burn rate of 1x means the service is consuming its error budget exactly at the intended pace.
• Critical Thresholds: A burn rate of 14.4x over a 1-hour window indicates the entire 30-day error budget will be exhausted in just 2 days.
• Dimensionless Metric: Burn rate is expressed as a multiplier, not a percentage, enabling consistent alerting across services with different SLOs.

Effective burn rate alerting requires pairing a short evaluation window with a long evaluation window to balance fast detection against noise suppression.

• Short Window (e.g., 1 hour): Detects a sudden spike in errors. A 14.4x burn rate here triggers a page.
• Long Window (e.g., 3 days): Confirms the problem is sustained and not a transient blip. A 1.4x burn rate here triggers a ticket.
• AND Logic: Alerts fire only when both windows breach their thresholds, preventing false positives from momentary fluctuations.
• Reset Period: The alert resolves only after the burn rate drops below the threshold for the entire long window duration.

Burn rate reveals distinct failure signatures that inform incident response urgency.

• Linear Consumption: A steady 1x burn rate. Normal operation; no action required.
• Exponential Spike: A sudden 10x+ burn rate. Indicates a catastrophic regression, often from a bad deployment. Requires immediate model rollback.
• Slow Leak: A persistent 2-3x burn rate over days. Often caused by gradual data drift or infrastructure degradation. Requires investigation but not emergency paging.
• Budget Exhaustion: When the error budget reaches zero, the service has failed its SLO. Standard practice is to freeze all feature deployments until reliability is restored.

Burn rate and error budget are mathematically coupled; burn rate is the first derivative of budget consumption.

• Error Budget = 1 - (Good Events / Total Events) over the SLO window, capped at the allowed threshold.
• Burn Rate measures the velocity of budget depletion.
• Policy Enforcement: When the error budget is critically low or exhausted, automated circuit breakers should reject non-critical traffic to preserve remaining budget for essential operations.
• Freeze Policy: A common Site Reliability Engineering (SRE) practice is to halt all production pushes when the burn rate indicates the budget will be exhausted within the next 24 hours.

Google's SRE framework recommends a multi-window, multi-burn-rate approach for comprehensive coverage.

• Critical Alert: 14.4x burn rate over 1 hour AND 1.4x over 5 minutes. Pages the on-call engineer immediately.
• Warning Alert: 1.4x burn rate over 3 days AND 1.1x over 6 hours. Creates a ticket for investigation during business hours.
• Exhaustion Alert: Error budget remaining < 10%. Notifies product management to halt feature launches.
• Rationale: This layered strategy ensures that both fast-burning outages and slow, chronic reliability erosion are detected and actioned appropriately.

Burn rate alerting is implemented using time-series metrics and PromQL-style queries.

• Metric Required: A counter of total requests and a counter of failed requests (e.g., HTTP 5xx responses or model prediction errors).
• Query Logic: Calculate the error ratio over the short window and divide by the allowed error budget ratio.
• Recording Rules: Pre-compute burn rate values to avoid expensive ad-hoc queries on high-cardinality data.
• Integration: Burn rate alerts feed directly into the escalation policy and can trigger automated runbook automation for common failure modes like model timeouts.

The explicit, quantifiable allowance for failure within an AI service, calculated as 1 - Service Level Objective (SLO) . It represents the maximum acceptable downtime or error rate over a rolling window.

• Mechanism: If the SLO is 99.9% availability, the error budget is 0.1% (approx. 43 minutes/month).
• Policy: Exhausting the budget triggers a feature freeze, halting new deployments until reliability is restored.
• Burn Rate: The speed at which this budget is consumed, normalized to the alerting window.

A precise, measurable target for system reliability over a specific time period, serving as the fulcrum between development velocity and operational stability.

• Definition: An internal goal, e.g., '99.95% of inference requests will return in < 200ms over 30 days'.
• Distinction: Unlike a Service Level Agreement (SLA) , breaching an SLO does not incur a financial penalty but triggers an engineering response.
• Burn Rate Alerting: Alerts fire based on how fast the SLO is being consumed, not just a static threshold breach.

A stability pattern that automatically stops requests to a failing AI service to prevent cascading failures and allow the downstream system to recover.

• States: Closed (normal), Open (failing, requests blocked), and Half-Open (probing for recovery).
• Application: If an LLM endpoint returns 5xx errors at a high rate, the breaker trips, immediately failing requests instead of adding load.
• Benefit: Protects the error budget from being consumed by a known-bad dependency.

A self-healing mechanism that instantly reverts a production model to its last known good version when a critical performance threshold is breached.

• Trigger: Often tied directly to the burn rate; if the error budget is consumed faster than a 14.4x threshold, a rollback is initiated.
• Artifacts: Requires strict versioning of model weights, inference code, and configuration.
• Safety: This is a last-resort automated action to stop the hemorrhage of the error budget.

A structured, written analysis of an AI incident that focuses on systemic root causes and process improvements rather than assigning individual fault.

• Goal: Understand how the error budget was burned, not who burned it.
• Output: Actionable remediation items to add resilience, such as better out-of-distribution detection or stricter guardrails.
• Culture: Essential for psychological safety in high-stakes AI operations.

The automated monitoring process that identifies statistical changes in production input data or model predictions relative to the training baseline.

• Data Drift: The distribution of input features (P(X)) changes, e.g., user prompts shift to a new domain.
• Concept Drift: The relationship between inputs and the target variable (P(Y|X)) changes.
• Impact: Undetected drift silently burns the error budget by degrading model accuracy without explicit errors.