Error Budget

An error budget is not an arbitrary number; it is mathematically derived from a Service Level Objective (SLO). If an SLO states that 99.9% of requests must complete successfully in a month, the error budget is the remaining 0.1% of allowable failures. For 1 million requests, this equates to a budget of 1,000 errors. This direct linkage ensures the budget is a precise, objective measure of acceptable risk.

Error budgets are calculated for a specific accounting period, typically a calendar month or a rolling 30-day window. This period resets, making the budget a renewable resource. This cadence aligns with engineering planning cycles (e.g., sprints, monthly reviews). Key operational questions include:

• How much budget remains this period?
• How fast are we consuming it?
• Will we exhaust it before the reset?

The primary function of an error budget is to objectively govern the pace of change. When the budget is healthy (e.g., only 30% consumed), teams have clear authority to deploy new LLM models, features, or infrastructure changes that carry reliability risk. If the budget is nearly exhausted, the focus must shift to stability work—fixing bugs, improving monitoring, or reducing technical debt—until the next period begins. This creates a data-driven, blameless framework for managing risk.

The budget is consumed whenever the service's actual performance falls below its SLO. For LLM services, this is typically measured via Service Level Indicators (SLIs) such as:

• Error Rate: Percentage of requests returning 5xx errors or failing content safety checks.
• Latency: Percentage of requests exceeding a P99 latency threshold (e.g., 10 seconds).
• Availability: Percentage of time the LLM endpoint is reachable and functional. Each violation event deducts a corresponding amount from the total budget.

The error budget is a shared resource owned collectively by the product and engineering teams responsible for the LLM service. It is not a performance target for individuals but a team constraint. This shared ownership fosters collaboration between developers, SREs, and product managers to make informed trade-offs between innovation and stability, moving discussions from subjective opinion to objective data.

By quantifying the cost of instability, the error budget becomes a powerful tool for technical and business prioritization. It answers critical questions:

• Should we launch a new high-risk feature now, or wait until next period?
• Is investing engineering weeks into latency optimization justified by the budget it will preserve?
• Does the proposed model architecture change pose an unacceptable risk to our reliability commitments? This transforms reliability from an abstract goal into a concrete, tradable asset.

An LLM chat service has an SLO of 2 seconds for P95 latency. Over a 30-day window, the budget allows for 43,200 seconds of excess latency (5% of total time).

• A poorly optimized prompt causes a spike, consuming 1,000 seconds of the budget.
• A subsequent GPU memory bottleneck consumes another 800 seconds.
• The team must now freeze feature deployments and focus on optimization until the budget resets, as further violations would breach the SLO agreement.

A retrieval-augmented generation (RAG) system has a composite SLO: 99.9% availability and <2% hallucination rate on factual queries.

• A vector database outage consumes 0.05% of the availability budget.
• Degraded retrieval due to embedding drift increases the hallucination rate to 3% for a cohort of users, consuming a significant portion of the quality budget.
• The combined consumption triggers an operational review, pausing all experiments with the retrieval pipeline until root causes are addressed.

A team uses their remaining error budget as a risk thermostat for releases.

• With 75% of the budget remaining, they confidently deploy a new, more capable but less tested model version using a canary deployment to 10% of traffic.
• With only 10% of the budget remaining, they restrict deployments to critical security patches only and mandate that any new change must first pass through a shadow deployment to prove it does not degrade metrics.
• This creates a data-driven release cadence that balances innovation with reliability.

Error budgets are consumed by measurable incidents that violate SLIs. Common LLM incidents include:

• Inference Failures: GPU OOM errors or failed health checks.
• Performance Degradation: Increased Time to First Token (TTFT) due to model bloat or inefficient batching.
• Quality Regressions: Spike in user-reported errors or a drop in output correctness scores against a golden dataset.
• Cost Overruns: While not a direct SLO, exceeding a cost-per-query threshold may be tied to a financial objective, consuming a separate budgetary allowance.

Teams use monitoring to avoid exhausting the budget prematurely.

• Real-time Dashboards: Grafana displays show budget burn rate alongside SLI metrics like latency and error rate.
• Statistical Process Control (SPC): Control charts on inter-token latency detect anomalies before they cause a major violation.
• Cohort Analysis: Comparing error rates for users of a new prompt template isolates its impact on the budget.
• Pre-mortems: Before a risky deployment, the team estimates its potential budget impact and defines a clear rollback trigger.

When a budget is exhausted or a major incident occurs, a formal process follows.

1. Service Freeze: All non-essential changes are halted.
2. Root Cause Analysis (RCA): The team investigates the underlying cause (e.g., a memory leak in the KV cache manager).
3. Remediation: Fixes are applied and validated.
4. Budget Reset: At the start of the next calendar period (e.g., month), the budget is fully restored.
5. Policy Review: The SLO targets themselves may be re-evaluated if they are consistently too strict or too loose.

A Service Level Objective is the specific, measurable target for a Service Level Indicator that defines the acceptable reliability or performance of an LLM service. An error budget is derived directly from an SLO.

• Example: "99.9% of LLM chat completions must have a latency under 2 seconds (P99)."
• The SLO defines the "good" state; the error budget quantifies the allowable "bad" state before the SLO is violated.

A Service Level Indicator is the quantitative measure of a service's behavior that an SLO targets. For LLMs, common SLIs include:

• Latency: Time to First Token (TTFT), Inter-Token Latency, end-to-end request time.
• Availability: Successful request rate (non-5xx HTTP responses).
• Quality: Rate of outputs flagged by a hallucination detection system or failing a correctness check.
• The SLI is the raw measurement; the SLO sets its target; the error budget tracks deviation from that target.

These are deployment strategies used to manage risk against the error budget.

• Canary Deployment: A new model version is released to a small percentage of traffic. Its performance SLIs are closely monitored. If errors spike and consume budget too quickly, the rollout is halted.
• Shadow Deployment: The new model processes requests in parallel with the stable version, but its outputs are discarded. This allows for comparison of SLIs (like latency or output drift) with zero user impact, informing a go/no-go decision for release.

Mean Time to Recovery is the average time required to restore a service to normal operation after a failure or SLO violation is detected. It is a critical metric for managing error budget consumption.

• A low MTTR means the team can halt budget burn quickly after an incident.
• MTTR encompasses detection time, diagnosis (Root Cause Analysis), mitigation (e.g., rolling back a bad deployment), and full remediation.
• Teams often use error budget policy to prioritize work that reduces MTTR.

Statistical Process Control is a methodology using control charts to monitor process behavior. In LLM operations, it's applied to SLIs to distinguish normal variance from significant anomalies that threaten the error budget.

• Control charts plot an SLI (e.g., P99 latency) over time with upper/lower control limits.
• A data point breaching a control limit signals a process potentially "out of control," triggering investigation before significant error budget is consumed.
• This provides a statistically rigorous method for anomaly detection in performance metrics.

These are quality-focused concepts that feed into error budget considerations for correctness SLIs.

• Golden Dataset: A curated set of input-output pairs used for continuous evaluation. A drop in performance on this dataset can consume an accuracy/quality error budget.
• Output Drift: A statistical change in the distribution of model outputs (e.g., sentiment, toxicity scores, response length) compared to a baseline. Significant drift may indicate model degradation, consuming budget allocated for quality preservation.
• Monitoring these helps define and defend SLOs for non-functional qualities like correctness.