Mean Time To Recovery (MTTR)

The elapsed time from the onset of a service failure or degradation until it is identified by the monitoring system. This phase depends on the sensitivity and coverage of health checks, anomaly detection algorithms, and alerting rules.

• Key SLIs: Alert latency, monitoring coverage.
• AI-Specific: Requires specialized monitors for model performance drift, hallucination rate spikes, or retrieval system failures.

The time spent isolating the root cause after detection. This involves analyzing telemetry, logs, and traces to pinpoint the faulty component—be it infrastructure, data pipeline, or the model itself.

• Key Tools: Distributed tracing, model inference latency dashboards, data drift detection systems.
• Complexity: In AI systems, diagnosis may require distinguishing between a model bug, corrupted input features, or a failing vector database retrieval step.

The time to implement a short-term fix that restores core service, even if at a reduced capability. The goal is to meet Service Level Objectives (SLOs) quickly, often through graceful degradation.

• AI Tactics: Falling back to a simpler, more reliable model; disabling a problematic RAG retrieval path; serving cached responses.
• SRE Principle: Prioritizes user-facing stability over perfect resolution.

The time required to implement a permanent fix and fully restore the service to its intended, pre-incident state. This phase closes the incident and may involve deployments or data corrections.

• AI Actions: Rolling back a faulty model via canary deployment; retraining on corrected data; patching a prompt injection vulnerability.
• Measurement: Ends when all mitigation measures are removed and standard SLO compliance is verified.

MTTR for AI services must account for unique failure vectors beyond traditional software.

• Model Degradation: Slow data drift or sudden performance collapse requiring retraining.
• Hallucination Outbreaks: A spike in factually incorrect outputs, necessitating context engineering or RAG pipeline fixes.
• Retrieval Failure: The vector database or semantic search system returns irrelevant context.
• Agentic Deadlocks: An autonomous agent gets stuck in a reasoning loop, requiring intervention in its cognitive architecture.

Proactive engineering practices directly improve MTTR by streamlining the recovery pipeline.

• Runbooks & Automation: Pre-written playbooks for common AI failures (e.g., "Restore Model from Registry").
• Observability: Comprehensive logging for model inference latency, token throughput, and agentic reasoning traces.
• Error Budgets: Using the error budget derived from SLOs to prioritize reliability work that reduces future MTTR.
• Chaos Engineering: Proactively testing failure scenarios in staging, such as vector database latency spikes.

A Service Level Objective (SLO) is a quantitative target for the reliability, performance, or quality of a service, expressed as a percentage of requests that must meet a specific Service Level Indicator (SLI) over a defined time window. For AI services, SLOs are critical for defining acceptable thresholds for metrics like latency, throughput, and output quality.

• Example: "99.9% of inference requests must have a latency under 100ms over a 30-day rolling window."
• SLOs are internal goals, distinct from external Service Level Agreements (SLAs) which carry contractual penalties.
• MTTR is often used to inform SLOs for service availability and resilience.

A Service Level Indicator (SLI) is a directly measurable metric that quantifies a specific aspect of a service's performance, serving as the basis for evaluating a Service Level Objective (SLO). For AI systems, SLIs are specialized to measure model behavior and infrastructure performance.

• Core AI SLIs: Model Inference Latency, Time To First Token (TTFT), Time Per Output Token (TPOT), Hallucination Rate, Retrieval Precision@K.
• Reliability SLIs: Error rate, request success rate, and system availability.
• MTTR itself is a key SLI for measuring operational resilience and the efficiency of incident response.

An error budget is the allowable amount of service unreliability, calculated as 100% minus the Service Level Objective (SLO). It defines the risk a team can accept for deploying changes or experiencing failures without violating the SLO.

• Calculation: If an SLO is 99.9% availability, the error budget is 0.1% unreliability.
• Usage: Error budgets guide the pace of innovation. Exhausting the budget triggers a focus on stability and reliability work.
• Relationship to MTTR: A high MTTR consumes the error budget rapidly during an incident. Reducing MTTR preserves the budget for planned changes and feature development.

Burn rate is the speed at which a service consumes its error budget, calculated as the percentage of the budget consumed per unit of time. It is a critical metric for triggering alerts based on the risk of an impending SLO violation.

• Fast Burn Rate: Indicates a severe, ongoing incident that will violate the SLO quickly. Requires immediate, all-hands response.
• Slow Burn Rate: Indicates a chronic, lower-severity issue that will violate the SLO over a longer period.
• MTTR Impact: A high MTTR during an incident directly causes a fast burn rate. Effective incident management aims to reduce MTTR to slow the burn and protect the SLO.

A canary deployment is a release strategy where a new version of a service (e.g., a new AI model) is deployed to a small, controlled subset of users or traffic. Its performance and stability are monitored against key SLIs before a full rollout.

• Purpose: To validate that a new release meets its SLOs and does not introduce regressions or failures.
• AI Context: Used for deploying new model versions, updated prompts, or changes to Retrieval-Augmented Generation (RAG) pipelines.
• Connection to MTTR: If a canary fails, the blast radius is limited. This containment reduces the potential impact and complexity of an incident, which can significantly lower the overall MTTR compared to a full-blast failure.

Graceful degradation is a system design principle where a service maintains partial or reduced functionality when components fail or experience high load. This allows it to continue serving users while protecting its core Service Level Objectives (SLOs).

• AI System Examples:
  • A RAG system falling back to keyword search if the vector database is slow.
  • A vision model returning lower-confidence results with caveats instead of failing entirely.
  • An agentic workflow skipping non-critical tool calls to complete the primary task.
• Impact on MTTR: Systems designed for graceful degradation can often remain operational while the root cause of a partial failure is diagnosed and fixed. This turns a total service outage into a degraded state, which typically has a lower severity and a longer acceptable MTTR.