SLO for Hallucination Rate

The Service Level Indicator (SLI) is the measurable metric: the percentage of model responses classified as hallucinations within a defined evaluation window. This is calculated as:

• (Number of Hallucinatory Outputs / Total Evaluated Outputs) * 100

The SLI must be paired with a precise operational definition of a hallucination (e.g., 'any factual claim in the output not directly supported by the provided source context'). Measurement typically requires a combination of automated classifiers and human-in-the-loop review for a statistically significant sample.

The SLO target is the maximum acceptable value for the hallucination rate SLI, expressed as a percentage over a compliance period (e.g., '≤ 2% over a 30-day rolling window').

The error budget is derived directly from this target. If the SLO is 98% non-hallucinatory (2% hallucination rate), the error budget is the permissible 2% of responses that can be hallucinations. This budget:

• Governs risk: It quantifies how much reliability can be 'spent' on deployments or experiments.
• Triggers action: Exhausting the budget should freeze feature launches and mandate a focus on improving factual accuracy.

This component defines how hallucinations are identified, which is critical for consistent measurement. It includes:

• Evaluation Framework: Specifies the tools and pipelines for scoring outputs (e.g., using LLM-as-a-judge with rubric-based prompts, embedding-based faithfulness scores, or human review).
• Sampling Strategy: Defines how requests are selected for evaluation (e.g., stratified random sampling across all user queries, oversampling of high-risk categories).
• Ground Truth & Context: Mandates that every evaluation must have access to the source context provided to the model (e.g., retrieved documents) to assess factual grounding.

Defines the rules for when the team is notified of SLO risk. Simple threshold alerts are noisy; effective policies use burn rate calculations.

• Burn Rate: The speed at which the error budget is being consumed. A burn rate of 1.0 means spending the budget as fast as the SLO window (e.g., 30-day budget in 30 days). A rate of 10.0 spends it 10x faster.
• Multi-Window Alerting: Configures alerts for different severities. Example:
  • Warning: Burn rate of 5.0 for 1 hour (rapid, short-term spike).
  • Critical: Burn rate of 1.0 for 6 hours (sustained degradation likely to exhaust budget). This separates brief incidents from serious, ongoing violations.

Explicitly states what is in and out of scope for the SLO. A hallucination rate SLO typically applies to the end-to-end generative service, not just the core LLM. This requires mapping dependencies:

• In-Scope: The final answer presented to the user after any post-processing.
• Critical Dependencies: The performance of upstream systems that directly affect hallucination rate, such as:
  • Retrieval system precision (if using RAG). Poor retrieval is a major source of apparent hallucinations.
  • Context window management. Truncation or incorrect context assembly can force the model to 'guess'. The SLO may necessitate composite SLOs or SLIs for these dependencies.

The prescribed actions for when the SLO is at risk or violated. This turns monitoring into operational improvement. The playbook should include:

• Immediate Mitigations: Steps to reduce hallucination rate quickly (e.g., enabling a more conservative prompt guardrail, temporarily routing traffic to a more accurate but slower model, disabling high-risk features).
• Diagnostic Procedures: A checklist for root cause analysis (e.g., check for data drift in user queries, audit recent retrieval index updates, review deployment logs for prompt changes).
• Long-Term Remediations: Engineering tasks to address root causes (e.g., improving retriever training data, implementing self-reflection or verification loops in the agent, expanding the synthetic evaluation dataset for edge cases).

A Service Level Indicator (SLI) is a directly measurable metric that quantifies a specific aspect of a service's performance. For AI systems, this includes:

• Hallucination Rate: The percentage of model outputs that are factually incorrect.
• Model Inference Latency: Time from request to response.
• Retrieval Precision@K: Relevance of retrieved documents in a RAG system. The SLI provides the raw data against which a Service Level Objective (SLO) is evaluated.

An error budget is the allowable amount of service unreliability, calculated as 100% - SLO. If an SLO for hallucination rate is 99.5% (≤0.5% errors), the error budget is 0.5%. This budget:

• Defines operational risk for deploying new model versions.
• Guides the pace of innovation and change management.
• When exhausted, triggers a focus on stability and remediation over new features. It is a core concept from Site Reliability Engineering (SRE) applied to AI ops.

An SLO for answer faithfulness sets a quantitative target for how well a model's generated answer is grounded in its provided source context. It is a more nuanced quality metric than a simple hallucination rate, specifically targeting Retrieval-Augmented Generation (RAG) systems. It measures contradictions or unsupported extrapolations, even if the output is plausible. Evaluation often uses Natural Language Inference (NLI) models or entailment scores to automatically assess claim-by-claim support.

Percentile latency is a statistical measure of request processing time, critical for defining performance SLOs for AI inference. It reveals the distribution of user experience:

• p50 (median): The latency for the typical user.
• p95: The latency for 95% of requests; a common target for SLOs.
• p99 (tail latency): The worst-case experience, often impacted by tail latency amplification in distributed systems. For LLMs, this is further broken down into Time To First Token (TTFT) and Time Per Output Token (TPOT).

Burn rate is the speed at which a service consumes its error budget, expressed as a percentage of the budget consumed per hour or day. It is the key metric for intelligent alerting on SLO violations. A fast burn rate over a short window (e.g., 100% in 1 hour) indicates a severe, acute incident. A slower burn rate over a longer window (e.g., 50% in 7 days) signals chronic degradation. Multi-window alerting uses both to reduce noise and distinguish between spikes and sustained problems.

A composite SLO is a Service Level Objective derived from the aggregation of multiple underlying SLIs or component SLOs. For a complex AI service—like an agent that retrieves data, reasons, and generates text—the user-facing SLO might be a composite of:

• Retrieval subsystem availability.
• Model inference latency (p95).
• Hallucination/faithfulness rate. The composite SLO represents the overall reliability experienced by the end-user, accounting for the failure modes of all critical dependencies.