Root Cause Analysis (RCA)

Effective RCA begins with a quantifiable, observable definition of the incident. This involves moving from vague descriptions ("the model is slow") to specific, measurable statements using Service Level Indicators (SLIs).

• Example: "P99 latency for the /chat/completions endpoint increased from 2.1s to 8.7s between 14:00 and 15:00 UTC, coinciding with a 300% spike in traffic for prompts containing code generation."
• This step establishes the what, when, and scope, creating a clear baseline for investigation and preventing scope creep.

Isolating the root cause requires correlating data across the entire LLM stack. Relying on a single metric is insufficient.

• Infrastructure Metrics: GPU utilization, memory pressure, KV cache hit rates, and network I/O from tools like Prometheus.
• Application Metrics: Time to First Token (TTFT), Inter-Token Latency, error rates, and token consumption.
• Model Quality Signals: Output drift scores against a golden dataset, spike in hallucination detection triggers, or changes in embedding drift for retrieval systems.
• Traffic & User Context: Cohort analysis to see if the issue affects specific user segments, prompt types, or model versions.
• Distributed Tracing with OpenTelemetry (OTel) is critical to follow a single request's path through APIs, middleware, and model inference.

Use structured methodologies to move from correlation to causation, avoiding cognitive biases like jumping to conclusions.

• The 5 Whys: Iteratively ask "why" to drill down from the symptom to the systemic cause. (e.g., Why was latency high? GPU memory exhausted. Why? KV cache size increased. Why? Average prompt length doubled due to a new user cohort.)
• Fishbone (Ishikawa) Diagrams: Visually map potential causes across categories like Model, Infrastructure, Data, Process, and People.
• Change Analysis: Systematically review all changes preceding the incident: new model deployment (canary or shadow), configuration updates, traffic pattern shifts, or data pipeline modifications.
• Blameless Postmortems: Focus on systemic factors and process gaps rather than individual error.

The goal of RCA is to find causes you can fix to prevent recurrence. Distinguish between root causes, contributing factors, and symptoms.

• A Root Cause is a fundamental, addressable failure in a process or system whose removal prevents the issue. Example: Lack of load testing for a new continuous batching configuration that failed under a specific traffic pattern.
• A Symptom is the observable effect (high latency).
• A Contributing Factor exacerbated the issue but didn't cause it (concurrent database maintenance).
• Effective actions target root causes: implementing automated anomaly detection on key SLIs, adding guardrails for prompt length, or improving continuous batching logic.

The RCA process is incomplete without clear, actionable follow-up. Documentation ensures institutional learning and accountability.

• Action Items: Each identified root cause must have a specific, assigned corrective action (e.g., "Implement auto-scaling based on P90 latency and queue depth" or "Add a pre-processing step to truncate context windows exceeding 8k tokens").
• Verification: Define how the fix will be validated (e.g., load test results, monitoring of a specific Service Level Objective (SLO) for one week).
• Communication: Share findings with relevant stakeholders (engineering, product) to align on priorities and error budget impact.
• This creates a closed feedback loop that improves system resilience.

RCA should be a core component of Service Level Objective (SLO) management. Incidents consume the error budget, and RCA dictates how to spend it wisely.

• Prioritization: RCAs for incidents that breach SLOs or consume significant error budget take highest priority.
• Action Triage: Corrective actions are evaluated based on their expected reduction in future error budget burn. Preventing a recurring 30-minute outage is more valuable than optimizing a non-critical path.
• Process Improvement: If repeated RCAs point to similar causes (e.g., configuration drift), it indicates a need to invest in better deployment processes or Statistical Process Control (SPC) for key metrics.
• This principle ensures RCA drives tangible business and reliability outcomes.

Performance degradation often stems from bottlenecks in the computational infrastructure serving the model. Key factors include:

• GPU Memory Exhaustion: Caused by large batch sizes, long context windows, or memory leaks in the KV Cache, leading to out-of-memory errors and failed requests.
• Compute Saturation: Insufficient GPU or CPU capacity to handle peak request loads, increasing Time to First Token (TTFT) and Inter-Token Latency.
• Network Latency: High latency between client, load balancer, and model instances, especially in distributed deployments.
• I/O Bottlenecks: Slow reads from disk or network-attached storage when loading model weights or retrieving context from vector databases.

Changes or anomalies in the input data presented to the LLM are a primary source of output quality issues.

• Prompt Drift: Unintended, gradual changes to prompt templates or few-shot examples deployed in production, altering model behavior.
• Input/Concept Drift: The statistical properties of real-world user queries shift over time, moving outside the distribution the model was optimized for, reducing answer relevance.
• Malformed or Adversarial Inputs: User prompts containing garbled text, extreme length, or crafted prompt injection attacks designed to jailbreak the model or exfiltrate data.
• Retrieval Augmentation Failures: Underlying semantic search returning irrelevant or outdated context from knowledge bases, leading to grounded hallucinations.

Problems originating from the model artifact itself or its runtime configuration.

• Output Drift / Embedding Drift: The model's statistical output distribution changes over time, potentially due to silent regressions in upstream model providers or unintended fine-tuning effects.
• Quantization Artifacts: Aggressive post-training quantization to reduce model size can introduce accuracy loss and generation artifacts, increasing perplexity.
• Incorrect Model Version or Branch: Deployment errors that serve a stale, untested, or development version of a model instead of the intended production version.
• Hyperparameter Misconfiguration: Suboptimal settings for inference parameters like temperature, top-p, or frequency penalty, leading to erratic, repetitive, or low-quality outputs.

Failures in the surrounding application and service ecosystem that the LLM depends on.

• Tool Calling/API Failures: External APIs, databases, or functions called by the LLM via tool calling frameworks time out, return errors, or provide malformed data.
• Context Window Management: Logic errors in truncating, summarizing, or chunking long conversations exceed the model's context limit, causing lost coherence.
• Rate Limiting & Throttling: Aggressive rate limits on internal or external model APIs cause request queuing and increased latency.
• State Management Errors: Bugs in session or conversation state tracking lead to incorrect context being passed to the model for a given user interaction.

The inability to detect and diagnose issues is itself a root cause, often stemming from insufficient telemetry.

• Lack of Distributed Tracing: Without OpenTelemetry traces, it is impossible to see the full path of a request across microservices (e.g., from API gateway to model to vector DB), obscuring the slow component.
• Insufficient Metrics: Missing key Service Level Indicators (SLIs) like token-level latency histograms, error rates by endpoint, or embedding drift scores.
• Poor Logging: Unstructured logs or logs lacking critical correlation IDs prevent reconstructing the sequence of events for a failed request.
• Delayed Alerting: Alerts based on coarse-grained metrics (e.g., overall error rate) instead of anomaly detection on key cohorts, delaying incident response and increasing Mean Time to Recovery (MTTR).

Incidents triggered by interaction effects and load dynamics within the system.

• Retry Storms & Feedback Loops: A downstream failure causes clients to retry requests, creating a surge that overloads recovering services. This is common in agentic systems where one agent's failure triggers others.
• Hot Keys / Herding: A sudden, viral user query or prompt overwhelms a specific, non-scalable part of the system (e.g., a search index for a trending topic).
• Deployment & Traffic Shifts: A canary deployment of a new model version with a hidden performance bug, or a sudden shift in user traffic patterns that exposes a scaling limit.
• Resource Contention: Noisy neighbor problems in multi-tenant clusters, where one tenant's high-load job degrades Tokens per Second (TPS) for others sharing GPU resources.

A Service Level Objective is a target value or range for a Service Level Indicator that defines the acceptable performance of an LLM service. In RCA, SLOs provide the benchmark against which incidents are measured. Violations trigger the RCA process.

• Example SLOs: "99% of requests must have a P99 latency under 2 seconds" or "Hallucination rate must remain below 1%."
• Error Budgets: Derived from SLOs, an error budget quantifies the allowable unreliability before an SLO is breached, guiding the urgency and scope of an RCA.

Statistical Process Control is a quality control methodology using statistical tools, like control charts, to monitor process behavior. In LLM monitoring, SPC helps distinguish normal metric variance from significant anomalies that warrant RCA.

• Control Charts: Plot metrics like latency, token throughput, or output scores over time with calculated control limits (e.g., 3-sigma).
• Special Cause Variation: Points outside control limits or showing non-random patterns indicate a process shift, triggering investigation.
• Proactive RCA: SPC enables the detection of gradual concept drift or output drift before it causes a major SLO violation.

These are controlled release strategies that provide comparative data, making RCA for new model versions more precise and lower risk.

• Canary Deployment: A new model version serves a small percentage of live traffic. Its performance (latency, error rate) is compared in real-time to the baseline, allowing rapid rollback if issues are detected.
• Shadow Deployment: The new version processes all live requests in parallel, but its outputs are discarded. This allows full-scale performance and correctness (e.g., via a golden dataset) comparison with zero user impact, generating rich data for pre-emptive RCA.
• Traffic Routing: Both strategies rely on sophisticated traffic management to split and route requests.

Mean Time to Recovery is a key reliability metric measuring the average time to restore service after an incident. RCA is a major component of MTTR, specifically the "diagnosis" and "remediation" phases.

• MTTR Breakdown:
  • Detection Time: From incident start to alert.
  • Diagnosis Time: RCA to identify root cause.
  • Mitigation Time: Implementing a fix or workaround.
  • Remediation Time: Applying a permanent solution.
• RCA's Goal: To reduce diagnosis time through effective tooling (tracing, logging) and processes, and to reduce future MTTR by implementing corrective actions that prevent recurrence.

A feedback loop is a system that collects user interactions and corrections on model outputs to improve the system. It is both a source of data for RCA and the mechanism for implementing long-term corrective actions identified by RCA.

• Data for RCA: User thumbs-down, corrections, or reported issues provide direct signals of model failures or degradations.
• Implementing Fixes: RCA may conclude that a concept drift has occurred. The feedback loop provides the labeled data needed for fine-tuning or updating a retrieval-augmented generation index.
• Closing the Loop: Effective RCA ensures feedback is analyzed and transformed into model or pipeline improvements, making the system more resilient over time.