Eval Failure Root-Cause Triage Prompt

This prompt is designed for AI reliability engineers and platform teams who need to quickly diagnose why a single production response failed an automated evaluation. The core job-to-be-done is to move from an alert—a failed eval score—to a structured, evidence-backed hypothesis about the root cause. It assumes you are not starting from scratch; you must already have access to the specific trace data for that request, including step-level attribution (retrieval, tool calls, model inference), the final model output, and the detailed results of the evaluation rubric that flagged the failure. This is a focused diagnostic tool, not a general-purpose analysis assistant.

Use this prompt when an eval alert fires in your monitoring dashboard and you need to rule out common failure categories before committing to a deep-dive investigation. For example, if a 'faithfulness' eval fails, you can feed the trace's retrieval steps and the final output into this prompt to determine if the root cause was a retrieval gap, a context window truncation, or a genuine hallucination. Do not use this prompt for aggregate trend analysis across thousands of sessions, for real-time incident response where trace data is not yet available, or for cases where you suspect the evaluation rubric itself is misconfigured and producing false positives. In those scenarios, you would first need to validate the eval or gather broader data before a per-trace triage is useful.

The prompt's value is in its structured output, which forces a systematic review rather than an ad-hoc scan of logs. It will ask you to correlate specific eval dimensions with trace events, consider a checklist of common failure modes (e.g., prompt injection, tool-call errors, stale few-shot examples), and produce a ranked list of hypotheses. The next step after using this prompt is to take the top hypothesis and conduct a targeted deep-dive, perhaps using a more specialized prompt like the 'Retrieval Gap Root-Cause for Eval Failure Prompt' or the 'Tool-Call Error Correlation with Eval Failure Prompt.' Avoid the temptation to skip this triage step and jump straight to a solution, as a misdiagnosed root cause often leads to a prompt change that fixes the symptom but introduces a new failure mode elsewhere.

The template below is designed to be pasted directly into your investigation notebook or wired into an automated triage pipeline. It accepts a failed eval record, the corresponding production trace, and your quality rubric, then produces a structured root-cause hypothesis. Every placeholder is enclosed in square brackets and must be replaced with real data before execution. The prompt forces the model to rule out common failure categories before proposing a deep-dive investigation path, reducing the risk of premature conclusions.

textCopy
You are an AI reliability engineer investigating a production eval failure.
Your job is to produce a structured root-cause hypothesis by correlating the eval score, trace events, and model output.

## INPUT

### Failed Eval Record
[EVAL_RECORD]

### Production Trace
[TRACE_DATA]

### Quality Rubric
[QUALITY_RUBRIC]

### Additional Context (optional)
[ADDITIONAL_CONTEXT]

## CONSTRAINTS
- Do not speculate beyond the evidence in the trace.
- Rule out common failure categories before proposing a deep-dive path.
- If the trace is incomplete, flag missing segments explicitly.
- Distinguish between prompt defects, retrieval gaps, tool failures, model behavior changes, and context truncation.
- Cite specific trace events, timestamps, or step IDs as evidence.

## OUTPUT SCHEMA
Return a JSON object with the following fields:
{
  "failure_summary": "One-sentence description of what failed and which eval dimension was violated.",
  "ruled_out_categories": [
    {
      "category": "prompt_defect | retrieval_gap | tool_failure | model_behavior_change | context_truncation | format_drift | hallucination | other",
      "ruled_out": true,
      "evidence": "Trace evidence supporting exclusion."
    }
  ],
  "primary_hypothesis": {
    "category": "prompt_defect | retrieval_gap | tool_failure | model_behavior_change | context_truncation | format_drift | hallucination | other",
    "confidence": "low | medium | high",
    "trace_evidence": ["Specific trace event IDs, timestamps, or step descriptions that support this hypothesis."],
    "explanation": "How the identified root cause led to the eval failure."
  },
  "alternative_hypotheses": [
    {
      "category": "...",
      "confidence": "low | medium",
      "trace_evidence": ["..."],
      "explanation": "..."
    }
  ],
  "missing_trace_data": ["List any trace segments or metadata that would strengthen the diagnosis if available."],
  "recommended_investigation": "Concrete next step: which trace segment to deep-dive, which metric to check, or which comparison to run.",
  "severity": "critical | high | medium | low",
  "affected_eval_dimensions": ["List which rubric dimensions were violated."]
}

## STEPS
1. Parse the eval record to identify which rubric dimensions failed and the severity of each failure.
2. Scan the trace for the step where the failure likely originated (retrieval, tool call, generation, routing, truncation).
3. For each common failure category, check whether trace evidence rules it out.
4. Identify the most likely root cause and cite specific trace events.
5. Note any alternative explanations that cannot be ruled out with available data.
6. Flag missing trace data that would improve diagnostic confidence.
7. Recommend a concrete next investigation step.

To adapt this template, replace each square-bracket placeholder with real data from your observability stack. The [EVAL_RECORD] should include the eval score, the dimension that failed, and the evaluator's output. The [TRACE_DATA] must contain step-level events with timestamps, tool-call arguments, retrieved documents, and the final model output. If your trace format differs, adjust the trace_evidence fields to reference your own event IDs or span IDs. The output schema is designed to be machine-readable so you can pipe results into an incident tracker or dashboard. Before shipping this prompt into an automated pipeline, validate that the JSON output parses correctly and that the model consistently populates ruled_out_categories before proposing a primary hypothesis.

This prompt is designed to be the first step in a structured investigation, not a standalone report generator. It should be wired into a workflow that fetches the relevant trace data, eval rubric, and model output automatically. The prompt's primary job is to produce a structured hypothesis and a checklist of ruled-out categories, which then guides deeper, more expensive diagnostic steps. Avoid using this prompt in a vacuum; its value comes from correlating data that is often siloed in separate observability, evaluation, and logging systems.

To implement this, build a harness function that accepts an eval_failure_event as input. This function must first query your trace store (e.g., LangSmith, Arize, a custom OpenTelemetry collector) to retrieve the full trace for the specific trace_id associated with the failure. It must also fetch the exact evaluation rubric and scorecard from your evaluation platform. The harness then populates the prompt's placeholders: [EVAL_RUBRIC] with the specific criteria that failed, [TRACE_JSON] with a serialized, step-by-step trace including tool calls, retrieved documents, and intermediate outputs, and [MODEL_OUTPUT] with the final production response. The [CONSTRAINTS] placeholder should be pre-filled with a standard instruction to rule out common failure categories (e.g., context truncation, tool-call errors, source-citation gaps) before proposing a root cause.

The output of this prompt is a structured JSON object containing a root_cause_hypothesis, a confidence_score, and a ruled_out_categories list. Your harness must validate this JSON against a strict schema. If validation fails, implement a single retry with the validation error message fed back into the [PREVIOUS_OUTPUT_AND_ERROR] placeholder. Crucially, log the validated hypothesis back to the original eval event as a diagnostic tag. This closes the loop between monitoring and investigation, allowing you to track whether context_window_truncation is a recurring hypothesis across multiple failures. For high-stakes domains like healthcare or finance, the harness should route the final hypothesis to a human-in-the-loop review queue before any prompt or system changes are authorized based on its findings.

For model choice, a model with strong reasoning capabilities and a large context window is required, as the prompt must ingest a potentially massive trace. GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro are suitable starting points. Do not use a lightweight model for this task, as it will likely fail to correlate events across a long trace. The primary failure mode of this harness is an incomplete trace: if your tracing infrastructure does not capture step-level inputs and outputs, the prompt will produce a low-confidence hypothesis. Your next step should be to instrument your application to capture the missing data, not to refine the prompt.