Hallucination Root Cause Diagnosis Prompt

This prompt is for prompt engineers and AI reliability engineers who need to move beyond detecting hallucinations to understanding why they occur. Use it when you have a batch of hallucinated outputs and need to classify the root cause before applying a fix. This is a diagnostic tool, not a repair prompt. It assumes you have already identified hallucinated examples and collected the corresponding prompt, context, and model configuration. The output is a structured root cause analysis that tells you whether to fix the prompt, improve retrieval, add guardrails, or accept a model limitation.

Do not use this prompt when you need to repair a single hallucinated output in production—use a retry or regeneration prompt instead. Do not use it when you haven't yet confirmed that the output is actually hallucinated; run a fact-checking or claim-verification prompt first. This prompt is most effective when you have at least 5–10 confirmed hallucination examples from the same prompt template, because pattern analysis requires a sample size large enough to distinguish systematic failures from random noise. Before running this prompt, gather the original user input, the full prompt template with placeholders resolved, any retrieved context or tool outputs, the model ID and configuration, and the flagged hallucinated output.

The prompt produces a structured root cause classification—such as context insufficiency, prompt ambiguity, knowledge cutoff issues, or reasoning failures—along with a confidence score and a specific, actionable fix recommendation. Wire the output into your prompt debugging workflow: if the root cause is context insufficiency, improve retrieval or chunking; if it's prompt ambiguity, rewrite the instructions; if it's a knowledge cutoff, add up-to-date context or switch models. For high-stakes domains, always have a human review the diagnosis before applying the recommended fix to production prompts.

The prompt below is designed to be pasted directly into your AI harness or playground. It forces the model to move beyond surface-level correction and instead perform a systematic diagnosis of why a hallucination occurred. The output is a structured JSON report suitable for logging, automated routing, or feeding into a downstream repair pipeline. Replace every square-bracket placeholder with your actual data before execution.

textCopy
You are a hallucination root cause diagnostician. Your job is to analyze a specific instance where a model produced a hallucinated or factually incorrect output. You will receive the original prompt, the provided context (if any), the model's incorrect output, and the ground-truth correction. You must classify the root cause, provide evidence for your classification, and recommend a concrete fix.

## INPUTS
- **Original Prompt:** [ORIGINAL_PROMPT]
- **Provided Context:** [PROVIDED_CONTEXT]
- **Incorrect Model Output:** [INCORRECT_OUTPUT]
- **Ground-Truth Correction:** [GROUND_TRUTH_CORRECTION]

## DIAGNOSIS DIMENSIONS
Analyze the failure across these dimensions. For each, state whether it contributed (YES/NO) and provide a brief evidence snippet.
1. **Context Insufficiency:** The provided context was missing the necessary fact, was ambiguous, or was truncated.
2. **Context Misinterpretation:** The context contained the fact, but the model misread, conflated entities, or drew an incorrect inference.
3. **Prompt Ambiguity:** The original prompt was underspecified, contradictory, or encouraged fabrication (e.g., "be creative").
4. **Knowledge Cutoff / Parametric Hallucination:** The model relied on outdated or incorrect internal knowledge instead of the provided context.
5. **Reasoning Failure:** The model combined facts incorrectly, made a logical error, or failed at multi-hop synthesis.
6. **Format or Constraint Pressure:** Output constraints (e.g., "must be under 50 words") forced the model to drop critical qualifiers or invent details.

## OUTPUT SCHEMA
Return a single JSON object with this exact structure:
{
  "failure_id": "[FAILURE_ID]",
  "primary_root_cause": "Context Insufficiency | Context Misinterpretation | Prompt Ambiguity | Knowledge Cutoff | Reasoning Failure | Format Pressure",
  "diagnosis": {
    "context_insufficiency": { "contributed": true/false, "evidence": "string" },
    "context_misinterpretation": { "contributed": true/false, "evidence": "string" },
    "prompt_ambiguity": { "contributed": true/false, "evidence": "string" },
    "knowledge_cutoff": { "contributed": true/false, "evidence": "string" },
    "reasoning_failure": { "contributed": true/false, "evidence": "string" },
    "format_pressure": { "contributed": true/false, "evidence": "string" }
  },
  "fix_recommendation": {
    "action": "REWRITE_PROMPT | ADD_CONTEXT | ADD_EXAMPLE | ADD_CONSTRAINT | ESCALATE_TO_HUMAN | FINE_TUNE | NO_FIX_NEEDED",
    "specific_change": "A concrete, one-sentence instruction for what to change.",
    "revised_prompt_snippet": "Optional. A corrected excerpt of the prompt or constraint."
  },
  "severity": "LOW | MEDIUM | HIGH | CRITICAL",
  "retry_likely_to_fix": true/false
}

## CONSTRAINTS
- Base your diagnosis strictly on the provided inputs. Do not invent external facts.
- If multiple dimensions contributed, set `primary_root_cause` to the most impactful one and note the others in the diagnosis fields.
- If the provided context is empty, `context_insufficiency.contributed` must be true.
- Do not wrap the JSON in markdown fences. Return only the raw JSON object.

To adapt this template, start by populating the four input fields with data from your production logs or evaluation harness. The [FAILURE_ID] should be a unique identifier from your tracing system so you can link the diagnosis back to a specific request. If your application does not use explicit context windows, set [PROVIDED_CONTEXT] to an empty string—the prompt will correctly flag context insufficiency. For batch analysis, wrap this prompt in a loop that iterates over logged failure records and writes each diagnosis to your observability platform. Always validate the output JSON against the schema before accepting the diagnosis; a malformed fix_recommendation object can break downstream automation.

This prompt is a diagnostic tool, not a repair tool. After classification, route the fix_recommendation.action to the appropriate workflow: REWRITE_PROMPT goes to your prompt registry, ADD_CONTEXT goes to your retrieval pipeline, and ESCALATE_TO_HUMAN should trigger a review ticket. For high-severity findings where retry_likely_to_fix is false, do not loop—escalate immediately. Run this diagnosis on a sample of failures weekly to detect systemic drift before it impacts users.

This prompt is designed to be called programmatically after a hallucination event has been detected by an upstream validator or LLM judge. The harness should assemble the required inputs—the original user query, the full prompt template with system instructions, the retrieved context (if RAG was used), and the flagged hallucinated output—into a single structured request. Because the diagnosis prompt itself performs analysis rather than taking action, the harness should treat its output as a structured diagnostic record, not as a command to automatically modify production prompts. The recommended integration pattern is a post-detection callback: when a hallucination is flagged with a confidence score above a defined threshold, the harness fires this diagnosis prompt, logs the root cause classification, and routes the finding to the appropriate remediation queue.

Input Assembly and Validation: Before calling the diagnosis prompt, validate that all required fields are present and non-empty. The [HALLUCINATED_OUTPUT] must be the exact text that failed validation, not a summary. The [PROMPT_TEMPLATE] should include the full system prompt and any few-shot examples, as root causes often hide in instruction ambiguity or example conflict. The [RETRIEVED_CONTEXT] field must be the raw context chunks provided to the model, not post-hoc retrieved passages, to accurately diagnose context insufficiency. Implement a pre-flight schema check that rejects the diagnosis request if any field is missing or truncated. For high-volume production systems, batch multiple hallucination events and run diagnosis in a separate async worker to avoid adding latency to the primary response path. Use a structured output constraint on the diagnosis prompt itself—require JSON output with root_cause_category, confidence, evidence, and fix_recommendation fields—so the harness can parse and route results without manual inspection.

Retry, Logging, and Escalation: The diagnosis prompt should be treated as a non-retryable analysis step. If the model fails to return valid JSON or the output schema validation fails, log the raw response and escalate to a human reviewer rather than retrying, because a malformed diagnosis provides its own signal about prompt fragility. Log every diagnosis result with a composite key of [timestamp, session_id, hallucination_id] to enable trend analysis over time. When the diagnosis returns root_cause_category: context_insufficiency with high confidence, route to the retrieval pipeline team for context quality review. When it returns prompt_ambiguity or instruction_conflict, route to the prompt engineering backlog. Implement a severity threshold: if the same root cause category appears more than N times within a rolling window, trigger an automated alert to the on-call channel. Do not automatically apply the fix_recommendation field to production prompts without human review—the diagnosis is advisory, and prompt changes require regression testing against your golden evaluation set. The harness should write the fix recommendation into a prompt versioning system as a proposed change, not a direct mutation.