Evidence Grounding Failure Mode Analysis Prompt

This prompt is a diagnostic tool for AI observability pipelines. Use it when a cited answer from a RAG system, document intelligence workflow, or any evidence-grounded generation fails validation. The failure could be a missing citation, an unsupported claim, a hallucinated source, or a quote that does not match its reference. Instead of blindly retrying the generation, this prompt classifies the specific grounding failure mode and recommends a targeted recovery action. It is designed for platform engineers, AI reliability teams, and MLOps engineers who need to route failures to the correct repair prompt, not for end-users.

This prompt assumes you have already captured three inputs: the failed answer text, the evidence or source passages provided to the model, and the validator error that flagged the output. Without all three, the classification will be unreliable. The prompt produces a structured failure mode label from a fixed taxonomy—such as missing_citation, unsupported_claim, hallucinated_source, quote_mismatch, or evidence_insufficient—along with a recommended recovery action and the name of the specific retry prompt to invoke next. This turns an opaque failure into a routable event.

Do not use this prompt when the failure is a format or schema error rather than a grounding problem. If the output is valid JSON with broken citations, use this prompt. If the output is malformed JSON that a parser rejects, use a structured output repair prompt instead. Do not use this prompt directly in a user-facing retry loop without a retry budget and escalation path. Repeated classification without recovery wastes compute and latency. Wire this prompt into a harness that invokes the recommended recovery prompt, re-validates the result, and escalates to human review after a defined number of attempts.

This prompt template is designed to be wired into an AI observability or debugging harness. It receives a failed cited answer and its associated evidence, then diagnoses the primary grounding failure mode from a defined taxonomy. The output is a structured classification and a single recommended recovery action, not a regenerated answer. Use this when you need automated triage before invoking a specific repair prompt.

textCopy
You are an evidence grounding auditor for a RAG system. Your job is to diagnose why a generated answer failed its citation and evidence checks. Do not regenerate the answer. Do not correct the facts. Classify the primary failure mode and recommend the single most appropriate recovery action.

## FAILED ANSWER
[FAILED_ANSWER]

## PROVIDED EVIDENCE PASSAGES
[EVIDENCE_PASSAGES]

## FAILURE TAXONOMY
Classify the primary failure into exactly one of the following modes:
- MISSING_CITATION: A factual claim lacks any citation.
- UNSUPPORTED_CLAIM: A cited claim is not supported by the referenced passage.
- QUOTE_INACCURACY: A direct quote does not match the source text.
- HALLUCINATED_SOURCE: A citation references a source not present in the provided evidence.
- SOURCE_CONFLICT: Multiple cited passages contradict each other without resolution.
- EVIDENCE_INSUFFICIENCY: The provided evidence is too thin to support the answer's claims.
- CITATION_DRIFT: A citation is attached to the wrong claim or paragraph.
- OVER_CITATION: A trivial or common-knowledge statement is unnecessarily cited.

## RECOVERY ACTION ROUTING
Based on the failure mode, recommend exactly one recovery action from this list:
- RETRY_WITH_CITATION_REQUIREMENT: Regenerate the answer with mandatory inline citations.
- RETRY_WITH_EVIDENCE_CONSTRAINT: Regenerate using only claims directly traceable to evidence.
- RETRY_WITH_QUOTE_VERIFICATION: Regenerate with exact quote matching against source text.
- RETRY_WITH_EXPANDED_RETRIEVAL: Re-retrieve with broader scope and regenerate.
- RETRY_WITH_CONFLICT_RESOLUTION: Reconcile conflicting sources and regenerate with disclosure.
- FLAG_FOR_HUMAN_REVIEW: Escalate to human review without retrying.
- ABSTAIN_WITH_EXPLANATION: Generate a refusal that explains the evidence gap.

## OUTPUT FORMAT
Return a single JSON object with these fields:
{
  "failure_mode": "<MODE>",
  "confidence": <0.0 to 1.0>,
  "evidence": "<Quote the specific text from the failed answer or evidence that supports this diagnosis>",
  "recommended_action": "<ACTION>",
  "action_rationale": "<One sentence explaining why this action is appropriate>"
}

## CONSTRAINTS
- Classify only the PRIMARY failure mode, even if multiple issues exist.
- The evidence field must contain a verbatim quote from the input.
- If you cannot determine the failure mode with confidence > 0.7, set failure_mode to "UNCLEAR" and recommended_action to "FLAG_FOR_HUMAN_REVIEW".

Adapt this template by replacing the taxonomy with your organization's specific failure categories. The recovery actions should map directly to retry prompts or escalation workflows you have implemented. For high-stakes domains like healthcare or legal, always route FLAG_FOR_HUMAN_REVIEW when confidence is below 0.9, not 0.7. Wire the output JSON into a routing engine that invokes the corresponding recovery prompt or queues the item for human review. Validate that the evidence field contains a substring match against the input before trusting the classification.

This prompt is designed as a diagnostic step within a larger citation recovery harness, not as a standalone chat interaction. It should be invoked after a cited answer has failed a validation gate—such as a citation completeness check, a source-to-claim alignment verification, or a hallucinated source detection scan. The harness should pass the failed answer, the original retrieved evidence, and the specific validator error messages as inputs to this analysis prompt. The output is a structured failure classification and a recommended recovery route, which the harness then uses to select the next retry or escalation action.

Input assembly requires three components: the [FAILED_ANSWER] (the full generated text with its citations), the [RETRIEVED_EVIDENCE] (the source passages provided to the generation model, with identifiers), and the [VALIDATOR_ERRORS] (structured error objects from upstream validation, including the validator name, the failed claim or citation, and the reason for rejection). The prompt template expects these to be injected into the respective placeholders. For high-throughput systems, consider pre-processing the validator errors into a concise summary to stay within context window limits. Model choice should favor models with strong reasoning capabilities for classification tasks; a smaller, faster model may misclassify subtle failure modes like citation drift or evidence stretching.

Output validation is critical before routing. The prompt instructs the model to return a JSON object with failure_mode, confidence, affected_claims, and recommended_recovery. The harness must validate that failure_mode matches one of the entries in your defined taxonomy (e.g., missing_citation, unsupported_claim, hallucinated_source, quote_inaccuracy, source_conflict, evidence_insufficiency). If the model returns an unknown failure mode or a confidence score below your threshold (e.g., < 0.7), do not route to an automated recovery prompt. Instead, escalate to a human review queue with the full analysis payload. This prevents a misdiagnosis from triggering an inappropriate retry loop that wastes compute and degrades the answer further.

Recovery routing uses the recommended_recovery field to select the next prompt in your playbook. Map each recovery type to a specific retry prompt: regenerate_with_citations routes to the Answer Regeneration with Mandatory Citations Prompt; correct_unsupported_claims routes to the Unsupported Claim Correction Prompt; resolve_source_conflict routes to the Source Conflict Resolution Prompt; expand_retrieval routes to the Retry with Expanded Retrieval Window Prompt; and abstain routes to the Answer Abstention with Evidence Explanation Prompt. Implement a retry budget that tracks how many times a given answer has been through the analysis-recovery loop. After a configurable maximum (e.g., 3 attempts), break the cycle and escalate to a human operator with the full trace of failure modes and recovery actions attempted.

Observability requires logging the full input and output of this analysis prompt at each invocation. Attach the failure_mode, confidence, and recommended_recovery as span attributes in your tracing system. This data is essential for identifying systemic grounding weaknesses—for example, if a specific retrieval source consistently triggers evidence_insufficiency, or if a particular model version exhibits a spike in hallucinated_source classifications. Use these aggregate signals to improve your retrieval pipeline, fine-tune your generation prompts, or adjust your validation thresholds, rather than treating each failure as an isolated incident.