RAG Answer Faithfulness Trace Audit Prompt

Quality engineers and RAG developers use this prompt to audit a single production trace for answer faithfulness. The prompt compares the final generated answer against every piece of retrieved evidence in the trace, producing a structured faithfulness score, a list of hallucinated or omitted claims, and a comparison against any available human faithfulness judgment. Use this prompt when you need to diagnose why a RAG system produced an unfaithful answer, calibrate automated faithfulness metrics, or build a review queue for high-risk responses.

This is a diagnostic trace-analysis prompt, not a real-time guard prompt. It assumes you already have a logged trace containing the user query, retrieved passages, final answer, and optional human rating. The prompt works best when traces include the full retrieved context—not just the top-k chunks—so the auditor can distinguish between 'evidence was retrieved but ignored' and 'evidence was never retrieved.' Run this prompt on sampled traces from production, on eval failures flagged by automated monitoring, or on user-reported hallucinations. Do not use this prompt as a streaming guardrail; its strength is thorough post-hoc analysis, not low-latency intervention.

Before running this prompt, verify that your trace contains the complete retrieved context and the final generated answer. If the trace is missing retrieval metadata or only stores truncated context, the faithfulness audit will produce unreliable results. After running the audit, feed the structured output into your evaluation pipeline: compare automated faithfulness scores against human judgments to calibrate thresholds, track hallucination categories over time, and route high-severity findings to a human review queue. Avoid using this prompt on traces where the answer was intentionally creative or speculative—faithfulness auditing requires a groundable answer against specific evidence.

This prompt template is designed to be pasted directly into your trace analysis tool, evaluation pipeline, or LLM-as-judge harness. It instructs the model to act as a quality engineer reviewing a complete retrieval-to-generation trace. The model must compare every factual claim in the final answer against the retrieved context, produce a faithfulness score, and flag specific hallucinations or omissions. Use square-bracket placeholders to inject the trace data, scoring rubric, and output schema before execution.

textCopy
You are a quality engineer auditing a RAG (Retrieval-Augmented Generation) system. Your task is to review the full production trace below and assess whether the final generated answer is faithful to the retrieved evidence.

## INPUT

### User Query
[USER_QUERY]

### Retrieved Context
[RETRIEVED_CONTEXT]

### Generated Answer
[GENERATED_ANSWER]

### Trace Metadata (optional)
[TRACE_METADATA]

## INSTRUCTIONS

1. Extract every factual claim from the Generated Answer. A factual claim is any statement that asserts something about the world, including dates, names, numbers, events, relationships, or properties.

2. For each claim, search the Retrieved Context for supporting evidence. A claim is supported only if the context directly states it or it can be logically inferred from the context without adding external knowledge.

3. Classify each claim into one of three categories:
   - **Grounded**: The claim is directly supported by the Retrieved Context.
   - **Hallucinated**: The claim contradicts the Retrieved Context or introduces information not present in the context.
   - **Omitted**: The Retrieved Context contains information relevant to the User Query that the Generated Answer failed to include.

4. Produce a faithfulness score from 0.0 to 1.0, where 1.0 means every claim is grounded and no relevant context was omitted. Use this formula: (Number of Grounded Claims) / (Number of Grounded Claims + Number of Hallucinated Claims). Penalize omissions separately in the omission flag list.

5. For each hallucinated claim, explain why it is unsupported and cite the specific passage from the Retrieved Context that contradicts it or the absence of supporting evidence.

6. For each omitted fact, cite the specific passage from the Retrieved Context that contains the missing information.

## CONSTRAINTS

- Do not use any knowledge outside the Retrieved Context.
- If the Retrieved Context is empty or irrelevant, mark all claims as hallucinated and set the faithfulness score to 0.0.
- If the Generated Answer is empty, return a score of 0.0 and note that no answer was provided.
- Do not penalize the answer for style, grammar, or fluency. Only assess factual faithfulness.
- If the User Query is ambiguous, note the ambiguity but still assess claims against the context as provided.

## OUTPUT SCHEMA

Return a JSON object with this exact structure:

{
  "faithfulness_score": float,
  "total_claims": int,
  "grounded_claims": int,
  "hallucinated_claims": int,
  "omitted_facts": int,
  "claim_analysis": [
    {
      "claim_text": string,
      "classification": "grounded" | "hallucinated",
      "supporting_evidence": string | null,
      "contradicting_evidence": string | null,
      "explanation": string
    }
  ],
  "omission_flags": [
    {
      "omitted_fact": string,
      "source_passage": string,
      "relevance": string
    }
  ],
  "overall_assessment": string
}

## EXAMPLES

### Example 1: Fully Faithful Answer
User Query: "What is the capital of France?"
Retrieved Context: "Paris is the capital and most populous city of France."
Generated Answer: "The capital of France is Paris."

Output:
{
  "faithfulness_score": 1.0,
  "total_claims": 1,
  "grounded_claims": 1,
  "hallucinated_claims": 0,
  "omitted_facts": 0,
  "claim_analysis": [
    {
      "claim_text": "The capital of France is Paris.",
      "classification": "grounded",
      "supporting_evidence": "Paris is the capital and most populous city of France.",
      "contradicting_evidence": null,
      "explanation": "The claim directly matches the retrieved context."
    }
  ],
  "omission_flags": [],
  "overall_assessment": "All claims are grounded. No omissions detected."
}

### Example 2: Hallucinated Claim
User Query: "When was the Eiffel Tower built?"
Retrieved Context: "The Eiffel Tower was completed in 1889."
Generated Answer: "The Eiffel Tower was built in 1887 and opened in 1889."

Output:
{
  "faithfulness_score": 0.5,
  "total_claims": 2,
  "grounded_claims": 1,
  "hallucinated_claims": 1,
  "omitted_facts": 0,
  "claim_analysis": [
    {
      "claim_text": "The Eiffel Tower was built in 1887.",
      "classification": "hallucinated",
      "supporting_evidence": null,
      "contradicting_evidence": "The Eiffel Tower was completed in 1889.",
      "explanation": "The context states completion in 1889, not 1887. The 1887 date is unsupported."
    },
    {
      "claim_text": "The Eiffel Tower opened in 1889.",
      "classification": "grounded",
      "supporting_evidence": "The Eiffel Tower was completed in 1889.",
      "contradicting_evidence": null,
      "explanation": "Completion in 1889 supports the opening claim."
    }
  ],
  "omission_flags": [],
  "overall_assessment": "One hallucinated claim detected. The 1887 construction date is not supported by the retrieved context."
}

## RISK LEVEL
[HIGH] - Faithfulness audits are used to detect hallucination in production RAG systems. Incorrect scores can lead to overconfident deployment of unreliable answers. Always validate a sample of audit outputs against human judgments before relying on automated scores for gating decisions.

After pasting this template, replace each square-bracket placeholder with real trace data. The [RETRIEVED_CONTEXT] should include the full text of every retrieved passage, not just IDs or summaries. If your trace system captures reranking scores or metadata filters, include those in [TRACE_METADATA] to help the model understand retrieval quality. For high-stakes domains such as healthcare or legal, always run a human review sample against the automated faithfulness scores to calibrate the model's judgment before trusting the output for gating or monitoring decisions.

The faithfulness audit prompt is not a one-off debugging tool; it belongs inside a structured evaluation harness that runs against production traces or sampled sessions. The harness must feed the prompt a complete trace object containing the user query, all retrieved chunks with their metadata, and the final generated answer. Before calling the model, validate that the trace includes retrieval timestamps, chunk rankings, and source identifiers. If any required field is missing, the harness should log the incomplete trace and skip evaluation rather than producing a misleading faithfulness score. This prompt works best with models that have strong instruction-following and structured output capabilities, such as GPT-4o, Claude 3.5 Sonnet, or equivalent. Avoid running it on models smaller than ~7B parameters without extensive few-shot calibration, as hallucination detection requires nuanced reasoning about evidence boundaries.

Wire the prompt into a batch evaluation pipeline that processes traces in groups, with each trace evaluated independently. The harness should enforce a strict JSON output schema using the model's native structured output mode or a post-processing validator. Define the expected schema fields: faithfulness_score (0-1 float), hallucination_flags (array of objects with claim, evidence_gap, severity), omission_flags (array of objects with missing_fact, available_evidence, impact), and comparison_to_human_judgment (object with agreement, discrepancy_notes). After each model call, validate that all required fields are present, scores are within range, and flag arrays contain the expected subfields. If validation fails, retry once with the same trace and a stronger constraint instruction appended to the prompt. If the retry also fails, route the trace to a human review queue with the raw trace, the failed output, and the validation error details. Log every evaluation attempt, including the prompt version, model, trace ID, raw output, validation result, and final disposition, so that evaluation quality itself can be audited over time.

For high-stakes domains such as healthcare, legal, or finance, do not rely solely on the model's faithfulness score. The harness should include a human review gate for any trace where the faithfulness score falls below 0.8, where hallucination flags carry severity: critical, or where the model's judgment disagrees with a prior human faithfulness label. Route these traces to a review interface that displays the original answer, the retrieved evidence side by side, and the model's flagged claims with evidence gaps highlighted. The reviewer can confirm, override, or annotate each flag. Store the human-reviewed result as ground truth for future eval calibration. Over time, this feedback loop improves both the prompt's accuracy and the team's understanding of where the RAG pipeline produces unfaithful answers. Avoid wiring this prompt into synchronous user-facing flows; the audit is an offline evaluation step, not a real-time guardrail. For continuous monitoring, schedule the harness to run on a sample of production traces daily, with aggregate faithfulness metrics pushed to your observability dashboard alongside retrieval quality and latency metrics.