Source-By-Source Synthesis Audit Prompt

This prompt is a post-generation verification harness for AI quality engineers who need an automated, structured audit trail before a RAG-generated answer reaches a user or a downstream system. Its job is to take a synthesized answer and the exact set of source passages that were provided to the generation model, then map every claim in the answer to an evidence span, flag unsupported statements, and score overall faithfulness. The ideal user is an evaluation engineer, a platform reliability engineer, or a product developer embedding a guardrail into a retrieval-augmented generation pipeline. Required context includes the full generated answer text and the complete, unmodified list of source passages that were in the model's context window at generation time. Without both, the audit cannot produce a valid claim-to-evidence mapping.

Use this prompt when you need a deterministic, evidence-backed audit report that can be logged, reviewed, or used as a gating condition before publishing an answer. It is appropriate for high-stakes domains—such as legal research, clinical decision support, financial analysis, or compliance documentation—where an unsupported claim can cause real harm. The prompt is designed to be wired into an automated pipeline: after answer generation, the audit runs, and if the faithfulness score falls below a threshold or unsupported claims are detected, the system can either regenerate, escalate for human review, or refuse to surface the answer. It is also useful during offline evaluation runs, where you need to measure grounding fidelity across a test suite of question-answer-source triples.

Do not use this prompt as a general-purpose hallucination detector for open-domain chat or for answers generated without explicit source passages. It assumes a closed-world evidence set: every claim must be traceable to at least one provided passage, and anything outside that set is flagged as unsupported. It is not designed to detect factual errors where the source itself is wrong, nor to assess whether the retrieved passages are sufficient to answer the question—those are separate concerns handled by evidence sufficiency and fact-checking prompts. If your generation pipeline uses multiple retrieval steps, re-ranking, or source fusion, ensure the exact passages seen by the generation model are captured and passed to this audit prompt without modification. The audit's value depends entirely on the integrity of that evidence snapshot.

The prompt below is designed to be dropped into your evaluation harness as a post-generation verification step. It instructs the model to act as an auditor, mapping each factual claim in a generated answer back to a specific evidence span in the provided source passages. Replace every square-bracket placeholder with your actual data before running the audit. The prompt is structured to produce a consistent JSON report that your application can parse, log, and use to gate whether the answer is shown to the user or sent for repair.

textCopy
You are a source-by-source synthesis auditor. Your job is to verify that every factual claim in a generated answer is directly supported by at least one provided source passage. You do not evaluate the quality of the answer itself—only whether each claim can be traced to evidence.

## INPUT
- Generated Answer: [GENERATED_ANSWER]
- Source Passages (with IDs): [SOURCE_PASSAGES]
- Claim Types to Audit: [CLAIM_TYPES]

## OUTPUT SCHEMA
Return a JSON object with the following structure:
{
  "audit_report": {
    "total_claims_found": <integer>,
    "supported_claims": <integer>,
    "unsupported_claims": <integer>,
    "partially_supported_claims": <integer>,
    "claims": [
      {
        "claim_id": "<string>",
        "claim_text": "<exact text from the answer>",
        "claim_type": "<factual | interpretive | quantitative | temporal | attribution>",
        "support_status": "<supported | unsupported | partially_supported>",
        "supporting_source_ids": ["<source_id>", ...],
        "supporting_evidence_spans": ["<exact quote from source>", ...],
        "auditor_notes": "<explanation of why this claim is or is not supported>"
      }
    ],
    "unsupported_claim_summary": "<summary of what the answer claims without evidence>",
    "overall_faithfulness_score": <float between 0.0 and 1.0>
  }
}

## CONSTRAINTS
- Extract every factual claim from the generated answer. A claim is any statement that asserts something true about the world.
- For each claim, search all source passages for direct supporting evidence. The evidence must be a verbatim span or a clear paraphrase that does not add or distort information.
- If a claim is partially supported (some parts have evidence, others do not), mark it as partially_supported and note which parts lack evidence.
- Do not mark a claim as supported if the source only implies it. Support must be explicit.
- If no source passage supports a claim, mark it as unsupported and leave supporting_source_ids and supporting_evidence_spans empty.
- The overall_faithfulness_score is the proportion of supported claims (counting partially_supported as 0.5) divided by total claims.
- If the generated answer contains no factual claims, return an empty claims array with a score of 1.0 and note that no claims were found.

## EXAMPLES
[FEW_SHOT_EXAMPLES]

## RISK LEVEL
[RISK_LEVEL]

To adapt this template, start by populating [GENERATED_ANSWER] with the full text output from your RAG pipeline or LLM call. Populate [SOURCE_PASSAGES] with the retrieved passages that were provided as context during generation, each labeled with a unique source ID. The [CLAIM_TYPES] field lets you scope the audit to specific categories—for example, you might only audit factual and quantitative claims if interpretive statements are expected to be subjective. The [FEW_SHOT_EXAMPLES] placeholder should contain 2–3 worked examples showing the model exactly how to extract claims, match evidence spans, and assign support statuses. These examples are critical for consistent output structure. The [RISK_LEVEL] field should be set to high, medium, or low to adjust the auditor's strictness—at high risk, even minor unsupported qualifiers should be flagged.

Before deploying this prompt into a production pipeline, validate the output against a golden dataset of known-supported and known-unsupported answer-source pairs. Measure precision (are flagged unsupported claims actually unsupported?) and recall (are all unsupported claims caught?). Common failure modes include the auditor missing claims embedded in compound sentences, accepting source paraphrases that subtly change meaning, and failing to detect when a claim combines information from multiple sources in a way that neither source individually supports. For high-stakes domains such as healthcare, legal, or financial applications, always route audit reports with unsupported claims to a human reviewer before the answer reaches the end user. Log every audit report with the answer version, source set, and timestamp for downstream traceability.

The Source-By-Source Synthesis Audit Prompt is designed as a post-generation verification harness, not a user-facing prompt. It should be called after your primary RAG pipeline produces an answer, receiving both the generated answer and the full set of retrieved source passages as inputs. The prompt's job is to produce a structured audit report that maps every claim in the answer back to specific evidence spans, flags unsupported or hallucinated statements, and computes a synthesis faithfulness score. This makes it suitable for automated CI/CD eval pipelines, pre-release answer quality gates, and production monitoring where every user-facing answer must pass a grounding check before delivery.

Wire this prompt into your application as a secondary model call that runs in the verification layer. The primary flow looks like: (1) retrieve passages, (2) generate the user-facing answer, (3) call the audit prompt with the answer and retrieved passages, (4) parse the structured audit output, and (5) decide whether to deliver, retry, or escalate. For the audit call, use a model with strong instruction-following and structured output capabilities—GPT-4o, Claude 3.5 Sonnet, or equivalent. Set response_format to JSON mode or use structured output APIs to enforce the audit schema. The audit output should include at minimum: a list of claims extracted from the answer, per-claim source spans with passage IDs and quote excerpts, a binary supported/unsupported flag per claim, and an aggregate faithfulness score. Validate the output schema before proceeding—if the model fails to produce valid JSON or the claim count doesn't match the answer's sentence count, retry once with a simplified schema or escalate to human review.

Build a decision gate around the audit results. If the faithfulness score falls below your threshold (start with 0.85 and calibrate based on domain risk), do not deliver the answer to the user. Instead, either regenerate the answer with stricter grounding instructions, expand retrieval to pull more passages, or surface a qualified response that explicitly flags unsupported claims. For high-risk domains such as legal, medical, or financial applications, any unsupported claim should trigger human review regardless of the aggregate score. Log every audit result—claim-level support flags, source passage IDs, and the raw audit output—to enable traceability, regression testing, and prompt improvement over time. This audit trail is also your primary defense when stakeholders ask whether the AI's answers are actually grounded in evidence.

Common failure modes to instrument for: the audit prompt itself hallucinating source spans that don't exist in the provided passages (validate that every returned passage ID and quote excerpt actually appears in the input), the audit prompt missing subtle unsupported claims that are paraphrased rather than invented (use spot-check evals comparing audit results to human annotations), and the audit prompt being overly strict and flagging reasonable inferences as unsupported (tune your threshold and consider a severity scale: fully supported, partially supported, unsupported, contradicted). Run this audit prompt against a golden dataset of known-good and known-bad answer-passage pairs before deploying, and include it in your regression test suite so that prompt changes to either the generation or audit step are caught before they degrade grounding quality.