Factual Consistency Check Against Source Prompt Template

This prompt is designed for RAG system builders and content verification teams who need to programmatically verify whether a generated answer is factually consistent with its source documents. It produces a structured consistency report that classifies every claim in the answer as supported, unsupported, or contradicted by the provided evidence. Use this when you need an automated judge to catch hallucinations before outputs reach users, when you are building evaluation pipelines for groundedness metrics, or when you need audit trails showing exactly which claims trace back to which source passages.

This prompt assumes you already have a generated answer and the source documents it should be grounded in. It does not generate answers, retrieve documents, or make subjective quality judgments about writing style. The ideal user is an AI engineer or evaluation lead who has already built a RAG pipeline and needs a reliable, structured consistency check that can run at scale—either as a pre-release gate in CI/CD or as a continuous monitoring layer in production. The output is designed to be machine-readable, making it suitable for automated alerting, dashboarding, and downstream retry or escalation logic.

Do not use this prompt when you need to generate answers, retrieve documents, or assess subjective qualities like tone, helpfulness, or style. It is also not a replacement for human review in high-stakes domains such as healthcare, legal, or finance—where factual errors carry regulatory or safety consequences. In those contexts, this prompt should feed a human review queue rather than serve as the final arbiter. For workflows requiring pairwise comparison of multiple outputs, use the Pairwise Groundedness Comparison Prompt instead. For sentence-level annotation suitable for training data curation, pair this with the Sentence-Level Grounding Verification Prompt to add granularity to your evaluation pipeline.

This is the core prompt template for the Factual Consistency Check. It instructs the model to act as an evaluator, comparing a generated answer against a set of source documents. The prompt is designed to produce a structured JSON report, making it suitable for direct integration into an automated evaluation harness. Before using it, ensure you have a clear [ANSWER] to evaluate and a set of [SOURCE_DOCUMENTS] that should contain all the information needed to verify the answer.

textCopy
You are a meticulous AI auditor. Your task is to evaluate the factual consistency of a [ANSWER] against the provided [SOURCE_DOCUMENTS].

For each distinct factual claim in the [ANSWER], determine its relationship to the [SOURCE_DOCUMENTS] and classify it into one of the following categories:
- **SUPPORTED:** The claim is directly stated in or can be logically inferred from the sources. Cite the specific source passage(s).
- **UNSUPPORTED:** The claim is not found in the sources. This is a potential hallucination.
- **CONTRADICTED:** The claim directly conflicts with information in the sources. This is a critical error.

# OUTPUT_SCHEMA
Return your evaluation as a single JSON object with the following structure:
{
  "overall_consistency_score": "HIGH" | "MEDIUM" | "LOW",
  "summary": "A one-sentence summary of the evaluation.",
  "claims": [
    {
      "claim_text": "The exact factual claim from the answer.",
      "status": "SUPPORTED" | "UNSUPPORTED" | "CONTRADICTED",
      "source_reference": "Quote the supporting or contradicting text from [SOURCE_DOCUMENTS], or null if UNSUPPORTED.",
      "explanation": "A brief explanation of the reasoning for the classification."
    }
  ]
}

# CONSTRAINTS
- Analyze every sentence in the [ANSWER] for factual claims.
- If a claim is SUPPORTED, you MUST provide a direct quote from the [SOURCE_DOCUMENTS] as the `source_reference`.
- If a claim is UNSUPPORTED, the `source_reference` field must be `null`.
- Do not make judgments about the quality or style of the writing, only its factual consistency.

# INPUTS
<source_documents>
[SOURCE_DOCUMENTS]
</source_documents>

<answer>
[ANSWER]
</answer>

To adapt this template, replace the [SOURCE_DOCUMENTS] and [ANSWER] placeholders with your data. For production systems, you can further constrain the [OUTPUT_SCHEMA] to match your application's data model exactly. If the source documents are long, consider pre-processing them to extract only the most relevant passages to fit within the model's context window. The overall_consistency_score provides a quick, high-level signal that can be used for pass/fail gating in a CI/CD pipeline, while the detailed claims array is invaluable for debugging and error analysis.

The Factual Consistency Check prompt is designed to run as a post-generation evaluation step in a RAG pipeline or content verification workflow. It should be called after a primary model generates an answer, receiving both the generated text and the source documents as inputs. The prompt returns a structured JSON report classifying each claim as supported, unsupported, or contradicted, which your application can then use to decide whether to surface the answer, flag it for review, or trigger a regeneration loop. This prompt is not a real-time guardrail—it is an asynchronous evaluator that adds latency proportional to the length of the generated text and the number of source passages provided.

Wire the prompt into your application by placing it behind a validation layer that checks the output schema before downstream consumption. The expected output is a JSON object with a claims array, where each element contains claim_text, grounding_status (one of supported, unsupported, contradicted), source_passage_id, and rationale. Implement a schema validator using Pydantic, Zod, or JSON Schema that rejects responses missing required fields or containing invalid enum values. If validation fails, retry once with the same inputs and a stronger format instruction appended to the prompt. Log all validation failures and retry attempts for observability. For high-stakes domains like healthcare or legal review, route any output where grounding_status is contradicted or where more than 10% of claims are unsupported to a human review queue before the answer reaches the end user. Model choice matters: use a model with strong instruction-following and JSON output capabilities, such as GPT-4o, Claude 3.5 Sonnet, or equivalent. Avoid smaller models that may hallucinate the grounding assessment itself, defeating the purpose of the check.

For evaluation pipelines, batch this prompt across a golden dataset of known grounded and hallucinated answer pairs to measure precision and recall of hallucination detection. Store each run's output alongside the ground truth labels to compute metrics over time. If you are using this prompt in a CI/CD gate for prompt releases, set a minimum recall threshold for hallucination detection—typically 0.90 or higher—and block deployment if the threshold is not met. Avoid running this check on every single user query in production without caching or sampling; the token cost scales with both input length and output verbosity. Instead, sample a percentage of traffic for continuous monitoring and run full evaluations during pre-release testing. The next step after implementing this harness is to build a dashboard that tracks grounding scores over time, alerting on drift so you know when your generation model or retrieval pipeline needs attention.