Multi-Document Factual Consistency Prompt

This prompt is designed for AI engineers and evaluation pipeline builders who need to verify that a generated answer is factually consistent across multiple retrieved source documents. The core job-to-be-done is detecting when a RAG system synthesizes an answer that is supported by one source but directly contradicted by another, or when the answer ignores a conflict that exists in the provided context. The ideal user is someone operating a production RAG system who already has a retrieval step in place and needs an automated judge to catch cross-document inconsistencies before the answer reaches the end user. Required context includes the full generated answer and the complete set of retrieved documents that were provided to the generator, with each document clearly delimited and identified by a source label.

Use this prompt when your retrieval pipeline pulls from multiple documents that may disagree with each other—for example, financial filings from different periods, medical literature with conflicting study results, or policy documents with version discrepancies. The prompt is specifically designed to catch three failure patterns: claims supported by one document but contradicted by another, claims that synthesize information from multiple documents in a way that creates a false implication, and answers that present a consensus where none exists in the source material. Do not use this prompt for single-document grounding checks, for detecting hallucinations where no source conflict exists, or for evaluating the quality of the retrieval step itself. This prompt assumes the retrieved documents are already provided and focuses exclusively on cross-document consistency analysis.

Before implementing this prompt, ensure your pipeline can reliably separate and label each source document with a persistent identifier that the judge can reference in its output. The prompt works best when documents are chunked at a granularity that preserves document boundaries—mixing sentences from different documents into the same chunk will defeat cross-document conflict detection. For high-stakes domains such as healthcare or legal applications, always route detected conflicts to a human reviewer rather than attempting automated resolution. The next section provides the copy-ready prompt template with placeholders for your specific document format and output schema requirements.

This template is designed to be dropped into a RAG evaluation pipeline. It instructs the model to act as a factual consistency judge, comparing a generated answer against a set of retrieved documents. The core job is to identify claims that are supported, unsupported, or contradicted by the evidence, with special attention to conflicts between sources. Use this template when you have multiple documents that may disagree and you need a structured consistency report before the answer reaches a user or a downstream system.

textCopy
You are a factual consistency auditor. Your task is to evaluate a generated answer against a set of source documents. You must identify every factual claim in the answer and determine its grounding status relative to the provided sources.

## INPUT
**Generated Answer:**
[ANSWER]

**Source Documents:**
[DOCUMENTS]

## INSTRUCTIONS
1. Extract every discrete factual claim from the Generated Answer.
2. For each claim, search all Source Documents for supporting or contradicting evidence.
3. Classify each claim as one of:
   - **SUPPORTED**: At least one source clearly supports the claim.
   - **UNSUPPORTED**: No source provides evidence for the claim.
   - **CONTRADICTED**: One or more sources directly contradict the claim.
   - **SOURCE_CONFLICT**: Multiple sources provide conflicting information on this claim.
4. For each claim, cite the specific source document(s) and relevant passage(s) that inform your classification.
5. If a claim is CONTRADICTED or SOURCE_CONFLICT, explain the nature of the contradiction.
6. Do not judge the truth of the sources themselves. Only judge consistency between the answer and the provided documents.

## OUTPUT FORMAT
Return a valid JSON object with this exact schema:
{
  "overall_consistency_score": number (0.0 to 1.0, where 1.0 means all claims are fully supported with no contradictions),
  "total_claims": number,
  "supported_claims": number,
  "unsupported_claims": number,
  "contradicted_claims": number,
  "source_conflict_claims": number,
  "claims": [
    {
      "claim_text": "string (the exact claim from the answer)",
      "status": "SUPPORTED | UNSUPPORTED | CONTRADICTED | SOURCE_CONFLICT",
      "source_references": [
        {
          "document_id": "string",
          "passage": "string (the relevant excerpt)",
          "relationship": "SUPPORTS | CONTRADICTS"
        }
      ],
      "explanation": "string (brief reasoning, required for CONTRADICTED and SOURCE_CONFLICT)"
    }
  ],
  "source_conflicts_summary": [
    {
      "claim_text": "string",
      "conflict_description": "string (explain what the sources disagree on)",
      "conflicting_documents": ["document_id_1", "document_id_2"],
      "recommendation": "string (suggest whether human review is needed)"
    }
  ]
}

## CONSTRAINTS
- Only use information present in the provided Source Documents.
- If the answer contains no factual claims, return an empty claims array with total_claims set to 0.
- Do not fabricate source references. If no source supports or contradicts a claim, mark it as UNSUPPORTED.
- For SOURCE_CONFLICT claims, always populate the source_conflicts_summary array.

To adapt this template, replace [ANSWER] with the generated text under evaluation and [DOCUMENTS] with your retrieved context, formatted with clear document identifiers. For production use, pre-process your documents to include stable IDs that persist across retrieval calls, enabling traceable source references in the output. If your RAG system already includes citation markers in the answer, add an optional [CITATIONS] input field and modify the instructions to cross-check those citations against the source documents. For high-stakes domains like healthcare or legal review, add a [RISK_LEVEL] parameter that adjusts the strictness of the contradiction detection and always route SOURCE_CONFLICT claims to a human review queue before any downstream action.

This prompt is designed to operate as a post-generation evaluation step in a RAG pipeline, not as a real-time user-facing response. After your primary LLM generates an answer from multiple retrieved documents, this prompt acts as a judge that compares the generated answer against all source documents simultaneously. The implementation harness must feed the prompt three structured inputs: the generated answer text, an array of source documents with unique identifiers, and a defined output schema for the consistency report. The prompt returns a structured JSON object containing per-claim analysis, source-level support/contradiction flags, and an overall consistency score. This output should never be shown directly to end users—it feeds your monitoring dashboards, alerting systems, and human review queues.

Wire this prompt into your application as an asynchronous evaluation worker. After the primary RAG response is generated, enqueue an evaluation job that calls this prompt with the answer and all retrieved documents. Use a model with strong instruction-following and JSON output capabilities—GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro are suitable defaults. Implement a validation layer that parses the JSON output and checks for required fields: claim_id, claim_text, source_id, support_status (must be one of supported, contradicted, unsupported, ambiguous), and explanation. If validation fails, retry once with the validation errors appended to the prompt as feedback. If the second attempt also fails, route the evaluation to a human review queue with the raw output and validation failure details attached. Log every evaluation result—including retries and failures—to your observability platform with the prompt version, model, latency, and consistency score for trend analysis.

For high-stakes domains like healthcare, legal, or finance, add a human-in-the-loop gate before any action is taken based on the consistency report. When the prompt detects source conflicts or flags claims as contradicted, automatically escalate the entire answer-document set to a review interface that shows the conflicting passages side by side. Do not suppress or auto-resolve contradictions—the prompt's job is detection, not adjudication. For production monitoring, track the rate of contradicted and unsupported claims over time as a key groundedness metric. Set thresholds that trigger alerts when hallucination rates exceed acceptable bounds. Finally, version this prompt alongside your primary RAG prompt in the same repository, and run regression tests against a golden dataset of known answer-document pairs with human-annotated consistency labels before any prompt change reaches production.