Citation Consistency Check Prompt

This prompt is designed for teams building multi-source synthesis systems, knowledge bases, or research assistants where a single answer draws on multiple citations. Its primary job is to detect when citations to the same source make contradictory claims, or when claims supported by different sources conflict with each other. The ideal user is an AI product engineer, a RAG pipeline architect, or a compliance reviewer who needs to surface inconsistencies before an answer reaches a user or auditor. You should use this prompt when your system retrieves and cites multiple documents, and the cost of presenting conflicting information is high—such as in legal research, clinical summarization, financial analysis, or any domain where contradictory evidence erodes trust.

To use this prompt effectively, you must provide the full generated answer and the complete set of cited source passages, not just source identifiers or summaries. The prompt requires the raw evidence text so it can compare claims against their alleged support. It works best when citations are already linked to specific claims or spans; if your system does not yet produce claim-to-source mappings, run a Claim-to-Source Linking Prompt first. The output is a structured consistency report that flags each conflict, identifies the conflicting claims and sources, classifies the conflict type (internal contradiction within a source, cross-source contradiction, or unsupported claim inflation), and recommends a resolution action such as suppressing one claim, presenting both with a conflict note, or escalating for human review.

Do not use this prompt as a substitute for source retrieval quality. It checks consistency among the sources you provide, not whether you retrieved the right sources. If your retrieval step returns irrelevant or low-authority documents, this prompt will faithfully report that their claims are consistent—which is a false sense of security. Pair it with a Passage Relevance and Filtering prompt upstream and a Source Trustworthiness and Authority Assessment prompt to ensure you are checking consistency on a solid evidence foundation. For high-stakes domains, always route unresolved conflicts to human review rather than letting the model silently choose which claim to present.

The prompt below is designed to be copied directly into your application or evaluation harness. It accepts a set of claims, their associated citations, and the source documents those citations reference. The model is instructed to compare claims made under the same source for internal consistency, then compare claims across different sources for contradictions. All placeholders are marked with square brackets and must be replaced with real data before execution. The output is a structured JSON report that can be parsed by downstream validation logic.

textCopy
You are a citation consistency auditor. Your task is to examine a set of claims, their citations to source documents, and the source document text itself. You must detect two types of inconsistency:

1. **Intra-source inconsistency**: Multiple citations to the same source document make claims that contradict each other or cannot all be true given the source text.
2. **Cross-source contradiction**: Citations to different source documents make claims that directly conflict with each other.

## INPUT

**Claims and Citations:**
[CLAIMS_AND_CITATIONS]

**Source Documents:**
[SOURCE_DOCUMENTS]

## OUTPUT SCHEMA

Return a JSON object with the following structure:

```json
{
  "consistency_report": {
    "total_claims_reviewed": <integer>,
    "total_sources_reviewed": <integer>,
    "intra_source_inconsistencies": [
      {
        "source_id": "<string>",
        "conflicting_claim_ids": ["<string>"],
        "description": "<string explaining the contradiction>",
        "relevant_source_text": "<excerpt from source>",
        "severity": "critical|major|minor"
      }
    ],
    "cross_source_contradictions": [
      {
        "claim_a_id": "<string>",
        "source_a_id": "<string>",
        "claim_b_id": "<string>",
        "source_b_id": "<string>",
        "description": "<string explaining the contradiction>",
        "relevant_text_a": "<excerpt>",
        "relevant_text_b": "<excerpt>",
        "severity": "critical|major|minor",
        "resolution_recommendation": "<string>"
      }
    ],
    "consistent_claims": ["<claim_id>"],
    "overall_assessment": "<summary string>"
  }
}

CONSTRAINTS

• Only flag a conflict if the source text clearly supports the conflicting interpretation. Do not flag stylistic differences or minor paraphrasing as inconsistencies.
• For each inconsistency, include the exact source text that demonstrates the conflict.
• If a claim appears unsupported by its cited source, note it as an intra-source inconsistency with severity "critical".
• If two sources make conflicting factual claims, recommend which source appears more authoritative or recent, and explain why.
• If no inconsistencies are found, return empty arrays for both inconsistency types and list all claim IDs under consistent_claims.
• Do not fabricate conflicts. When in doubt, classify the claims as consistent.

RISK LEVEL

[RISK_LEVEL]

EXAMPLES

[EXAMPLES]

To adapt this prompt for your workflow, replace [CLAIMS_AND_CITATIONS] with a structured list mapping each claim ID to its text and cited source ID. Replace [SOURCE_DOCUMENTS] with the full text or relevant excerpts of each source document, keyed by source ID. The [RISK_LEVEL] placeholder should be set to high for regulated domains, which triggers stricter conflict detection and a requirement for human review before the report is acted upon. The [EXAMPLES] placeholder can be populated with one or two annotated few-shot examples showing a clear intra-source inconsistency and a clear cross-source contradiction, which significantly improves the model's precision on ambiguous cases.

After generating the consistency report, validate the output against the JSON schema before passing it downstream. If the report contains critical severity findings, route it for human review rather than auto-applying resolution recommendations. For production pipelines, log every consistency report alongside the input claims and sources so that reviewers can trace the model's reasoning. Avoid using this prompt on extremely long document sets without first chunking and pre-filtering relevant passages, as context-window limits will degrade detection quality.

The Citation Consistency Check prompt is designed to be called after a multi-source synthesis step, not as a standalone user-facing feature. In a typical RAG pipeline, the application first retrieves documents, generates an answer with citations, and then passes the generated text plus the full set of cited sources into this consistency check. The prompt expects a structured input containing the original answer, a list of claims extracted from that answer, and the complete text of each cited source. Wire this as a post-generation validation step before the answer reaches the user, with the consistency report determining whether the answer is released, flagged for review, or sent back for regeneration.

For production integration, wrap the prompt call in a function that validates the output schema before proceeding. The consistency report must contain a conflicts array where each conflict has a type field (one of intra_source_contradiction, inter_source_contradiction, or inconsistent_interpretation), a severity field (critical, major, minor), and a recommendation field. Implement a post-processing validator that rejects any response missing these fields or containing malformed conflict objects. If validation fails, retry once with a repair prompt that includes the validation error message. After two failures, escalate to a human review queue with the original answer, sources, and partial consistency report attached. Log every consistency check result—including the input claims, sources, conflict details, and final disposition—for audit and prompt improvement analysis.

Choose a model with strong reasoning capabilities for this task, as detecting subtle inconsistencies across multiple documents requires careful comparison. GPT-4o, Claude 3.5 Sonnet, or equivalent models perform well here; smaller or faster models often miss cross-document contradictions. Set the temperature low (0.0–0.2) to maximize deterministic output. If your pipeline processes long documents, truncate each source to the most relevant passages before passing them to the consistency check to stay within context windows. For high-stakes domains like legal or medical content, always route critical severity conflicts to human review regardless of the recommendation field, and never auto-resolve conflicts by picking one source over another without explicit human approval. The consistency report should inform a decision, not make it autonomously.

Build an eval harness that tests the consistency checker against known conflict cases. Create a golden dataset of answer-source pairs with labeled contradictions, including intra-source inconsistencies (same document says two different things), inter-source contradictions (two sources disagree on a fact), and false-positive cases where sources appear to conflict but don't. Measure precision and recall on conflict detection, and track the false-positive rate specifically—flagging non-conflicts as conflicts erodes trust in the pipeline faster than missing real ones. Run this eval suite on every prompt version change and model upgrade. If the false-positive rate exceeds 10%, add few-shot examples of non-conflicting source pairs to the prompt or adjust the severity threshold before releasing.