Source Conflict Detection Prompt

This prompt is designed for multi-source AI systems, particularly RAG pipelines, that retrieve information from several documents before generating an answer. Its job is to detect when those retrieved documents contain contradictory information on the same factual question. It categorizes the type of conflict, identifies which sources disagree, and flags claims that require human adjudication before they can be used in a downstream answer or decision. Use this prompt when your system cannot afford to silently pick one source over another, such as in financial analysis, medical research, legal document review, or any evidence-backed workflow where conflicting data must be surfaced rather than averaged away.

The ideal user is an AI engineer or product developer building a RAG system where multiple documents are retrieved per query and source trustworthiness varies. The prompt requires a specific input shape: a user question or topic, plus a set of retrieved documents with source identifiers. It is not a general-purpose fact-checker for a single document, nor is it a replacement for a structured knowledge base with authoritative conflict resolution rules. It works best when the retrieved documents are topically coherent but may originate from different authors, time periods, or data sources. You should avoid using this prompt when you have a single authoritative source, when conflicts are expected and acceptable (e.g., opinion aggregation), or when the cost of running an extra LLM call for conflict detection outweighs the risk of presenting contradictory information to the user.

Before wiring this prompt into production, define what happens after a conflict is detected. The output is a structured conflict report, not a resolved answer. Your application must decide whether to halt generation and escalate for human review, present the conflict to the user with source attribution, or apply a resolution policy based on source recency or authority. Do not use this prompt as a silent filter that picks a winner without traceability. In regulated or high-risk domains, always route detected conflicts to a human review queue and log the full conflict report for audit purposes. If your system generates an answer despite a detected conflict, include a visible warning and cite the disagreement explicitly.

This prompt template is designed to be dropped directly into your evaluation harness or LLM orchestration layer. It expects a specific factual question and a set of retrieved documents, then produces a structured conflict report. The template uses square-bracket placeholders for all dynamic inputs—replace these with your data at runtime before sending the request to the model. The output schema is strict JSON to ensure downstream parsers can consume the conflict report without additional normalization.

textCopy
You are a source conflict detection system for a multi-document RAG pipeline. Your job is to identify when retrieved documents contain contradictory information about the same factual question.

## INPUT

**Factual Question:** [QUESTION]

**Retrieved Documents:**
[DOCUMENTS]

## TASK

1. Extract all factual claims from each document that are relevant to answering the question.
2. Identify pairs of claims that directly contradict each other.
3. For each contradiction, determine the conflict type and which claim requires human adjudication.

## CONFLICT TYPES

- **FACTUAL_DISAGREEMENT**: Two documents assert mutually exclusive facts (e.g., "released in 2023" vs. "released in 2024").
- **SCOPE_CONFLICT**: One document makes a broader claim than another supports (e.g., "all regions" vs. "North America only").
- **TEMPORAL_STALENESS**: A newer document contradicts an older one due to a change over time.
- **SOURCE_QUALITY_DISPUTE**: Documents disagree and one has clearly higher authority or recency.
- **INTERPRETATION_DIVERGENCE**: Documents describe the same event differently without clear factual error.

## CONSTRAINTS

- Only flag contradictions where both claims cannot be simultaneously true.
- Do not flag differences in phrasing or emphasis as contradictions.
- If documents address different aspects of the question without conflict, report NO_CONFLICT.
- Assign a severity of HIGH, MEDIUM, or LOW based on the impact on answering the question.
- For each conflict, identify which claim(s) require human review before use.

## OUTPUT SCHEMA

Return valid JSON only:
{
  "question": "string",
  "documents_analyzed": number,
  "conflict_status": "CONFLICT_FOUND" | "NO_CONFLICT" | "INSUFFICIENT_EVIDENCE",
  "conflicts": [
    {
      "conflict_id": "string",
      "conflict_type": "FACTUAL_DISAGREEMENT" | "SCOPE_CONFLICT" | "TEMPORAL_STALENESS" | "SOURCE_QUALITY_DISPUTE" | "INTERPRETATION_DIVERGENCE",
      "severity": "HIGH" | "MEDIUM" | "LOW",
      "claim_a": {
        "text": "string",
        "source_document_index": number,
        "source_excerpt": "string"
      },
      "claim_b": {
        "text": "string",
        "source_document_index": number,
        "source_excerpt": "string"
      },
      "conflict_explanation": "string",
      "requires_human_adjudication": boolean,
      "adjudication_question": "string | null"
    }
  ],
  "uncontested_claims": [
    {
      "text": "string",
      "source_document_index": number,
      "agreement_level": "UNANIMOUS" | "MAJORITY" | "SINGLE_SOURCE"
    }
  ]
}

Adaptation guidance: Replace [QUESTION] with the user's factual query and [DOCUMENTS] with your retrieved passages, each prefixed with an index number (e.g., [0], [1]). If your retrieval system already provides source metadata such as publication dates or authority scores, include those inline to improve SOURCE_QUALITY_DISPUTE and TEMPORAL_STALENESS detection. For high-stakes domains such as healthcare or legal review, set requires_human_adjudication to true for all HIGH-severity conflicts and route them to a review queue before any downstream answer generation.

Validation and testing: Before deploying this prompt, run it against a golden dataset of known conflicting and non-conflicting document pairs. Measure precision (did it flag only real conflicts?) and recall (did it catch all planted contradictions?). Common failure modes include over-flagging stylistic differences as INTERPRETATION_DIVERGENCE and missing TEMPORAL_STALENESS when document dates are not provided. Add a post-processing validator that rejects any output where conflict_status is CONFLICT_FOUND but the conflicts array is empty, as this indicates a schema compliance failure. For production pipelines, log every conflict report alongside the retrieved documents to enable retrospective adjudication and prompt improvement.

The Source Conflict Detection Prompt is designed to sit between retrieval and answer generation in a multi-source RAG pipeline. After retrieving documents but before synthesizing a final answer, pass the user's factual question and all retrieved passages through this prompt. The model returns a structured conflict report that downstream logic can use to decide whether to proceed with generation, request more sources, or escalate to a human reviewer. This is not a prompt you fire once and ignore—it is a gating step that prevents contradictory information from silently polluting your AI's output.

Wire the prompt into your application as a pre-generation validation stage. After retrieval, call the LLM with the prompt template, passing [USER_QUESTION] and [RETRIEVED_DOCUMENTS] as structured inputs. Parse the JSON output and check the conflict_detected boolean. If false, proceed to answer generation with the original sources. If true, evaluate the conflict_severity field: low conflicts can trigger automatic source re-ranking or additional retrieval passes; medium conflicts should append a caveat to the generated answer noting the disagreement; high or critical conflicts must halt generation and route to a human review queue with the full conflict report attached. Implement a retry loop with a maximum of two additional retrieval attempts for low conflicts before escalating. Log every conflict detection event—including the raw prompt, model response, parsed conflict report, and routing decision—for auditability and prompt debugging.

For model selection, use a model with strong reasoning and structured output capabilities. GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro all perform well on this classification-plus-explanation task. Avoid smaller or faster models that may miss subtle contradictions or produce malformed JSON. Set temperature=0 or very low to maximize consistency across runs. If your application has strict latency requirements, consider running conflict detection in parallel with answer generation and aborting the answer path if a conflict is found, rather than making detection a serial blocker. For high-stakes domains like healthcare, legal, or finance, always require human sign-off on any detected conflict before the information reaches an end user—no exceptions.

Validation must happen at two levels. First, validate the JSON structure: confirm all required fields are present, conflict_detected is a boolean, conflict_severity is one of the allowed enum values, and conflicting_claims is an array with the expected claim structure. Second, validate the semantic content: spot-check that the cited source passages actually exist in your retrieved documents and that the quoted text matches. A common failure mode is the model hallucinating source references—always cross-reference source_id values against your actual retrieval results. If validation fails, retry once with an explicit error message appended to the prompt. If it fails again, escalate to human review with both failed attempts logged.

Build an eval harness before shipping. Create a golden dataset of 50–100 known-conflict and no-conflict cases with human-annotated ground truth. Measure precision (did detected conflicts actually exist?), recall (did the prompt catch all real conflicts?), and severity agreement (did the model's severity rating match the human rating within one level?). Run this eval suite on every prompt version change and every model upgrade. Track false positives carefully—over-flagging non-conflicts as conflicts will create unnecessary human review burden and erode trust in the pipeline. A false negative (missing a real conflict) is worse than a false positive, so tune your severity thresholds conservatively.