Evidence Ranking with Contradiction Detection Prompt

This prompt is designed for research assistants, decision-support systems, and RAG pipelines that must handle conflicting information. Instead of returning a flat list of relevant passages, it ranks evidence into tiers and explicitly annotates contradictions between sources. Use this when your downstream synthesis step needs to know not just what the evidence says, but where sources disagree. This prompt assumes you have already retrieved a set of candidate passages and need to order them for a specific claim or query. It is not a retrieval step and does not perform web search or database lookup.

The ideal user is a developer or AI engineer building a system where conflicting evidence is a first-class concern—think legal research tools, medical literature review assistants, financial due diligence platforms, or any RAG application where presenting a unified answer without surfacing disagreement would be misleading. The prompt expects structured inputs: a clear [CLAIM_OR_QUERY] to evaluate against, a [RETRIEVED_PASSAGES] array with unique IDs and full text, and optional [SOURCE_METADATA] like publication dates or authority scores. The output is a tiered ranking (e.g., 'strongly supporting', 'weakly supporting', 'neutral', 'contradicting') with explicit conflict annotations linking disagreeing passage pairs and a brief rationale for each conflict.

Do not use this prompt when you only need relevance sorting without contradiction awareness—the simpler Multi-Passage Evidence Ranking Prompt Template or Top-K Evidence Selection Prompt Template will be faster and cheaper. Avoid it when your retrieval set is already homogeneous or from a single authoritative source, as contradiction detection adds latency and token cost without benefit. This prompt is also inappropriate for real-time streaming applications where the full tiered output cannot be rendered incrementally. For high-stakes domains like healthcare or legal, always pair this prompt with human review of the conflict annotations before they reach end users, and log the ranked output alongside the final synthesized answer for auditability.

Before deploying, test the prompt against the Evidence Ranking Failure Mode Catalog Prompt to catch position bias, length bias, or authority overfitting in your contradiction annotations. If your use case requires calibrated confidence scores alongside conflict detection, consider chaining this prompt with the Evidence Confidence Score Assignment Prompt. For production, implement a validation harness that checks the output schema, verifies that every conflict annotation references valid passage IDs, and triggers a retry or fallback to a simpler ranking prompt if the contradiction detection produces malformed or empty conflict arrays.

This template is the core instruction set for an evidence ranking and contradiction detection workflow. It is designed to be dropped into a system prompt or a single-turn user message where the model receives a claim and a set of retrieved passages. The prompt forces the model to organize evidence into explicit strength tiers—Strong Support, Weak Support, Neutral, Weak Contradiction, and Strong Contradiction—rather than producing an undifferentiated list. Each passage assignment must include a specific rationale tied to passage content, and the model is explicitly instructed to identify and annotate pairs of passages that conflict with each other or with the broader consensus. This structured output makes downstream synthesis and citation decisions auditable.

textCopy
You are an evidence analyst. Your task is to rank the provided evidence passages by how strongly they support or contradict a given claim, and to flag any contradictions between passages.

CLAIM:
[CLAIM]

EVIDENCE PASSAGES:
[EVIDENCE_PASSAGES]

INSTRUCTIONS:
1. Assign each passage to one of five strength tiers:
   - STRONG_SUPPORT: The passage directly and specifically supports the claim with concrete details.
   - WEAK_SUPPORT: The passage is generally consistent with the claim but lacks specificity or directness.
   - NEUTRAL: The passage is relevant to the topic but neither supports nor contradicts the claim.
   - WEAK_CONTRADICTION: The passage suggests inconsistency with the claim but is indirect or qualified.
   - STRONG_CONTRADICTION: The passage directly and specifically contradicts the claim with concrete details.

2. For each passage, provide:
   - passage_id: The identifier from the input.
   - tier: One of the five tiers above.
   - rationale: A specific explanation referencing the passage content that justifies the tier assignment. Quote key phrases where helpful.

3. After ranking all passages, identify any contradictions:
   - List pairs of passages that conflict with each other or with the consensus of the ranked set.
   - For each conflict, specify the passage_ids involved and describe the nature of the disagreement.

4. If no passages qualify for a tier, state that explicitly rather than forcing an assignment.

OUTPUT FORMAT:
Return a JSON object with the following structure:
{
  "ranked_evidence": [
    {
      "passage_id": "string",
      "tier": "STRONG_SUPPORT | WEAK_SUPPORT | NEUTRAL | WEAK_CONTRADICTION | STRONG_CONTRADICTION",
      "rationale": "string"
    }
  ],
  "contradictions": [
    {
      "passage_ids": ["string", "string"],
      "conflict_description": "string"
    }
  ],
  "empty_tiers": ["string"]
}

CONSTRAINTS:
- Do not invent evidence not present in the passages.
- If the evidence is insufficient to assess the claim, note this in the output rather than guessing.
- When passages conflict, do not resolve the conflict—only flag it.

To adapt this template, replace [CLAIM] with the assertion or question the evidence must be evaluated against, and [EVIDENCE_PASSAGES] with a structured list of retrieved passages, each carrying a unique passage_id and the passage text. If your retrieval system includes source metadata such as publication date or authority score, inject that into each passage entry so the model can use it in its rationale. The output schema is intentionally strict: downstream code should validate that every input passage_id appears exactly once in the ranked_evidence array and that no fabricated passage_ids appear. For high-stakes domains such as legal or clinical review, route outputs where contradictions are detected to a human review queue before any answer synthesis occurs. Run periodic eval checks comparing tier assignments against a golden dataset to detect drift in ranking behavior.

This prompt is designed to sit between retrieval and answer generation in a RAG pipeline. It accepts a set of retrieved passages and a query or claim, then returns a ranked list of evidence tiers with explicit contradiction annotations. The primary integration point is after your retrieval step (vector search, hybrid search, or keyword retrieval) and before your synthesis or answer-generation step. The output schema is structured so downstream components can route high-confidence, non-contradictory evidence directly to generation, while flagging contradictory or low-confidence clusters for separate handling—such as a conflict-resolution prompt, a human review queue, or a 'multiple perspectives' display to the user.

Integration pattern: Call this prompt with a batch of retrieved passages (typically 10–30) and the user's query or the claim being evaluated. Parse the JSON output into a typed object in your application code. Use the tier field to gate downstream behavior: STRONG_CONSENSUS passages can feed directly into answer generation with high confidence; MODERATE_SUPPORT passages should be included but with uncertainty language; WEAK_OR_ISOLATED passages should be excluded or presented with explicit caveats; and any passage flagged with contradiction: true and a conflict_cluster_id should be routed to a conflict-resolution workflow. Validation: Before passing ranked evidence to answer generation, validate that the output contains the expected fields (passage_id, tier, relevance_score, contradiction, rationale), that scores are within the defined range, and that conflict_cluster_id values are consistent (passages in the same cluster should reference each other). If validation fails, retry once with the validation errors appended to the prompt as feedback. If retry also fails, log the failure and escalate to a human review queue rather than proceeding with unvalidated rankings.

Model selection: This task requires strong reasoning and comparison capabilities. Use a capable model such as Claude 3.5 Sonnet, GPT-4o, or Gemini 1.5 Pro. Avoid smaller or faster models for this step unless you have calibrated their ranking accuracy against your eval set—ranking with contradiction detection is a higher-cognitive-load task than simple relevance scoring. Latency and cost: This prompt processes multiple passages in a single call, which is more efficient than pairwise comparisons but still consumes significant tokens. Budget approximately 500–1500 output tokens per batch of 20 passages. For high-throughput systems, consider batching passages into groups of 15–20 and running ranking in parallel across shards, then merging results with a secondary deduplication step. Logging and observability: Log the full prompt input, output, validation results, and any retry attempts. Include conflict_cluster_id values and tier distributions in your monitoring dashboards. A sudden increase in contradiction clusters may indicate a retrieval quality problem, a data corpus issue, or an emerging real-world controversy that your system should surface rather than suppress. Human review gates: For regulated or high-risk domains (legal, medical, financial), always route outputs with tier: WEAK_OR_ISOLATED or any contradiction cluster to a human reviewer before the evidence reaches end users. The prompt's contradiction annotations make this triage straightforward: if contradiction: true appears in any passage ranked above your quality threshold, pause the pipeline and request review.

What to avoid: Do not use this prompt's output to automatically resolve contradictions by picking a 'winner.' The prompt identifies conflicts; it does not adjudicate them. Resolution logic—whether it's a second LLM call, a human decision, or a 'both sides' display—belongs in a separate component. Do not skip validation and proceed directly to answer generation with unvalidated rankings; malformed tier assignments or inconsistent conflict clusters will produce misleading or overconfident answers downstream. Finally, do not treat this prompt as a replacement for retrieval quality monitoring. If your retrieval step consistently returns weak or irrelevant passages, ranking them accurately won't fix the underlying retrieval gap—invest in your retrieval pipeline first, then use this prompt to make the best use of what you retrieve.