Disputed Evidence Ranking Prompt for Synthesis

This prompt is designed for evidence ranking pipelines that must handle conflicting sources before answer synthesis. The core job-to-be-done is taking a set of retrieved passages, a target claim or question, and optional source metadata, then producing a ranked list where passages contradicted by more reliable evidence are demoted and unresolved disagreements are explicitly flagged. The ideal user is an AI engineer or RAG pipeline builder who has already retrieved passages and performed basic conflict detection—this prompt focuses on ranking, not on discovering conflicts from scratch. You need it when your retrieval set contains sources that disagree on material facts, and you cannot safely synthesize an answer without first establishing which evidence carries more weight.

The prompt assumes you have already run a conflict detection step that identified contradictory passage pairs or clusters. It expects inputs such as a set of passages with unique IDs, a target claim or question, and optional metadata fields like source authority score, recency, or domain. The output should be a ranked list with each passage's position, a conflict-aware score, and flags for passages involved in unresolved disputes. Do not use this prompt when all passages are consistent or when you only need relevance scoring without conflict awareness—simpler ranking prompts handle those cases with lower latency and cost. Also avoid this prompt when the conflict detection step has not been performed, because the model will attempt to detect conflicts inline, which is unreliable and produces inconsistent rankings.

Before implementing, verify that your upstream conflict detection produces structured conflict pairs or clusters that this prompt can consume. If your retrieval set is large, consider pre-filtering to the top-N most relevant passages before running conflict-aware ranking to control token usage. The prompt works best when source metadata includes reliability signals—without them, the model may default to superficial cues like writing style or assertiveness. After ranking, always run a grounding verification step on the synthesized answer to confirm that demoted or flagged passages were handled appropriately. The next section provides the copy-ready template you can adapt to your evidence schema and ranking criteria.

This prompt template is designed to be dropped directly into your evidence ranking pipeline. It accepts a set of evidence passages, a target claim or question, and optional source reliability metadata, then produces a ranked list that accounts for disputes. The template forces the model to identify contradictions, demote sources that are contradicted by more reliable evidence, and flag disagreements that cannot be resolved with the available information. Copy the template below, replace the square-bracket placeholders with your data, and integrate it into your application's ranking step before answer synthesis.

textCopy
You are an evidence ranking specialist. Your job is to rank a set of evidence passages by how strongly they support a claim, while accounting for disputes between sources.

## INPUT

**Claim or Question:**
[CLAIM]

**Evidence Passages:**
[EVIDENCE_PASSAGES]

**Source Reliability Metadata (optional):**
[SOURCE_METADATA]

## RANKING RULES

1. Score each passage on relevance, specificity, and direct support for the claim.
2. When two or more passages contradict each other, compare their source reliability. Demote the less reliable source.
3. If reliability is equal or unknown, flag the disagreement as unresolved rather than picking a side.
4. Do not elevate a passage just because it is confident or well-written. Prefer passages with specific evidence over general statements.
5. Demote passages that are contradicted by multiple independent sources, even if those sources have lower individual authority scores.
6. If a passage is outdated and a more recent source contradicts it, demote the outdated passage regardless of its original authority.

## OUTPUT FORMAT

Return a JSON object with this exact structure:

{
  "ranked_passages": [
    {
      "passage_id": "string",
      "rank": number,
      "relevance_score": number (0-1),
      "support_strength": "strong_support" | "moderate_support" | "weak_support" | "neutral" | "weak_contradiction" | "moderate_contradiction" | "strong_contradiction",
      "disputes": [
        {
          "contradicted_by": "passage_id",
          "conflict_type": "direct_factual" | "methodological" | "temporal" | "scope_difference" | "interpretation",
          "resolution": "resolved_in_favor" | "resolved_against" | "unresolved",
          "explanation": "string"
        }
      ],
      "demotion_reason": "string | null",
      "explanation": "string"
    }
  ],
  "unresolved_disagreements": [
    {
      "claim": "string",
      "conflicting_passage_ids": ["string"],
      "severity": "critical" | "high" | "medium" | "low",
      "recommendation": "string"
    }
  ],
  "ranking_notes": "string"
}

## CONSTRAINTS

- Do not invent evidence. Only use the provided passages.
- If no passages are provided, return an empty ranked_passages array.
- If all passages are irrelevant, return them with low relevance scores and explain why.
- For unresolved disagreements, explain what additional information would be needed to resolve them.
- If source reliability metadata is not provided, treat all sources as having equal but unknown reliability.

Adaptation guidance: Replace [CLAIM] with the specific claim or question being evaluated. [EVIDENCE_PASSAGES] should contain your retrieved passages, each with a unique passage_id, the passage text, and optionally a source identifier. [SOURCE_METADATA] can include authority scores, recency timestamps, domain classifications, or any reliability signals your system already computes. If you don't have reliability metadata, remove that section and the model will treat all sources equally. The output schema is designed to feed directly into a downstream synthesis prompt that can use the ranked list and unresolved disagreements to produce a balanced, caveated answer.

Validation and testing: Before deploying this prompt, run it against a golden dataset of passages with known contradictions. Verify that the model correctly demotes contradicted sources, flags unresolved disagreements, and does not hallucinate disputes between passages that actually agree. Pay special attention to false positives where paraphrased agreement is misclassified as contradiction. If your use case is high-risk—such as medical, legal, or financial decision support—add a human review step for any output where unresolved_disagreements contains critical or high severity items. Log every ranking output with the input passages and source metadata for auditability.

What to do next: Wire this prompt into your evidence ranking step, before answer synthesis. Feed the ranked_passages array into your synthesis prompt, and use unresolved_disagreements to decide whether to present conflicting evidence to the user, express uncertainty, or refuse to answer. If you observe ranking instability—where the same passages produce different rankings across runs—consider lowering the temperature, adding few-shot examples of correct dispute handling, or moving to a more deterministic model for this step.

The Disputed Evidence Ranking Prompt is designed to sit inside an evidence processing pipeline, typically after retrieval and before answer synthesis. It consumes a set of evidence passages and a claim or question, then produces a ranked list with dispute annotations. The prompt is not a standalone chatbot interaction; it expects structured input and must return structured output that downstream components can consume. The primary integration points are the retrieval system (which supplies the passages), a conflict detection module (which may pre-label disputes), and the synthesis engine (which consumes the ranked evidence). In high-stakes domains such as legal review, clinical decision support, or financial analysis, this prompt should be wrapped in a harness that enforces output validation, logs every ranking decision, and escalates ambiguous cases for human review before the ranked evidence reaches the synthesis step.

Wire the prompt into your application as a stateless ranking service. The request payload should include the claim under evaluation, the retrieved passages with source metadata, and any pre-computed conflict flags or reliability scores. The response must conform to a strict JSON schema: an ordered array of evidence items, each with a rank position, a dispute status (disputed, uncontested, unresolved), a list of contradicting source IDs, and a brief ranking rationale. Implement a validation layer that checks for schema compliance, duplicate ranks, missing contradicting source references when dispute status is 'disputed', and rationales that fail to reference specific passage content. If validation fails, retry once with a repair prompt that includes the validation errors. If the retry also fails, log the failure and route the case to a human review queue. For model selection, use a model with strong instruction-following and structured output capabilities (such as GPT-4o or Claude 3.5 Sonnet) with a low temperature setting (0.1–0.2) to maximize ranking stability. Do not use this prompt with models that have weak JSON adherence or that tend to hallucinate source references.

The most common production failure mode is ranking instability when conflict severity is ambiguous. Two passages may partially disagree on methodology while agreeing on conclusions, or they may conflict on a sub-claim that is tangential to the main question. The prompt's ranking output can oscillate between treating these as disputed versus uncontested across similar inputs. To mitigate this, implement an eval harness that runs the prompt against a golden set of evidence clusters with known dispute labels and measures ranking consistency across multiple runs. If consistency drops below a threshold (e.g., 90% agreement on dispute classification for the same input), consider adding few-shot examples that demonstrate edge cases, or implement a secondary consensus check that compares the prompt's output against a simpler pairwise conflict classifier. For audit-sensitive deployments, log every ranking decision with the full prompt input, model response, validation result, and any human override. This audit trail is essential for compliance reviews and for debugging ranking drift after model updates. Never send raw user questions directly to this prompt without first verifying that the evidence passages have been retrieved from approved, versioned sources.