Multi-Passage Evidence Ranking Prompt Template

This prompt is for RAG pipeline engineers and search architects who need to order multiple retrieved passages by relevance, specificity, and support strength for a given query. Use it when your retrieval step returns a flat, unranked set of documents and you need a scored, ordered list before answer generation or citation selection. The prompt produces a ranked list with numerical scores and a short rationale for each passage, making the evidence selection step explicit and auditable. This is the right tool when you need to justify why certain passages were prioritized over others, either for downstream model consumption or for human review of the ranking logic.

Do not use this prompt when you have a single passage, when you need a binary relevant/not-relevant filter, or when your downstream task requires only the top-1 passage without explanation. This prompt assumes you have already retrieved candidate passages and need to impose order and justification on them. If your retrieval system already returns a scored list from a vector database or search engine, consider whether you need re-ranking with a different set of criteria—this prompt is for the case where retrieval scores are absent, untrustworthy, or insufficiently aligned with your specific evidence quality dimensions. For strict budget-constrained selection, use the Top-K Evidence Selection Prompt Template instead; for pairwise calibration, use the Pairwise Evidence Comparison Prompt.

Before wiring this into production, verify that your retrieval step is returning a diverse enough candidate set that ranking adds value. If your retrieval consistently returns only one or two relevant passages, ranking overhead is wasted latency and cost. Also consider whether your downstream answer generation model exhibits position bias—if it overweights early passages, your ranking order becomes load-bearing and requires additional evaluation. For high-stakes domains where ranking errors could produce misleading answers, pair this prompt with the Evidence Ranking Harness with Eval Checks Prompt to gate rankings before they reach users.

This template is the core instruction set for a multi-passage evidence ranker. It expects a user query and a set of retrieved passages, then produces a ranked list with scores and rationale. The square-bracket placeholders are designed to be replaced by your application code at runtime, not by an end user. Before integrating this into production, you must define your output schema, scoring dimensions, and failure-handling logic.

textCopy
You are an evidence ranking specialist. Your task is to evaluate and rank multiple text passages based on their relevance, specificity, and support strength for a given query.

## INPUT
[QUERY]

[PASSAGES]

## RANKING CRITERIA
Rank each passage on a scale of 1-10 for each dimension below, then compute a weighted total score:
- **Relevance (weight: 0.4):** How directly the passage addresses the core question or topic in the query.
- **Specificity (weight: 0.3):** The level of concrete detail, facts, or data the passage provides. Vague or general statements score lower.
- **Support Strength (weight: 0.3):** How strongly the passage's content can be used to answer or substantiate a claim related to the query.

## CONSTRAINTS
[CONSTRAINTS]

## OUTPUT FORMAT
Return a valid JSON object with the following structure. Do not include any text outside the JSON object.
{
  "ranked_passages": [
    {
      "passage_id": "string",
      "relevance_score": number,
      "specificity_score": number,
      "support_strength_score": number,
      "weighted_total": number,
      "rank": number,
      "rationale": "A one-sentence explanation of the ranking decision."
    }
  ],
  "unranked_passages": ["passage_id"],
  "ranking_notes": "A brief summary of any global observations, such as overall evidence quality or conflicts."
}

## INSTRUCTIONS
1. Analyze the [QUERY] to understand the core information need.
2. For each passage in [PASSAGES], assign scores for relevance, specificity, and support strength.
3. Calculate the weighted total for each passage.
4. Sort passages by weighted total in descending order and assign a rank starting from 1.
5. If a passage is completely irrelevant or unparseable, place its ID in the `unranked_passages` array.
6. Adhere strictly to the [CONSTRAINTS] provided.
7. Output only the JSON object.

To adapt this template, replace [QUERY] with the user's original question, and [PASSAGES] with a pre-formatted string of your retrieved documents, each clearly delimited with an ID (e.g., [ID: doc_1] Content...). The [CONSTRAINTS] placeholder is critical for production control; inject rules like "Exclude passages older than 2023," "Prioritize peer-reviewed sources," or "Maximum of 5 ranked passages." If your application requires a different output schema, replace the JSON structure in the prompt, but ensure your downstream parser and validation logic are updated to match. For high-stakes domains, always route the final ranked list for human review before it is used to generate a user-facing answer.

The multi-passage evidence ranking prompt operates as a critical middleware step between retrieval and answer generation. In a typical RAG pipeline, you retrieve N passages (often 20–50), but your downstream answer generator can only consume K passages (often 3–10) due to context window or latency constraints. This prompt transforms an unsorted retrieval set into a ranked, scored, and justified list that downstream components can trust. The implementation harness must handle three concerns: input assembly (gathering passages, query, and optional metadata), output validation (ensuring the model returns a parseable ranking with required fields), and integration gating (deciding whether the ranking quality is sufficient to proceed or whether re-retrieval or human review is needed).

Wire the prompt into your application as a dedicated ranking service with a strict contract. On the input side, assemble a payload containing the user query, the list of retrieved passages (each with a unique passage ID, full text, and optional source metadata like publication date or authority tier), and any ranking constraints such as top-K count, diversity requirements, or recency weighting. On the output side, enforce a JSON schema that requires each ranked passage to include the passage ID, a relevance score (normalized 0.0–1.0), a strength tier (high/medium/low/insufficient), and a one-sentence selection rationale. Implement a validation layer that checks: (a) all output passage IDs exist in the input set, (b) scores are within range, (c) the ranking is non-empty, and (d) rationales are present and non-generic. If validation fails, retry once with the error message injected into the prompt as a correction hint. If the second attempt also fails, log the failure and fall back to a simpler ranking heuristic such as cosine similarity to the query embedding. For high-stakes domains like legal or medical evidence, add a human review step when confidence scores fall below a configurable threshold or when contradiction detection flags are raised.

Model choice matters for ranking consistency. Use a model with strong instruction-following and structured output capabilities—GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro are good starting points. Avoid smaller or older models that exhibit position bias (favoring the first or last passages) or length bias (favoring longer passages regardless of relevance). To mitigate position bias in the harness, randomize passage order in the input payload before sending it to the model. For latency-sensitive applications, consider batching multiple ranking requests or using a faster model for initial filtering followed by a more capable model for final ranking of the top candidates. Log every ranking request with the input passages, output ranking, validation results, and any retry attempts. This trace data is essential for debugging ranking failures, calibrating scoring thresholds, and building evaluation datasets for future prompt improvements. Before shipping, run the ranking harness against a golden dataset of query-passage pairs with known relevance judgments and measure ranking correlation metrics like NDCG or Kendall's tau to establish a quality baseline.