Evidence Ranking and Selection Prompt

This prompt is for RAG system builders who retrieve more candidate passages than can fit in a single context window. Its job is to act as a relevance filter and ranking engine: given a user query and a set of retrieved documents, it selects the most pertinent passages, ranks them by evidentiary value, and provides a concise rationale for each selection. The ideal user is an AI engineer or technical decision maker integrating this into a production retrieval pipeline where context budgets are tight and hallucination risks must be minimized through explicit evidence selection.

Use this prompt when your retrieval step returns a noisy candidate set—for example, a vector search that pulls back 20 chunks but your generation step can only accommodate 5. It is appropriate for fact-checking, question answering, and any workflow where downstream accuracy depends on the quality of the evidence passed to a synthesis model. The prompt forces the model to justify its selections, which creates an audit trail and helps catch ranking errors before they propagate. It is not a replacement for a retrieval system; it assumes you already have a candidate set. Do not use it when you have only one or two passages, when latency constraints prohibit an intermediate ranking step, or when the evidence is so domain-specialized that a general-purpose model cannot reliably assess relevance without fine-tuning.

Before deploying, calibrate the prompt against two known failure modes: position bias, where the model over-ranks passages at the beginning or end of the input list, and recency bias, where it favors passages with recent-sounding language even when older sources are more authoritative. Your evaluation harness should shuffle passage order across test runs and include deliberately stale but relevant passages to measure these biases. If the ranking task is high-stakes—such as legal, medical, or financial verification—always route the final ranked set to a human reviewer and log the model's selection rationale alongside the original candidate set for auditability.

This template is designed for RAG systems that retrieve more candidate passages than can fit in the model's context window. Instead of silently truncating results or using a simple vector-similarity cutoff, this prompt instructs the model to explicitly rank passages by relevance and select the top subset. The output includes a rationale for each selection, which is critical for debugging retrieval quality and preventing position bias from influencing the final answer generation step.

textCopy
You are an evidence ranking specialist. Your task is to evaluate a set of retrieved passages against a user query and select the most relevant passages for answering that query.

## INPUT

**User Query:** [QUERY]

**Retrieved Passages:**
[PASSAGES]

## CONSTRAINTS

- Select exactly [TOP_K] passages.
- Do not select passages that are irrelevant, redundant, or contain only background information.
- Prefer passages that directly address the query with specific facts, data, or authoritative statements.
- If multiple passages contain the same information, select the most complete and well-sourced version.
- If fewer than [TOP_K] passages are relevant, select only the relevant ones and leave the remaining slots empty.
- Do not modify, summarize, or paraphrase the passage text in your output.

## OUTPUT FORMAT

Return a JSON object with the following structure:

{
  "selected_passages": [
    {
      "passage_id": "string (the identifier from the input)",
      "relevance_score": number (0.0 to 1.0),
      "rationale": "string (one sentence explaining why this passage is relevant to the query)",
      "passage_text": "string (the full original passage text)"
    }
  ],
  "discarded_passages": [
    {
      "passage_id": "string",
      "discard_reason": "string (one of: 'irrelevant', 'redundant', 'insufficient_detail', 'outdated', 'low_authority')"
    }
  ],
  "selection_summary": "string (brief explanation of the overall selection strategy and any notable trade-offs)"
}

## EXAMPLE

**Query:** What is the capital of France?

**Passages:**
[1] Paris is the capital and most populous city of France.
[2] France is a country in Western Europe.
[3] The Eiffel Tower is located in Paris.
[4] Lyon is a major city in France known for its cuisine.

**Output:**
{
  "selected_passages": [
    {
      "passage_id": "1",
      "relevance_score": 1.0,
      "rationale": "Directly states that Paris is the capital of France, exactly answering the query.",
      "passage_text": "Paris is the capital and most populous city of France."
    }
  ],
  "discarded_passages": [
    {"passage_id": "2", "discard_reason": "irrelevant"},
    {"passage_id": "3", "discard_reason": "irrelevant"},
    {"passage_id": "4", "discard_reason": "irrelevant"}
  ],
  "selection_summary": "Only passage 1 directly answers the query. The remaining passages provide context about France or Paris but do not address the specific question about the capital."
}

To adapt this template, replace [QUERY] with the user's original question, [PASSAGES] with your formatted retrieval results (each passage must include a unique identifier), and [TOP_K] with the number of passages your downstream context window can accommodate. The passage format you provide should include both an ID and the full text. If your retrieval system returns metadata like source URLs or publication dates, include those in the passage objects so the model can use them for authority and recency judgments. For production use, add a [RISK_LEVEL] field that triggers additional validation when ranking is used for high-stakes decisions, and consider adding [DOMAIN] to adjust relevance criteria for specialized fields like medicine or law.

The Evidence Ranking and Selection Prompt is designed to sit between your retrieval step and your answer generation step in a RAG pipeline. It receives a set of candidate passages and a user query, then returns a relevance-ranked list with explicit selection rationale. This is not a standalone prompt—it is a middleware component that must be integrated into a broader workflow. The primary integration points are: (1) a retrieval system that produces more passages than your context window can hold, (2) a ranking harness that calls this prompt and parses its structured output, and (3) a downstream generator that consumes only the top-N ranked passages. The harness must handle validation of the ranked output, retry logic for malformed responses, and mitigation of known ranking biases such as position bias (favoring first-retrieved passages) and recency bias (favoring passages with recent dates regardless of relevance).

To wire this into an application, build a ranking service that accepts a query string and a candidates array of passage objects (each with id, text, source, and optional date fields). The service should randomize the input order of candidates before sending them to the model to reduce position bias—do not pass passages in retrieval-score order. Call the prompt with a structured output schema requiring a ranked list of passage IDs with relevance scores and a one-sentence rationale per passage. Parse the response with a JSON schema validator. If validation fails, retry once with the error message appended to the prompt. If the second attempt fails, log the failure and fall back to the original retrieval order as a degraded ranking. For high-stakes verification workflows, route outputs with low confidence scores or conflicting rationales to a human review queue before they reach the answer generator.

Model choice matters here. Ranking tasks benefit from models with strong comparative reasoning. Use a capable instruction-following model (such as Claude 3.5 Sonnet or GPT-4o) rather than a lightweight model. Set temperature=0 for deterministic rankings. If your candidate set is large (more than 20 passages), consider a two-stage approach: first use a fast embedding-similarity filter to reduce candidates to a manageable top-K, then apply this prompt for fine-grained ranking. Log every ranking output with the input candidates, the model's ranked order, and the downstream usage (which passages were actually fed to the generator). This trace data is essential for debugging hallucination incidents—you can determine whether the generator hallucinated despite having good evidence ranked highly, or whether the ranker failed to surface the right passage.

Bias testing should be part of your harness, not an afterthought. Periodically run eval sets where you deliberately place the most relevant passage at the beginning, middle, and end of the candidate list to measure position bias. Similarly, test with passages that have varying publication dates to detect recency bias. If you observe systematic bias toward certain positions or dates, add explicit instructions to the prompt template to counteract it (e.g., 'Evaluate each passage independently of its position in the list' or 'Do not prefer recent passages unless recency is relevant to the query'). Store these eval results alongside your prompt version so you can detect regressions when you update the prompt or switch models.