Evidence Ordering Prompt for Multi-Document Synthesis

This prompt is designed for RAG system builders who face a common bottleneck: the retrieval step returns more passages than can fit into the generation context window, and the raw retrieval order is not optimized for downstream synthesis. The job-to-be-done is to rank, deduplicate, and sequence retrieved evidence so the generation model receives a clean, prioritized set of passages rather than a noisy dump. The ideal user is an AI engineer or backend developer integrating this prompt as a filter between retrieval and generation, typically in a pipeline where retrieval recall is high but precision and ordering are insufficient for reliable answer quality.

Use this prompt when you observe that your generator is missing key facts buried in low-ranked passages, repeating information from near-duplicate sources, or producing inconsistent answers because evidence ordering varies across requests. It is particularly valuable when you have a fixed context budget and need to make deliberate trade-offs about which passages to include, compress, or drop. The prompt requires several inputs to function correctly: a set of retrieved passages with source identifiers, the original user query or task description, an ordering strategy selection (such as relevance-first, authority-weighted, or recency-prioritized), and a token budget constraint. Without these, the model cannot make informed ordering decisions and will default to arbitrary ranking.

Do not use this prompt when retrieval already returns a small, highly precise set of passages that fits comfortably in the context window—the overhead of ordering adds latency without meaningful quality improvement. It is also inappropriate when the downstream generator has its own effective internal attention mechanisms for long contexts and adding an intermediate step introduces unnecessary complexity. Avoid this prompt in streaming or real-time applications where the added latency of a separate ordering call breaks user experience expectations. For high-risk domains such as healthcare or legal review, this prompt should be paired with human-in-the-loop validation of the ordered evidence set before generation, as incorrect prioritization can silently drop critical information. After implementing this prompt, the next step is to wire it into your pipeline with validation checks that verify the output contains the expected number of passages, that deduplication markers are present, and that the total token count respects the specified budget.

This template is designed to be pasted into your prompt assembly layer. It instructs the model to act as an evidence ordering engine, taking a user query and a set of retrieved documents, and returning a structured, ranked list. The core job is to prioritize the most relevant and authoritative passages while managing your context window budget. Replace every [PLACEHOLDER] with runtime values from your retriever and application logic before sending the request.

textCopy
You are an evidence ordering engine for a multi-document RAG system. Your task is to analyze a set of retrieved documents against a user query and produce a ranked, deduplicated list of evidence passages for downstream answer generation.

## INPUT
**User Query:** [USER_QUERY]
**Retrieved Documents:** [RETRIEVED_DOCUMENTS]
**Context Window Budget:** [MAX_OUTPUT_TOKENS]
**Ordering Strategy:** [ORDERING_STRATEGY]
**Source Authority Weights:** [AUTHORITY_WEIGHTS]

## TASK
1.  **Relevance Scoring:** Score each document's relevance to the user query on a scale of 0.0 to 1.0.
2.  **Deduplication:** Identify near-duplicate passages. For each set of duplicates, keep only the one from the most authoritative source and note the merged sources.
3.  **Ordering:** Sort the remaining unique passages into a single list based on the selected `[ORDERING_STRATEGY]` (e.g., 'relevance_then_authority', 'recency_weighted', 'diverse_perspectives').
4.  **Budget Truncation:** Truncate the final list to fit within the `[MAX_OUTPUT_TOKENS]` budget, adding an explicit note about how many passages were omitted.

## OUTPUT SCHEMA
Return a single JSON object with the following structure:
{
  "ordered_evidence": [
    {
      "rank": 1,
      "passage_id": "string",
      "relevance_score": 0.95,
      "source_authority": "high",
      "merged_source_ids": ["doc_1", "doc_5"],
      "passage_text": "string"
    }
  ],
  "omitted_passage_count": 3,
  "omission_note": "string explaining why passages were omitted (e.g., low relevance, budget constraints)"
}

## CONSTRAINTS
- Do not synthesize an answer to the user query. Only rank and order the provided evidence.
- If no documents are relevant, return an empty `ordered_evidence` array and a clear omission note.
- Preserve the original text of the passages exactly; do not paraphrase.

To adapt this template, start by mapping your retriever's output fields to the [RETRIEVED_DOCUMENTS] placeholder, ensuring each document object has a unique ID, text, and source metadata. The [ORDERING_STRATEGY] and [AUTHORITY_WEIGHTS] variables should be controlled by your application logic, not the end-user, to enforce system-wide policies. For high-stakes domains like healthcare or finance, you must add a human review step before the ordered evidence is used for final answer generation. Always validate the output JSON against the schema; a malformed response should trigger a retry or fallback to a default ordering.

The evidence ordering prompt is not a standalone artifact. It sits between retrieval and generation in a RAG pipeline. The typical call flow is: user query → retrieval (vector, keyword, hybrid) → deduplication → evidence ordering prompt → top-k selection → answer generation prompt. The ordering prompt receives a list of candidate passages with metadata (source, date, retrieval score) and returns a ranked, deduplicated, and relevance-scored list. The application layer, not the prompt, enforces the context window budget by truncating to the top N passages after ordering.

Wire the prompt into your application with a thin wrapper that handles input assembly, output parsing, and validation. Before calling the model, assemble the [RETRIEVED_PASSAGES] array with id, text, source, date, and retrieval_score fields. Inject the [ORDERING_STRATEGY] (e.g., relevance_then_recency, authority_weighted, diversity_aware) and [CONTEXT_BUDGET_TOKENS] as runtime parameters. After the model responds, parse the JSON output and validate: every passage ID in the output must exist in the input; relevance scores must be in the declared range; deduplication markers must reference valid pairs; and the total estimated token count of the ordered list must not exceed the budget. If validation fails, retry once with the validation errors appended to the prompt as [PREVIOUS_ERRORS]. If the retry also fails, fall back to retrieval-score ordering and log the incident for review.

For production observability, log the full prompt, the model response, validation results, and the final ordered list to your trace store. Tag traces with prompt_version, ordering_strategy, passage_count, and model_id. Set up an eval harness that runs weekly: sample 50 queries, retrieve passages, run the ordering prompt, and have a separate LLM judge prompt score the ordering against criteria like 'most relevant passage appears first' and 'duplicates are correctly merged.' Track the pass rate over time. If you observe ordering failures clustering around specific retrieval sources or query types, use those patterns to tune the ordering strategy or add few-shot examples to the prompt. Avoid calling this prompt on every user request if retrieval returns fewer than 3 passages; a simple sort by retrieval score is cheaper and equally effective for small sets.