Evidence Ranking with Output Contract Prompt Template

This prompt is designed for RAG system builders and search engineers who need a deterministic, machine-readable ranking of evidence passages before answer synthesis or citation selection occurs. The core job-to-be-done is converting an unordered set of retrieved candidate passages into a strictly ordered JSON array with explicit relevance scores, strength tiers, and selection rationale. You should use this prompt when your downstream application code requires a predictable output contract—no regex parsing, no retry-on-malformed-JSON, no guessing whether the model ranked passages or just summarized them. The ideal user is an engineer integrating this prompt into a retrieval pipeline where the ranking output feeds directly into a citation formatter, an answer generation step with a context budget, or a gating function that discards low-tier evidence before synthesis.

This prompt assumes you have already retrieved a set of candidate passages and need to order them. It is not a retrieval query writer, a passage rewriter, or a general-purpose summarization prompt. Do not use it when you need to generate search queries, expand user questions, or decide which documents to retrieve in the first place. Do not use it when your evidence set is a single passage—ranking a set of one produces noise, not value. Do not use it when your downstream consumer is a human reading a narrative report; this prompt optimizes for machine consumption with strict JSON output, not for prose explanations. If you need human-readable ranking explanations, pair this prompt with the Evidence Ranking with Explainability Output Prompt or add a separate explanation generation step after ranking is complete.

Before deploying this prompt, verify that your retrieval step produces candidate passages with stable identifiers that can survive the ranking and citation pipeline. The output contract requires a passage_id field for each ranked item, which means your retrieval system must assign and preserve IDs through the ranking step. If your retrieval system returns raw text without identifiers, add an ID assignment step before ranking. Also confirm that your application can handle the strength_tier enum values (HIGH, MEDIUM, LOW, INSUFFICIENT) and the relevance_score range (0.0 to 1.0) without additional normalization. The prompt includes a validation harness and retry logic for malformed rankings, but those guards only work if the upstream inputs are well-formed. Start with a small retrieval set (5-15 passages) and validate ranking stability before scaling to larger sets where position bias and length bias become more pronounced.

This prompt template is designed to be pasted directly into your system instructions or sent as a user message. It enforces a strict output contract: the model must return only a valid JSON array of ranked evidence objects. Each object includes the passage ID, a relevance score, a strength tier, and a concise rationale. The template uses square-bracket placeholders for all variable inputs, making it safe to inject data from your retrieval pipeline without risking prompt injection or format corruption.

textCopy
You are an evidence ranking engine. Your only job is to rank the provided evidence passages by how strongly they support answering the user's query.

## INPUT
Query: [QUERY]
Evidence Passages: [PASSAGES_JSON_ARRAY]

## RANKING CRITERIA
1. Relevance: How directly the passage addresses the query.
2. Specificity: How detailed and precise the information is.
3. Support Strength: How well the passage substantiates a potential answer.
4. Authority: Prefer passages from [AUTHORITY_SIGNALS] when available.

## CONSTRAINTS
- [CONSTRAINTS]

## OUTPUT CONTRACT
Return ONLY a valid JSON array. No other text. Each object must have these exact keys:
- "passage_id": string
- "relevance_score": number (0.0 to 1.0)
- "strength_tier": "HIGH" | "MEDIUM" | "LOW" | "INSUFFICIENT"
- "rationale": string (one sentence explaining the ranking)

## EXAMPLE OUTPUT
[
  {
    "passage_id": "doc_3_para_2",
    "relevance_score": 0.95,
    "strength_tier": "HIGH",
    "rationale": "Directly states the requested metric with a specific figure and date."
  }
]

## RISK LEVEL
[RISK_LEVEL]

Now rank the evidence.

To adapt this template, replace each square-bracket placeholder with your application data. The [PASSAGES_JSON_ARRAY] should be a serialized JSON array of objects, each containing at minimum an id and content field. The [CONSTRAINTS] placeholder lets you inject domain-specific rules, such as 'Exclude passages older than 2023' or 'Downgrade passages from unverified sources.' The [RISK_LEVEL] field should be set to HIGH if the ranking influences safety-critical decisions; in that case, add a human-review step after ranking. Always validate the returned JSON against the schema before passing it to downstream answer generation. If the model returns malformed JSON or extra text, use a repair prompt or retry with a stricter system instruction.

The Evidence Ranking with Output Contract prompt is designed to be called from application code, not used as a one-off chat. The prompt returns a strict JSON schema, which means your application must validate the response before passing ranked evidence downstream to answer generation or citation selection. Treat this prompt as a deterministic function call with a typed return value. If the model returns malformed JSON, missing required fields, or scores outside the defined range, the application must catch the error and retry rather than silently propagating broken rankings.

Wire the prompt into a validation-first harness. After receiving the model response, parse the JSON and validate it against the output contract: check that rankings is a non-empty array, each entry has a valid passage_id, relevance_score is a float between 0.0 and 1.0, strength_tier is one of the allowed enum values (high, medium, low, insufficient), and selection_rationale is a non-empty string. Reject any response where scores are identical across all passages (a sign of ranking collapse) or where the top-ranked passage has a strength_tier of insufficient without explanation. Log validation failures with the raw response and error details for debugging.

Implement a retry loop with a maximum of 3 attempts. On validation failure, re-send the prompt with the same inputs but append a brief error context to the system message: 'Your previous response failed validation. Ensure all fields match the required schema exactly. Relevance scores must be distinct where passages differ in quality.' If the third attempt still fails, fall back to a simpler ranking prompt without the strict output contract, log the incident, and alert the engineering team. Never silently return unranked or partially ranked evidence to downstream consumers.

For model selection, use a model with strong JSON-following behavior and sufficient context window for your retrieval set. GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro are appropriate defaults. Avoid models known to struggle with structured output consistency. Set temperature to 0 or a very low value (0.1 maximum) to reduce ranking instability. If your application requires reproducible rankings for audit purposes, log the full prompt, model version, and response for every call. For high-throughput systems, consider caching rankings for identical query-evidence pairs with a TTL appropriate to your evidence freshness requirements.

Log everything that matters for debugging. At minimum, capture: the query, the number of input passages, the model and version used, the raw response, validation status, retry count, and the final ranked output. If the ranking is used to gate downstream answer generation (e.g., only passages with strength_tier of high or medium are passed forward), log the gating decision and which passages were excluded. This trace data is essential for diagnosing ranking quality issues, position bias, and evidence selection errors in production. Wire these logs into your existing observability stack rather than creating a separate system.

When to escalate beyond this prompt: If your retrieval set regularly exceeds 20 passages, split the ranking into batches and merge results with a second ranking pass. If evidence conflicts are common in your domain, pair this prompt with the Evidence Ranking with Contradiction Detection variant. If rankings must be explainable to end users, switch to the Evidence Ranking with Explainability Output prompt. And if ranking quality directly affects safety-critical decisions, add a human review step for low-confidence rankings or cases where the top passage has a relevance_score below 0.5.