Multi-Passage Quote Ranking Prompt

This prompt is for RAG pipeline architects and search engineers who need to order multiple candidate quotes by their strength in supporting a specific claim. It operates after retrieval and extraction, producing a ranked list with per-quote justification. Use it when downstream answer generation or citation selection depends on a prioritized evidence set rather than a flat list of passages. The prompt assumes you already have a target claim and a set of candidate quotes extracted from source documents. It does not perform extraction itself and should not be used as a standalone fact-checker or answer generator.

The ideal input is a clean set of verbatim quotes with source metadata, paired with a single, well-formed claim. The prompt works best when quotes are already filtered for basic relevance—ranking quality degrades if the input set contains mostly irrelevant passages. You should not use this prompt when you need to extract quotes from raw documents, when you need to verify factual accuracy against external ground truth, or when the claim itself is ambiguous and requires clarification. For extraction, use a dedicated quote extraction prompt first. For verification, pair this with a claim verification prompt that checks the top-ranked quote against external evidence.

Before deploying this prompt, define your evaluation criteria. Common metrics include NDCG against human relevance judgments, downstream answer quality impact, and citation precision in the final output. The prompt includes a structured output schema that makes automated evaluation straightforward—each quote receives a rank, a support strength score, and a justification string. Wire these outputs into your eval harness to catch regressions when you update retrieval or extraction upstream. If your use case involves regulated content, add a human review step before ranked quotes feed into customer-facing answers.

This template provides the exact prompt structure for the multi-passage quote ranking task. It is designed to be copied directly into your application code or prompt management system. The prompt enforces a strict JSON output contract, instructs the model not to introduce external knowledge, and includes a clear fallback for cases where no quote provides adequate support. Use this as your starting point before adding domain-specific ranking criteria or integrating it into a larger RAG pipeline.

textCopy
Rank the following candidate quotes by how strongly they support the target claim. Consider relevance, specificity, and directness of support. Do not introduce external knowledge. Return only the JSON object specified in the output format.

**Target Claim:**
[CLAIM]

**Candidate Quotes:**
[QUOTE_LIST]

**Ranking Criteria (optional):**
[RANKING_CRITERIA]

**Output Format:**
Return a single JSON object with a "ranked_quotes" array. Each element must have:
- "quote_id": the original identifier from the candidate list
- "rank": integer starting at 1 for strongest support
- "relevance_score": float between 0.0 and 1.0
- "justification": one sentence explaining why this quote received this rank

If no quote provides adequate support, return {"ranked_quotes": []}.

Adapting the template: The [QUOTE_LIST] placeholder expects a structured list of quote objects, each with a unique quote_id and the quote text. Format this as a numbered list or a JSON array depending on what your model handles best. The optional [RANKING_CRITERIA] field lets you inject domain-specific priorities—for example, you might instruct the model to weigh recency over specificity for time-sensitive claims, or to prioritize quotes from peer-reviewed sources in scientific contexts. If your use case involves regulated or high-stakes decisions, add a [CONSTRAINTS] section that requires the model to flag ambiguous rankings or request human review when support strength differences are marginal.

What to do next: After copying this template, wire it into a validation harness that checks the output JSON structure, verifies that all returned quote_id values exist in the input set, and confirms that ranks are sequential integers starting at 1. For production deployments, pair this prompt with an LLM judge eval that compares the model's rankings against human-annotated ground-truth relevance judgments. Monitor for common failure modes such as the model assigning identical scores to quotes with clearly different support strength, or returning rankings that contradict the explicit criteria you provided.

This prompt is designed to sit between quote extraction and answer generation in a RAG pipeline. After your retrieval system returns candidate passages and your extraction prompt pulls verbatim quotes, this ranking prompt orders them by relevance, specificity, and support strength before they reach the answer generator. The ranked output reduces noise, prevents weak evidence from diluting the final answer, and gives downstream components a clear priority order for citation selection. The implementation harness must enforce the output contract, handle failures gracefully, and leave an audit trail that explains why certain quotes were promoted or demoted.

Call the model with structured output mode enabled. For OpenAI, set response_format to json_object and provide the output schema in the system message or as a tool definition. For Claude, use the structured output API with the same JSON schema. Validate the returned JSON against the expected contract immediately: check that ranked_quotes is an array, each entry has quote_text, rank, relevance_score, and justification fields, and ranks are sequential integers starting from 1. If validation fails, implement a retry with temperature variation—bump temperature by 0.1 on each retry up to a maximum of 0.4, and cap retries at three attempts. After three failures, log the raw output and route to a human review queue rather than silently passing bad data downstream.

If the model returns an empty ranked_quotes array, this is a signal that no candidate quote met the relevance threshold. Do not pass an empty list to the answer generator. Instead, trigger a fallback path: either expand the retrieval scope by reformulating the query and re-running extraction, or signal insufficient evidence to the answer generator so it can produce an honest abstention response. Log the ranking justification for every quote in the output—this justification text is your audit trail for debugging ranking quality, explaining citation choices to users, and running offline evaluation against ground-truth relevance judgments. For high-throughput systems, batch multiple claims with their candidate quote sets into a single request if the model context window permits, but keep each claim's quotes clearly delimited to avoid cross-contamination in ranking decisions.