Multi-Turn Evidence Ranking Prompt with History

This prompt is for retrieval engineers and RAG pipeline builders who need to re-rank a set of retrieved passages when the user's follow-up question shifts the conversational context. The core job-to-be-done is producing a relevance-ordered list of evidence chunks that weights each passage against both the immediate query and the established conversation history. Without this step, a naive re-ranker will treat every turn as a fresh search, discarding the shared context that makes follow-ups coherent. The ideal user is someone integrating this into a retrieval pipeline—typically a backend engineer or AI architect—who already has a vector or keyword retrieval stage and needs a second-pass ranking that is conversation-aware.

Use this prompt when your application supports multi-turn chat or copilot interactions over a knowledge base, and when the user's current question cannot be answered well by treating it in isolation. It is appropriate for scenarios where prior turns have narrowed the domain (e.g., 'Tell me more about the second option'), introduced constraints ('Only show me results from 2024'), or established entities that should influence relevance. The prompt requires you to supply the conversation history, the current question, and a list of candidate passages with identifiers. It returns a ranked list with relevance scores and brief justifications, which your application can then threshold, truncate, or pass to a synthesis prompt.

Do not use this prompt for single-turn Q&A where no conversation history exists—a standard relevance ranking prompt will be simpler and cheaper. Avoid it when the history is extremely long and exceeds your context budget; in that case, apply a session summarization or history compression prompt first, then feed the compressed context into this ranker. This prompt is also not a substitute for initial retrieval. It assumes you already have a candidate set from a vector database, keyword index, or hybrid search. If your retrieval stage is returning zero results or entirely off-topic passages, fix that pipeline before adding a re-ranker. Finally, in high-stakes domains like clinical or legal Q&A, the ranking justifications should be logged and reviewed, not blindly trusted, because a ranking error can silently drop the most authoritative source.

This prompt template is designed for retrieval engineers who need to re-rank evidence when conversation context shifts. It takes the current user question, the conversation history, and a set of retrieved passages, then produces a relevance ranking that weights each passage against both the immediate query and the established conversational state. The output is a structured ranking with scores and explanations, suitable for downstream answer synthesis or evidence selection.

textCopy
You are an evidence ranking system for a multi-turn conversational AI. Your job is to score and rank retrieved passages based on their relevance to the current user question, taking into account the conversation history for context.

## CONVERSATION HISTORY
[HISTORY]

## CURRENT QUESTION
[QUESTION]

## RETRIEVED PASSAGES
[PASSAGES]

## RANKING CRITERIA
1. Direct relevance to the current question (primary weight).
2. Consistency with or relevance to prior conversation turns (secondary weight).
3. Information novelty — prefer passages that add new, non-redundant information.
4. Temporal relevance — if the history or question implies a time frame, prefer passages that match.
5. Source authority — if source metadata is provided, prefer more authoritative or recent sources.

## OUTPUT FORMAT
Return a JSON object with the following structure:
{
  "ranked_passages": [
    {
      "passage_id": "string",
      "relevance_score": 0.0-1.0,
      "relevance_explanation": "Brief explanation of why this passage is relevant to the current question and conversation context",
      "key_facts": ["fact from passage relevant to question"]
    }
  ],
  "ranking_summary": "Brief summary of the ranking logic applied",
  "conversation_context_used": "How the conversation history influenced the ranking"
}

## CONSTRAINTS
- Score each passage independently, then rank by score descending.
- If a passage is irrelevant to both the question and conversation context, score it below 0.3.
- If the conversation history indicates a topic shift, deprioritize passages relevant only to the old topic.
- Do not fabricate facts. Only use information present in the passages.
- If no passages are relevant, return an empty ranked_passages array and explain why.

To adapt this template, replace [HISTORY] with the formatted conversation turns, [QUESTION] with the current user query, and [PASSAGES] with the retrieved evidence set including passage IDs and source metadata. For high-stakes domains, add a [RISK_LEVEL] parameter that adjusts the scoring threshold for inclusion and requires a human review flag when confidence is low. Wire the output into your evidence selection pipeline by filtering passages above a configurable relevance threshold before passing them to the answer synthesis step.

This prompt is designed to sit between your retrieval system and your answer generation step. It expects a structured payload containing the current user question, a compressed or full conversation history, and a batch of retrieved passages. The output is a ranked list of passages with relevance scores and a brief justification for each ranking decision. In a production harness, you should call this prompt after every retrieval event in a multi-turn session—not just on the first turn—because the relevance of previously retrieved passages can shift dramatically as the conversation evolves.

Wire the prompt into your application as a stateless ranking service. Construct the input payload by merging the latest user utterance with a conversation history summary (use the Session Summarization Prompt for Long Conversations if the history is large) and the top-N passages from your retriever. Validate the output against a strict schema: each ranked passage must include a passage_id, a relevance_score (float 0-1), and a justification string. If the model returns malformed JSON or missing fields, retry once with a repair prompt that includes the raw output and the expected schema. Log every ranking event—including input context, raw model output, parsed rankings, and any validation errors—so you can debug relevance drift and model inconsistencies later.

For high-stakes domains, insert a human review step when the top-ranked passage's score falls below a configurable threshold (e.g., 0.6) or when the ranking justification indicates the model is uncertain about context relevance. This prevents the downstream answer generator from synthesizing a response from weak evidence. Before deploying, build an eval harness that tests the ranker against conversation scenarios where the topic shifts, the user corrects a prior answer, or the retriever returns partially stale passages. Measure whether the ranker correctly deprioritizes passages that were relevant two turns ago but are now off-topic. Avoid calling this prompt on every keystroke or partial utterance—batch it on complete user turns to control latency and cost.