Source Ranking Explanation Prompt Template

This prompt is for RAG pipeline engineers who need to generate a human-readable explanation of how a set of retrieved sources was ranked. It takes the final ranked list, the ranking criteria, and the original query, and produces a structured explanation that a user or auditor can read to understand why one source appears above another. Use this when your application surfaces ranked evidence to end users, when compliance reviewers demand transparency into your retrieval logic, or when you need to debug ranking quality by inspecting the model's stated reasoning.

Do not use this prompt to perform the ranking itself. It explains an already-computed ranking. It does not re-rank, re-score, or select evidence. The ranking algorithm must run before this prompt is called. This prompt is a post-hoc explainability layer, not a ranking engine. If you need to rank sources, use a dedicated ranking prompt or a retrieval scoring model first, then pass the results here for explanation. The prompt requires three concrete inputs: the original user query, the final ordered list of sources with their metadata, and the explicit criteria that drove the ranking. Without all three, the explanation will be vague or misleading.

Before wiring this into production, define what a good explanation looks like for your use case. For audit scenarios, you may need strict traceability from each ranking decision to a specific criterion. For user-facing features, you may prioritize readability and conciseness. Build eval checks that compare the generated explanation against the actual ranking algorithm's behavior—if the explanation says source A outranks source B because of recency, but your ranker actually prioritized authority, the explanation has failed. Always validate explanation fidelity before surfacing it to users or auditors.

This prompt template is designed to be dropped directly into your RAG pipeline's explainability layer. It takes a pre-computed ranking—the output of your retrieval and scoring system—and produces a structured JSON explanation that non-technical users can understand. The prompt does not re-rank sources; it explains the ranking your system already produced. This separation of concerns keeps your ranking algorithm as the single source of truth while the LLM handles the communication layer.

textCopy
You are an explainability engine for a retrieval-augmented generation system. Your job is to produce a human-readable explanation of why a set of sources was ranked in a specific order. You do not re-rank the sources. You explain the ranking that was already computed.

## User Query
[USER_QUERY]

## Ranking Criteria
The sources were ranked using the following criteria, in order of importance:
[RANKING_CRITERIA]

## Ranked Sources
Below is the final ranked list. Each source includes its rank position, identifier, title, author, publication date, relevance score, and a short content snippet.

[RANKED_SOURCE_LIST]

## Instructions
Generate a structured explanation with these sections:

1. **Ranking Summary**: A 2-3 sentence overview of how the sources were ordered, referencing the criteria.
2. **Per-Source Explanation**: For each source, explain why it received its rank. Reference the specific criteria that drove its position. For the top-ranked source, explain what made it the strongest match. For lower-ranked sources, explain what factors deprioritized them relative to higher-ranked sources.
3. **Key Trade-offs**: If any sources were close in score, explain what tipped the balance. If any criteria conflicted (e.g., high relevance but low recency), describe how the conflict was resolved.

## Constraints
- Do not invent criteria that are not listed in [RANKING_CRITERIA].
- Do not claim a source was ranked for a reason that contradicts its metadata or snippet.
- If a source's rank is primarily driven by its score, state that explicitly rather than fabricating additional justification.
- Use the source identifiers exactly as provided in [RANKED_SOURCE_LIST].
- If the ranking criteria include weights or thresholds, reference them in your explanation.
- Write in plain English suitable for a non-technical end user. Avoid jargon like 'cosine similarity' or 'BM25' unless the criteria explicitly use those terms.

## Output Format
Return a JSON object with this schema:
{
  "ranking_summary": "string",
  "per_source_explanations": [
    {
      "source_id": "string",
      "rank": number,
      "explanation": "string"
    }
  ],
  "key_tradeoffs": [
    {
      "description": "string",
      "sources_involved": ["string"]
    }
  ]
}

To adapt this template, replace the four bracketed placeholders with your pipeline's actual data. [USER_QUERY] should contain the original question or search string. [RANKING_CRITERIA] must list the exact criteria your ranker used, including any weights or thresholds—this is what the LLM will reference, so be precise. [RANKED_SOURCE_LIST] needs a structured dump of your ranked results: each entry should include at minimum the source identifier, title, author, publication date, relevance score, and a content snippet. The snippet is critical because the LLM uses it to ground its explanations in actual document content rather than hallucinating reasons.

The output schema is intentionally flat and deterministic. Each source gets exactly one explanation object with its rank and source_id. The key_tradeoffs array captures edge cases where scores were close or criteria conflicted. This structure makes it straightforward to validate in your application layer: check that every source_id in the input appears in the output, verify that ranks match, and confirm that explanations reference only criteria present in [RANKING_CRITERIA]. For high-stakes deployments, add a validation step that compares the explanation's claims against the source snippets to catch hallucinated justifications before they reach users.

This prompt works best with models that have strong instruction-following and JSON output capabilities. GPT-4, Claude 3.5 Sonnet, and similarly capable models handle the constraint-following well. Weaker models may drift into re-ranking behavior or fabricate criteria. If you observe these failure modes, add a stronger preamble: 'You are forbidden from changing the rank order. Your only job is to explain the existing ranking.' For production systems, log every explanation alongside the input ranking for auditability, and consider periodic human review of a sample to detect explanation drift over time.

Wire this prompt after your ranking algorithm produces a final ordered list. The prompt is a post-processing step that runs before the ranked list is surfaced to the user. In a typical RAG pipeline, the flow is: query → retrieval → scoring/ranking → this prompt → user-facing display. Call this prompt synchronously before rendering the ranked source list in your UI. If the prompt fails or times out, fall back to displaying the ranked list without explanations rather than blocking the user.

Log every explanation alongside the ranking inputs for audit and debugging. If your application is in a regulated domain, store the explanation as part of the decision record. For latency-sensitive applications, consider running this prompt asynchronously after the ranked list is displayed, then injecting explanations when they are ready. Set a timeout of 5 seconds for the model call. If the output fails JSON schema validation, retry once with the validation error message appended to the prompt. If the retry also fails, log the failure and surface the ranked list without explanations. Do not retry more than once to avoid latency spikes.

Choose a model that balances explanation quality with latency. Faster models (e.g., GPT-4o-mini, Claude Haiku) work well for this task since the prompt requires structured reasoning over an already-ranked list rather than complex generation. Validate the output against a strict JSON schema that enforces the explanation structure, required fields, and source reference integrity. Every explanation must reference sources by their IDs from the input ranking list—reject outputs that invent source IDs or omit ranked sources without justification. Before deploying, run this prompt against a golden dataset of ranked lists with known good explanations to establish baseline eval scores for explanation fidelity and schema compliance.