Evidence Ranking Explainability Prompt Template

Use this prompt when your RAG pipeline requires transparent, auditable explanations for why each retrieved passage was ranked at its specific position. This is essential for debugging retrieval quality, satisfying compliance requirements in regulated industries, or providing stakeholders with a clear rationale for evidence selection decisions. The ideal user is a search or retrieval engineer who has already implemented a ranking step and now needs to generate faithful, structured explanations for each ranking decision, including relevance factors, credibility weights, and explicit exclusion reasons for lower-ranked items.

This prompt is not a ranking prompt itself. It assumes that ranking has already occurred and focuses solely on generating explanations of that ranking. Do not use this prompt to perform the initial relevance scoring or passage ordering. Instead, wire it into your pipeline after your ranker has produced an ordered list of passages with associated metadata such as relevance scores, source credibility indicators, and retrieval timestamps. The prompt requires a ranked list of passages and their metadata as input and produces a structured explanation for each ranking decision. It is particularly valuable when you need to log decision traces for later audit, compare ranking behavior across prompt versions, or provide human reviewers with the reasoning behind automated evidence selection.

Avoid using this prompt for real-time, low-latency applications where explanation generation would add unacceptable overhead. It is designed for offline audit workflows, debugging sessions, and compliance review, not for user-facing streaming responses. If your system requires explanations to be surfaced to end users, consider generating them asynchronously and storing them for later retrieval. Additionally, do not rely on this prompt to detect ranking errors or hallucinations in the upstream ranker. It explains decisions faithfully based on the provided ranking and metadata, but it will not flag if the ranking itself is incorrect. Pair it with a separate evaluation step that compares explanations against ground-truth relevance judgments to detect discrepancies.

Before integrating this prompt, ensure you have a well-defined output schema for the explanations, including fields for rank position, passage identifier, relevance factors, credibility weights, and exclusion reasons. The prompt works best when the upstream ranker provides rich metadata that the explanation generator can reference. If your ranker only outputs scores without justification, the generated explanations may be superficial. To get started, prepare a batch of ranked passages from your production pipeline, define the JSON output contract you need, and run the prompt with a subset of queries to calibrate explanation quality before full deployment.

This template is designed to be dropped directly into your evidence selection pipeline. It forces the model to articulate the specific relevance factors, credibility weights, and exclusion reasons that drove each ranking decision. The output is structured for both human review and automated evaluation, making it suitable for regulated domains where evidence selection decisions must be explainable to auditors, compliance teams, or end users.

textCopy
You are an evidence ranking explainability system. Your task is to produce a detailed, auditable explanation for why each retrieved passage was assigned its final rank position.

## INPUT
- Query: [USER_QUERY]
- Ranked Passages (in final order): [RANKED_PASSAGES_JSON]
- Ranking Criteria Used: [RANKING_CRITERIA]
- Source Metadata (per passage): [SOURCE_METADATA_JSON]

## OUTPUT SCHEMA
Return a valid JSON object with this exact structure:
{
  "ranking_explanations": [
    {
      "passage_id": "string",
      "rank": integer,
      "relevance_factors": [
        {
          "factor": "string (e.g., 'query_term_overlap', 'entity_match', 'semantic_similarity')",
          "weight": number (0.0 to 1.0),
          "evidence": "string (specific text or metadata supporting this factor)"
        }
      ],
      "credibility_factors": [
        {
          "factor": "string (e.g., 'source_authority', 'recency', 'domain_expertise')",
          "weight": number (0.0 to 1.0),
          "evidence": "string (specific metadata or source attribute)"
        }
      ],
      "exclusion_reasons": [
        {
          "reason": "string (why this passage was not ranked higher, or why lower passages were excluded)",
          "compared_to": "string (passage_id of higher-ranked passage if applicable)"
        }
      ],
      "summary_justification": "string (2-3 sentence plain-English explanation of this passage's rank)"
    }
  ],
  "global_observations": {
    "ranking_philosophy": "string (how the criteria were applied overall)",
    "notable_tradeoffs": ["string (e.g., 'recency prioritized over authority for time-sensitive query')"],
    "exclusion_summary": "string (why passages beyond the top-K were excluded)"
  }
}

## CONSTRAINTS
- Every relevance and credibility factor must cite specific evidence from the passage text or source metadata. Do not use vague claims like "seems relevant."
- Exclusion reasons must reference a specific higher-ranked passage or a concrete ranking criterion that was not met.
- Weights must sum to 1.0 within relevance_factors and within credibility_factors for each passage.
- If a passage was excluded from the final set, include it with rank=null and explain why it was excluded.
- Do not fabricate metadata. Use only the provided source metadata.
- If the ranking criteria include recency, explicitly state the date comparison.

## EXAMPLES
Query: "What are the side effects of Drug X?"
Ranked Passage: { "passage_id": "p3", "text": "Clinical trial NCT123 showed that 12% of patients experienced mild nausea when taking Drug X...", "source": "FDA Label 2024" }
Explanation:
{
  "passage_id": "p3",
  "rank": 1,
  "relevance_factors": [
    { "factor": "entity_match", "weight": 0.6, "evidence": "Passage explicitly mentions 'Drug X' and 'side effects' via 'experienced mild nausea'" },
    { "factor": "information_density", "weight": 0.4, "evidence": "Provides specific percentage (12%) and symptom (nausea) rather than general warning" }
  ],
  "credibility_factors": [
    { "factor": "source_authority", "weight": 0.7, "evidence": "Source is FDA Label, the highest-authority drug information source" },
    { "factor": "recency", "weight": 0.3, "evidence": "Publication date 2024, within acceptable freshness window for drug safety" }
  ],
  "exclusion_reasons": [],
  "summary_justification": "Ranked first because it directly answers the query with specific, quantitative side effect data from the most authoritative source (FDA Label 2024). No higher-ranked passage exists."
}

## RISK LEVEL
[HIGH/MEDIUM/LOW - if HIGH, require human review of explanations before downstream use]

Adaptation guidance: Replace [RANKED_PASSAGES_JSON] with your already-ranked passage list including text and IDs. The [RANKING_CRITERIA] placeholder should contain the actual criteria your upstream ranker used (e.g., 'BM25 score weighted 0.4, semantic similarity weighted 0.4, recency weighted 0.2'). The [SOURCE_METADATA_JSON] must include at minimum source name, date, and authority level for each passage. If your system uses a different ranking philosophy, adjust the factor categories in the output schema accordingly. For high-risk domains, set [RISK_LEVEL] to HIGH and route explanations through a human review queue before they reach end users or downstream synthesis prompts.

Validation and eval: Before deploying, validate that the model's output conforms to the JSON schema using a structural validator. Then run an explanation faithfulness eval: take a sample of ranked passages, have a domain expert independently score the explanations for accuracy against the actual ranking criteria, and measure whether the model's stated factors match the criteria your upstream ranker used. A common failure mode is the model inventing plausible-sounding but incorrect justifications when the real ranking was driven by a simple BM25 score. If explanation faithfulness drops below your threshold, add a constraint requiring the model to explicitly map each factor back to a named ranking criterion from [RANKING_CRITERIA].

What to avoid: Do not use this prompt to generate the ranking itself—it assumes passages are already ranked. Do not skip the source metadata input; without it, the model will hallucinate credibility justifications. Do not treat the explanations as ground truth for model training without human verification, especially in regulated domains. If your upstream ranker uses opaque embeddings or learned weights that cannot be decomposed into named factors, this prompt will produce plausible but unfaithful explanations—consider switching to an interpretable ranker or disclosing the limitation.

This prompt is designed to sit between retrieval and answer synthesis in a RAG pipeline. It consumes a set of retrieved passages and produces a ranked list with human-readable explanations for each ranking decision. The output is not the final answer—it is an intermediate artifact that downstream components (synthesis prompts, citation formatters, audit loggers) consume. The harness must enforce a strict JSON output contract so that the ranking decisions are machine-readable and auditable, not buried in free-text prose. This means the application layer must validate the JSON schema, handle malformed outputs with retries, and log the full ranking trace before passing the top-K passages to the next stage.

Wiring the prompt into an application starts with defining the input contract. The prompt expects a [QUERY] string and a [PASSAGES] array of objects, each containing at minimum id, text, and optional metadata fields like source_title, date, and credibility_score. The application should assemble these from the retrieval step, ensuring each passage has a stable identifier for downstream citation. The prompt's [OUTPUT_SCHEMA] placeholder should be replaced with a concrete JSON schema specifying an array of ranked passages, each with rank, passage_id, relevance_score, credibility_weight, exclusion_reason (null if included), and explanation. The [CONSTRAINTS] placeholder should include the maximum number of passages to return, any minimum relevance threshold, and rules for handling ties or near-duplicates. The [RISK_LEVEL] placeholder should be set to high when the ranking decisions influence answers in regulated domains, triggering additional validation and human review gates.

Validation and retry logic is critical because malformed JSON or missing fields will break downstream synthesis. The harness should parse the model's response immediately and validate it against the expected schema. If validation fails, the system should retry with the same input and an added error message describing the specific validation failure—this is more effective than a generic retry. Set a maximum of 2 retries before falling back to a simpler ranking heuristic or escalating for human review. For high-risk domains, log every ranking decision with the full input, output, and validation result. This trace becomes audit evidence and enables regression testing when the prompt or model changes. Consider storing the ranking trace alongside the final answer so that reviewers can reconstruct why specific evidence was used or excluded.

Model choice and latency considerations matter here. This prompt requires reasoning about relevance, credibility, and exclusion—tasks that benefit from stronger models. Use a model with strong instruction-following and JSON output capabilities, such as GPT-4o or Claude 3.5 Sonnet, especially when the passage set is large or the domain is complex. For latency-sensitive applications, consider batching multiple ranking requests or using a smaller model with few-shot examples tuned to your specific ranking criteria. If the passage set exceeds the model's context window, pre-filter with a lightweight relevance scorer before invoking this explainability prompt. Never truncate passages silently—the harness should track which passages were excluded due to context limits and log that as a potential evidence gap.

Integration with downstream components requires the harness to extract the ranked passage IDs and pass them to the synthesis prompt in order. The synthesis prompt should receive only the top-K passages that meet the relevance threshold, along with their explanations as optional context for citation formatting. Do not pass the full ranking output to the synthesis prompt unless the synthesis prompt is explicitly designed to use ranking metadata—most synthesis prompts perform better with a clean, ordered evidence set. The harness should also expose the ranking trace to observability tools so that operators can monitor ranking quality over time, detect drift in relevance scoring, and compare ranking behavior across prompt versions. Start with a simple integration that logs rankings and validates outputs, then add complexity only when production data shows specific failure patterns.