Evidence Confidence Score Assignment Prompt

This prompt is for RAG pipeline architects and search engineers who need to move beyond a flat list of retrieved documents. Its job is to assign a calibrated confidence tier—high, medium, low, or insufficient—to each evidence passage based on specificity, directness of support, and source quality. Use it when your downstream answer generation or citation logic must gate on evidence strength, not just relevance. The ideal user is someone integrating this step between retrieval and generation in a production system where hallucination risk must be controlled programmatically.

Deploy this prompt when you have a set of candidate passages and need a structured, machine-readable confidence label for each one before they enter the final context window. It is appropriate for high-stakes domains like legal research, clinical decision support, or financial analysis where using weak evidence is worse than admitting uncertainty. The prompt expects a query, a claim or question, and a list of passages with source metadata. It is not designed for ranking passages against each other in a head-to-head comparison—use the Pairwise Evidence Comparison Prompt for that. It is also not a replacement for a retrieval model; it assumes you already have candidates and need to score their evidentiary weight.

Do not use this prompt when you need a single ranked list sorted by relevance. For that, use the Multi-Passage Evidence Ranking Prompt Template. Avoid it when your primary goal is to detect contradictions between sources—the Evidence Ranking with Contradiction Detection Prompt handles that explicitly. If you are operating under a strict token budget and need to select a top-K subset, combine this prompt's output with the Top-K Evidence Selection Prompt Template. Finally, if your system requires a detailed, human-readable explanation for every score, use the Evidence Ranking with Explainability Output Prompt instead. This prompt prioritizes structured, gating-ready labels over narrative justification.

This prompt template is designed to be dropped into a RAG pipeline immediately after retrieval and before answer generation. It forces the model to evaluate each evidence passage against explicit criteria—specificity, directness of support, and source quality—and assign a calibrated confidence tier. The output is a ranked list with confidence labels that downstream components can use for gating, citation selection, or answer abstention. The template uses square-bracket placeholders so you can wire in your own retrieval results, query context, and output constraints without modifying the core evaluation logic.

textCopy
You are an evidence quality evaluator for a retrieval-augmented generation system. Your task is to assign a confidence tier to each evidence passage based on how well it supports the user's query.

## INPUT
**User Query:** [QUERY]
**Retrieved Evidence Passages:**
[EVIDENCE_PASSAGES]

## EVALUATION CRITERIA
For each passage, assess three dimensions on a scale of 1-5:
1. **Specificity:** How precisely does the passage address the query? (1 = vague/general, 5 = highly specific)
2. **Directness of Support:** Does the passage directly support an answer, or is it tangential? (1 = unrelated, 5 = directly confirms or denies)
3. **Source Quality:** Consider the source's authority, recency, and reliability based on available metadata. (1 = low quality/unknown, 5 = high authority/verified)

## CONFIDENCE TIERS
- **HIGH:** Combined score >= 12, with no dimension below 3. The passage provides strong, direct, authoritative support.
- **MEDIUM:** Combined score 8-11, or any dimension at 2. The passage is relevant but has gaps in specificity, directness, or authority.
- **LOW:** Combined score 5-7, or any dimension at 1. The passage is weakly relevant and should not be used as primary evidence.
- **INSUFFICIENT:** Combined score < 5. The passage does not meaningfully support the query and should be discarded.

## CONSTRAINTS
[CONSTRAINTS]

## OUTPUT SCHEMA
Return a JSON object with the following structure:
{
  "ranked_evidence": [
    {
      "passage_id": "string",
      "confidence_tier": "HIGH|MEDIUM|LOW|INSUFFICIENT",
      "scores": {
        "specificity": 1-5,
        "directness": 1-5,
        "source_quality": 1-5
      },
      "rationale": "Brief explanation of the assigned tier."
    }
  ],
  "summary": {
    "total_passages": number,
    "high_count": number,
    "medium_count": number,
    "low_count": number,
    "insufficient_count": number,
    "sufficient_evidence_available": true|false
  }
}

## INSTRUCTIONS
1. Evaluate every passage independently before comparing.
2. If source metadata is missing, score source_quality as 2 (unknown) unless the content itself signals authority.
3. If no passage reaches HIGH confidence, set sufficient_evidence_available to false.
4. Do not fabricate scores. If a passage is genuinely insufficient, label it as such.

To adapt this template for your pipeline, replace [QUERY] with the user's original question and [EVIDENCE_PASSAGES] with your retrieval results formatted as an array of objects containing at minimum a passage_id and text field. The [CONSTRAINTS] placeholder lets you inject domain-specific rules—for example, requiring recency thresholds for financial data or mandating that only peer-reviewed sources can reach HIGH confidence. After the model returns its JSON output, validate the structure against the schema before passing ranked passages to your answer generation step. For high-stakes domains, add a human review gate when sufficient_evidence_available is false or when the gap between HIGH and MEDIUM counts exceeds a threshold you define.

This prompt is designed to sit between retrieval and answer generation in a RAG pipeline. It takes a set of retrieved passages and assigns each a confidence tier—high, medium, low, or insufficient—based on specificity, directness of support, and source quality. The output is a confidence-labeled ranking that downstream components can use to gate answer generation: only passages meeting a minimum confidence threshold should be passed to the answer synthesizer. This prevents the model from building answers on weak or tangential evidence.

Wire the prompt into your application as a post-retrieval, pre-generation step. After your retriever returns candidate passages, batch them with the user query and any available source metadata (publication date, author authority, document type) into the [EVIDENCE_PASSAGES] and [QUERY] placeholders. The model should return a structured JSON array with each passage ID, its assigned tier, and a brief rationale. Validate the output immediately: confirm every passage ID in the input appears in the output, that tier values are only high, medium, low, or insufficient, and that rationales are non-empty strings. If validation fails, retry once with the same input and a stronger constraint instruction appended. If the second attempt fails, log the failure and route the request to a human review queue rather than proceeding with unlabeled evidence.

For production deployments, implement a confidence gate before answer generation. Define a minimum tier threshold—typically medium or higher—and filter out passages below that threshold. If no passages meet the threshold, the system should return a grounded refusal rather than fabricating an answer from weak evidence. Log every confidence assignment with the passage ID, tier, rationale, and model version for auditability. When source metadata is available, consider running a secondary check: if a passage from a high-authority source receives a low confidence score, flag it for human review, as this may indicate a retrieval-query mismatch rather than genuinely weak evidence. Avoid using this prompt on retrieval sets larger than 20 passages without chunking, as confidence assignment quality degrades with long context windows and position bias becomes more pronounced.