Evidence Scoring Prompt with Confidence Intervals

This prompt is for RAG pipeline architects and search ranking engineers who need calibrated uncertainty in evidence quality assessment. Use it when downstream systems must make risk-aware selection decisions based on evidence scores, and a single point estimate is insufficient. The prompt produces a relevance score with explicit upper and lower confidence bounds, a calibration flag, and a justification for the uncertainty range. This is not a prompt for simple passage filtering or binary relevance judgments. It assumes you already have a set of retrieved passages and a query, and you need a structured, interval-based score to feed into a re-ranking or selection algorithm that accounts for model uncertainty.

The ideal user is building a production system where evidence selection errors have measurable consequences—such as a financial research assistant where outdated or low-authority sources could mislead analysts, or a medical literature RAG system where overconfident passage selection could surface weak evidence for clinical questions. In these scenarios, a single relevance score hides the model's uncertainty. A passage scored 0.85 might actually be anywhere from 0.60 to 0.95 in the model's own assessment, and downstream logic needs that range to decide whether to include, exclude, or flag the passage for human review. This prompt is also appropriate when you are building evaluation datasets for evidence scoring and need to capture model calibration alongside raw scores.

Do not use this prompt when a simple relevance threshold is sufficient for your use case, when latency budgets cannot accommodate the additional tokens required for confidence interval generation, or when your downstream system cannot consume interval data. If you are building a prototype or a low-stakes internal tool, start with a single-point relevance scoring prompt and only introduce confidence intervals when you have evidence that uncertainty matters for your outcomes. Similarly, if your retrieval pipeline already produces well-calibrated scores from a dedicated cross-encoder model, adding an LLM-based confidence layer may introduce unnecessary complexity and cost. Reserve this prompt for systems where the LLM's own uncertainty assessment adds decision value beyond what retrieval scores provide.

Before deploying, validate that the confidence intervals are empirically meaningful. Run the prompt against a calibration set with known relevance labels and check whether the reported intervals actually contain the true relevance values at the expected rate. If the model reports 90% confidence intervals but the true value falls outside the interval 30% of the time, the prompt is miscalibrated and should not drive downstream decisions without adjustment. Pair this prompt with an eval harness that logs interval coverage rates, average interval width, and calibration error metrics. When the calibration flag indicates low confidence in the score itself, route the passage to human review or apply a penalty in the downstream ranking algorithm.

The prompt below assigns a quality score to each evidence passage along with upper and lower confidence bounds. It is designed for production RAG pipelines where downstream selection, synthesis, or refusal logic needs to understand not just how good the evidence is, but how certain the model is about that assessment. Use this template when you need risk-aware evidence ranking—for example, when overconfident scoring on ambiguous passages could cause hallucinated answers or when compliance reviewers need to see uncertainty expressed explicitly.

textCopy
You are an evidence quality assessor for a retrieval-augmented generation system. Your job is to score each evidence passage on how well it supports answering the query, and to express your confidence in that score using upper and lower bounds.

## INPUT
Query: [QUERY]
Evidence Passages:
[EVIDENCE_PASSAGES]

## SCORING CRITERIA
For each passage, evaluate:
1. Relevance: How directly does the passage address the query?
2. Specificity: Does the passage contain concrete facts, figures, or details, or is it vague?
3. Authority: Based on [AUTHORITY_SIGNALS], how credible is the source?
4. Recency: Given the query's temporal needs and the passage date, is the information current?

## OUTPUT SCHEMA
Return a JSON object with this exact structure:
{
  "passages": [
    {
      "passage_id": "string",
      "score": number (0.0 to 1.0),
      "confidence_lower": number (0.0 to 1.0),
      "confidence_upper": number (0.0 to 1.0),
      "factor_breakdown": {
        "relevance": number (0.0 to 1.0),
        "specificity": number (0.0 to 1.0),
        "authority": number (0.0 to 1.0),
        "recency": number (0.0 to 1.0)
      },
      "uncertainty_rationale": "string explaining why the confidence interval is wide or narrow"
    }
  ],
  "calibration_notes": "string describing any systematic uncertainty patterns across the batch"
}

## CONSTRAINTS
[CONSTRAINTS]

## EXAMPLES
[EXAMPLES]

## INSTRUCTIONS
- Score every passage independently. Do not compare passages to each other.
- The confidence interval must reflect genuine uncertainty. If a passage is ambiguous, contradictory, or missing key details, widen the interval.
- If you cannot assess a factor due to missing information, set that factor to null and widen the confidence interval accordingly.
- The uncertainty_rationale must cite specific passage content that drove the interval width.
- Do not invent information. If the passage lacks authority signals, note that rather than guessing.

Adaptation guidance: Replace [QUERY] with the user's question or information need. [EVIDENCE_PASSAGES] should be a list of retrieved passages with IDs, text, and metadata including publication dates and source identifiers. [AUTHORITY_SIGNALS] is a description of what makes a source authoritative in your domain—for example, "peer-reviewed journals and government publications are high authority; personal blogs are low authority." [CONSTRAINTS] can include domain-specific rules such as "medical evidence older than 5 years must have its recency score capped at 0.3" or "legal evidence from overturned precedents must be flagged." [EXAMPLES] should contain 2-3 scored passages showing both narrow and wide confidence intervals so the model learns what calibrated uncertainty looks like in your domain. If your application cannot parse the full JSON schema, simplify the output to {score, confidence_lower, confidence_upper, rationale} and move factor breakdowns to a separate debugging prompt.

What to do next: After copying this template, run it against a calibration set of 20-30 passages where you have human-scored ground truth. Check whether the model's confidence intervals actually contain the human scores at the expected rate—if you request 90% confidence intervals, roughly 90% of human scores should fall within the bounds. If intervals are systematically too narrow, add examples of ambiguous passages with wide intervals. If intervals are too wide to be useful, tighten the criteria language or reduce the number of factors the model must assess simultaneously. Wire the output into your downstream selection logic so that passages with wide confidence intervals trigger additional retrieval, human review, or conservative answer strategies rather than being treated the same as high-confidence passages.

The Evidence Scoring Prompt with Confidence Intervals is designed to be called as a scoring microservice within a RAG pipeline, not as a one-off chat interaction. It should sit between retrieval and answer generation, receiving a batch of candidate passages and returning calibrated scores with upper and lower confidence bounds. The prompt expects a structured input containing the user query, the retrieved passages with metadata, and the scoring rubric. The output is a JSON array of scored passages, each with a point estimate and a confidence interval. This structured contract makes it straightforward to integrate into an application: call the model, parse the JSON, validate the scores, and pass the ranked list downstream.

Wiring the prompt into an application requires a thin orchestration layer. In pseudocode: scored_passages = model.call(prompt_template.render(query=query, passages=retrieved_passages, rubric=scoring_rubric)). After receiving the response, the harness must validate that every passage in the input set received a score, that all scores fall within the expected range (e.g., 0.0–1.0), and that confidence intervals are well-formed (lower ≤ point ≤ upper). Any passage missing a score or with a malformed interval should trigger a retry with a repair prompt that includes the original output and a specific error message. If the retry also fails, the harness should log the failure and either fall back to a simpler scoring method or escalate for human review, depending on the risk level of the downstream task.

Model choice and tool use matter here. This prompt benefits from models with strong instruction-following and structured output capabilities. If your model provider supports native JSON mode or structured output constraints, enable them to reduce parsing failures. For high-throughput systems, consider batching multiple queries into a single call if the context window permits, but be aware that larger batches can introduce position bias in scoring. The prompt does not require external tools or retrieval during scoring—it operates on the passages already provided. However, you may want to add a calibration check as a post-processing step: compare the model's confidence intervals against empirical score distributions from a held-out evaluation set to detect systematic overconfidence or underconfidence. Log every scored batch with the query, passage IDs, scores, confidence intervals, and model version for auditability and regression testing.

Failure modes to monitor include: the model assigning identical scores to all passages (indicating the rubric is being ignored), confidence intervals that are implausibly narrow on ambiguous passages (overconfidence), and score inversions where clearly irrelevant passages outrank relevant ones. Implement a guardrail that flags batches where the score variance is below a threshold or where the top-ranked passage has a confidence interval wider than a configured maximum. These batches should be routed to a human review queue or a more expensive model for re-evaluation. For high-stakes domains like legal or medical evidence ranking, always require a human-in-the-loop approval step before scored evidence is used to generate answers that users will see.

Next steps after implementation: build an eval harness that runs the prompt against a golden dataset of passage-query pairs with known relevance judgments. Measure rank correlation (e.g., Kendall's tau or NDCG) between model scores and human judgments, and track calibration error between stated confidence intervals and actual error rates. Use these metrics as release gates before promoting a new prompt version to production. Avoid the temptation to treat the confidence intervals as true statistical guarantees—they are the model's expressed uncertainty, not a calibrated measurement, unless you have empirically validated them against your specific data distribution.