Confidence Score Annotation Prompt for Retrieved Passages

This prompt is for RAG pipeline engineers who need a calibrated, per-passage confidence signal before answer generation. It instructs the model to act as an evidence annotator, producing a numerical relevance score and a structured justification for each retrieved passage against a user query. Use this when downstream answer generation, evidence ranking, or refusal logic depends on knowing which passages are strong enough to trust. This is not a prompt for generating the final answer. It is a pre-answer filtering and scoring step that makes evidence quality explicit and auditable.

Deploy this prompt when you need to implement a gating mechanism that prevents low-quality passages from reaching your answer generator. It is especially valuable in systems where retrieval returns noisy or mixed-quality results, where you must log evidence quality decisions for audit trails, or where you need to calibrate refusal thresholds based on passage-level scores. The prompt works best with capable instruction-following models (such as GPT-4, Claude 3.5 Sonnet, or Gemini 1.5 Pro) that can produce consistent numerical judgments when given clear scoring rubrics. For high-throughput pipelines, consider batching multiple passage-query pairs in a single request to reduce latency, but keep batches small enough that the model does not lose scoring consistency across passages.

Do not use this prompt when retrieval already returns highly filtered, top-k results from a dense embedding search with a tight similarity threshold. In those cases, a simpler relevance binary classifier or a threshold on the retrieval score may suffice. Avoid this prompt when latency budgets are extremely tight (sub-200ms) and you cannot afford an extra model inference step before answer generation. For regulated or high-stakes domains, always pair this prompt with human review of the scoring decisions, especially for passages near the accept/reject boundary. The next section provides the copy-ready prompt template you can adapt for your own retrieval pipeline.

This prompt template is the core instruction set for a confidence score annotator. It is designed to be placed directly into your RAG pipeline, right after retrieval and before answer generation. The prompt instructs the model to evaluate each passage independently against the user's query, producing a structured JSON object that includes a numerical score, a justification, and a flag for sufficiency. The strict JSON output contract ensures that downstream systems—such as evidence gates, answer generators, or monitoring dashboards—can parse the results reliably without fragile regex or unstructured text parsing.

textCopy
You are a strict evidence evaluator. Your task is to assess the relevance and sufficiency of each provided passage for answering the user's query. You must output a single, valid JSON object with no additional text, markdown fences, or commentary.

## User Query
[USER_QUERY]

## Passages to Evaluate
[PASSAGES_ARRAY]

## Output Schema
Return a JSON object with the following structure:
{
  "passages": [
    {
      "passage_id": "string",
      "relevance_score": number (0.0 to 1.0),
      "sufficiency_flag": "sufficient" | "partial" | "insufficient",
      "justification": "string explaining the score, referencing specific content from the passage and the query."
    }
  ]
}

## Scoring Rubric
- **1.0 (Perfectly Relevant):** The passage directly answers the core question with specific, unambiguous facts.
- **0.7-0.9 (Highly Relevant):** The passage contains key information but may require minor inference or combination with another passage.
- **0.4-0.6 (Partially Relevant):** The passage touches on the topic but lacks the specific details needed to answer the query.
- **0.1-0.3 (Tangentially Relevant):** Only a keyword or general domain overlaps; the passage does not address the user's intent.
- **0.0 (Not Relevant):** The passage is completely unrelated to the query.

## Sufficiency Flag Rules
- **sufficient:** The passage alone contains enough information to fully answer the query.
- **partial:** The passage provides some useful information but is not enough on its own.
- **insufficient:** The passage does not provide any actionable information to answer the query.

## Constraints
- Do not fabricate information. The justification must be grounded in the passage text.
- If a passage is empty or contains only garbled text, assign a relevance_score of 0.0 and a sufficiency_flag of "insufficient".
- Output only the JSON object. No other text.

To adapt this template for your application, replace the [USER_QUERY] and [PASSAGES_ARRAY] placeholders at runtime. The [PASSAGES_ARRAY] should be a serialized JSON array of objects, each with at least an id and content field. For high-stakes domains, consider adding a [RISK_LEVEL] placeholder to adjust the strictness of the rubric, and always log the raw prompt and model output for auditability. Before deploying, run this prompt against a golden dataset of 50-100 passage-query pairs with human-annotated scores to calibrate the model's interpretation of the rubric and identify systematic biases, such as over-scoring long passages or under-scoring contradictory ones.

The confidence score annotation prompt is not a standalone tool; it is a pre-generation gate that should sit between your retrieval step and your answer generation step. In a typical RAG pipeline, you retrieve N passages, then you must decide which passages are strong enough to use. This prompt acts as a scoring filter: it takes each passage, the user query, and your evidence standards, and returns a structured confidence score with justification. The output is consumed programmatically—your application reads the scores, applies a threshold (e.g., discard passages below 0.6), and passes only the surviving passages to the answer generation prompt. This means the prompt must return machine-parseable JSON, not free-text commentary, and your harness must validate that structure before the pipeline continues.

Wiring the prompt into your application requires a few concrete components. First, define a strict output schema in your prompt (e.g., {"passage_id": string, "confidence_score": float, "justification": string, "sufficiency_flag": "sufficient"|"insufficient"|"partial"}) and enforce it with a JSON validator after the model responds. If validation fails, implement a retry loop (maximum 2 retries) that feeds the validation error back to the model with a repair instruction. Second, log every score with the passage ID, query, timestamp, and model version—this is essential for calibration analysis later. Third, if your use case is high-stakes (healthcare, legal, finance), insert a human review step for passages that fall in a borderline range (e.g., 0.4–0.7) rather than auto-discarding them. Fourth, choose a model that follows structured output instructions reliably; smaller or older models may drift into narrative responses, so test with your actual retrieval outputs before deploying.

Calibration is the long-term maintenance task. After deployment, periodically sample scored passages and have human annotators assign their own confidence scores. Compare the model's scores to human judgments using metrics like Expected Calibration Error (ECE) or simple bucket accuracy. If the model consistently overestimates confidence (a common failure mode), adjust your threshold upward or add calibration instructions to the prompt (e.g., 'prefer lower scores when evidence is indirect'). Also monitor for score drift when you change retrieval sources, update the knowledge base, or switch model versions. A passage that scored 0.8 last month may score 0.5 today if the retrieval index has shifted. Build a dashboard that tracks score distributions over time and alerts you when the mean confidence shifts by more than a standard deviation. Finally, avoid the temptation to skip this annotation step and feed all retrieved passages directly to answer generation—that path leads to hallucination from irrelevant context, which is exactly what this prompt is designed to prevent.