Uncertainty Expression from Evidence Prompt Template

This prompt is designed for safety-critical application builders and customer-facing AI teams who need to express calibrated uncertainty based on evidence quality. Use it when your system retrieves evidence, synthesizes an answer, and must communicate confidence to users with specific reasons for uncertainty. This prompt belongs in the post-retrieval, pre-response stage of a RAG pipeline. It takes retrieved evidence passages, the original query, and any generated answer as inputs, then produces a structured confidence statement.

The ideal user is an AI engineer or product developer integrating a confidence layer into a user-facing answer. The required context includes the original user query, a set of retrieved evidence passages, and the draft answer generated from that evidence. The prompt's job is to audit the relationship between the evidence and the answer, producing a confidence score and a plain-language explanation of what factors increased or decreased confidence. This is not a hallucination detector; it does not compare the answer against a ground-truth database. It reasons over the provided evidence only.

Do not use this prompt when evidence is absent entirely; use the Evidence-Based Refusal Explanation Prompt Template instead. Do not use this prompt for raw hallucination detection; pair it with the Hallucination Flagging with Source Evidence Prompt Template for a complete guardrail. Avoid using this prompt when the answer is a simple fact lookup with a single, authoritative source—the overhead of a structured confidence statement adds latency without proportional user value. In high-stakes domains such as healthcare or legal, the output of this prompt should be reviewed by a human before being surfaced to the end user.

This prompt template instructs the model to act as an evidence analyst. It assesses the provided context across three critical dimensions—quality, agreement, and completeness—before producing a final uncertainty statement. The output is designed to be integrated directly into user-facing applications where overconfidence can erode trust or create safety risks. Replace the square-bracket placeholders with your specific query, retrieved evidence, and risk tolerance before use.

textCopy
You are an evidence analyst. Your task is to assess the quality, agreement, and completeness of the provided evidence to answer a user's query. You must then produce a calibrated uncertainty statement.

Follow these steps strictly:
1.  **Evidence Quality Assessment:** Evaluate each piece of evidence for relevance, authority, and recency. Note any credibility concerns.
2.  **Source Agreement Analysis:** Determine if the evidence sources agree, disagree, or address different aspects of the query. Identify any contradictions.
3.  **Information Completeness Check:** Identify specific information gaps between the query and the evidence. State what is missing.
4.  **Uncertainty Statement Generation:** Synthesize the above into a single, clear uncertainty statement. This statement must:
    *   Express a calibrated confidence level (e.g., High, Moderate, Low, Insufficient Evidence).
    *   Explicitly state the primary reasons for uncertainty, referencing the quality, agreement, or completeness findings.
    *   Avoid hedging language that obscures the specific reason for doubt.
    *   If evidence is sufficient and in agreement, state confidence clearly without overstatement.

**Constraints:**
*   Do not answer the user's query directly. Only provide the uncertainty statement.
*   Base your assessment strictly on the provided evidence. Do not use outside knowledge.
*   If no evidence is provided, the confidence level must be "Insufficient Evidence."

**Inputs:**
<User Query>
[USER_QUERY]
</User Query>

<Evidence Set>
[EVIDENCE_SET]
</Evidence Set>

<Risk Profile>
[RISK_PROFILE]
</Risk Profile>

**Output Format:**
Provide your response in a single markdown code block with the following structure:
```json
{
  "evidence_quality_summary": "string",
  "source_agreement_summary": "string",
  "completeness_gaps": ["string"],
  "uncertainty_statement": "string",
  "confidence_level": "High | Moderate | Low | Insufficient Evidence"
}

To adapt this template, start by populating [USER_QUERY] and [EVIDENCE_SET] with your data. The [RISK_PROFILE] placeholder is a critical control lever; use it to describe the cost of being wrong in your specific domain (e.g., "A wrong answer could lead to an incorrect medical dosage. Prefer lower confidence when evidence is not from a peer-reviewed journal."). This instruction directly influences the model's calibration. After running the prompt, validate that the output JSON parses correctly and that the confidence_level field contains only one of the four allowed enum values. For high-risk applications, a human reviewer should confirm that the stated reasons for uncertainty in the uncertainty_statement field are logically consistent with the summaries of quality, agreement, and completeness before the output is shown to an end user.

The Uncertainty Expression from Evidence prompt is not a standalone artifact—it is a component in a safety-critical pipeline. In production, this prompt typically sits after evidence retrieval and ranking, and before the final response is delivered to a user or downstream system. The harness must enforce that the model never expresses confidence without grounding it in the provided evidence. This means the application layer must validate that every confidence statement references specific source attributes (agreement, completeness, recency) rather than allowing the model to generate vague or unsubstantiated uncertainty language.

Start by defining a strict output schema for the confidence statement. Require a JSON object with fields like confidence_level (enum: high, medium, low, insufficient_evidence), confidence_rationale (array of reasons, each keyed to a specific evidence factor), and uncertainty_factors (array of named factors with severity scores). Validate this schema immediately after generation. If the model outputs free-text without structured fields, reject the response and retry with a stronger format constraint. For high-risk domains such as clinical or legal applications, add a human review trigger when confidence_level is low or insufficient_evidence, or when uncertainty_factors contains items with severity above a configurable threshold. Log every confidence output alongside the evidence set, retrieval metadata, and the final decision (delivered, escalated, or rewritten) for auditability.

Retry logic should be conservative. If validation fails due to missing fields or unparseable JSON, retry once with the same evidence but a reinforced instruction emphasizing the required schema. If the model produces a confidence statement that contradicts the evidence—such as claiming high confidence when sources conflict—do not retry. Instead, flag the output for review and log the contradiction. Model choice matters here: use a model with strong instruction-following and structured output capabilities (such as GPT-4o or Claude 3.5 Sonnet with tool-calling mode) rather than a smaller model prone to format drift. For latency-sensitive applications, consider running the uncertainty prompt in parallel with the answer generation prompt, then merging results before user delivery. Always version your prompts and track which prompt version produced each confidence output so that regressions can be traced during incident review.