Answer Factuality Confidence Scoring Prompt

This prompt is designed for product teams that need to surface calibrated uncertainty to end-users, not just detect hallucinations internally. The job-to-be-done is assigning a defensible confidence score to a generated answer based on how well the provided evidence covers and consistently supports its claims. The ideal user is an AI engineer or product manager integrating this as a post-generation step in a RAG pipeline, where the output—a score and a human-readable explanation—is rendered alongside the answer in a user interface. You need the original user question, the final generated answer, and the complete set of retrieved source passages that were available during generation. Without all three inputs, the prompt cannot assess coverage gaps or cross-reference claims against evidence.

Use this prompt when you want to display a confidence indicator (e.g., 'High confidence,' 'Partial support,' 'Low evidence') to help users calibrate their trust. It is appropriate for customer-facing Q&A, internal knowledge base search, and any product surface where a wrong answer carries a non-trivial cost. The prompt works best after you have already run a retrieval step and generated an answer; it is a verification layer, not a replacement for faithful synthesis prompts or citation checks. Do not use this prompt as your sole hallucination guardrail in high-stakes domains like healthcare or legal without additional human review. It estimates confidence from evidence coverage and consistency, but it cannot detect subtle fabrications that sound plausible and align with the source material's tone without actually being stated.

This prompt is not a fact-checker that validates claims against ground truth—it only compares the answer to the provided context. If your retrieval step returns incomplete or irrelevant passages, the confidence score will reflect that gap, but it cannot flag that the retrieval itself failed to find the right document. For that, pair this prompt with an evidence sufficiency gate that runs before answer generation. Also avoid using this prompt for real-time streaming answers where the full text is not yet available; it requires the complete answer to assess overall evidence coverage. If your product needs sentence-level grounding or claim-by-claim verification, use the Sentence-Level Grounding Check Prompt or the Claim-by-Claim Source Verification Prompt instead—this prompt operates at the whole-answer level and is optimized for user-facing confidence display, not granular debugging.

Before wiring this into production, define what score thresholds trigger different UI treatments (e.g., scores below 0.7 get a warning icon, below 0.4 get a 'could not verify' label). Run the prompt against a golden dataset of answers with known evidence relationships to calibrate your thresholds. Expect the model to occasionally produce scores that feel directionally correct but numerically inconsistent across similar cases—log the raw scores and explanations over time to tune your display logic rather than treating the number as a precise measurement. If the confidence explanation contains hedging language that would confuse users, add a post-processing step that maps the explanation to a predefined set of user-facing confidence labels.

This template is the core instruction set for an LLM-based confidence scorer. It takes a user question, a generated answer, and the source evidence used to produce that answer, then outputs a calibrated confidence score and a human-readable explanation. The prompt is designed to be model-agnostic and can be dropped directly into an evaluation pipeline, a pre-release gate, or a user-facing uncertainty display system. All dynamic inputs are represented as square-bracket placeholders that your application must populate at runtime.

textCopy
You are an expert factuality auditor. Your job is to assess how well a generated answer is supported by the provided source evidence and assign a calibrated confidence score.

## INPUTS

**User Question:**
[USER_QUESTION]

**Generated Answer:**
[GENERATED_ANSWER]

**Source Evidence:**
[SOURCE_EVIDENCE]

## TASK

1. Extract every factual claim from the Generated Answer.
2. For each claim, determine whether it is:
   - **Fully Supported**: The evidence directly confirms the claim.
   - **Partially Supported**: The evidence supports some aspects but not all, or the support is implicit.
   - **Unsupported**: The evidence does not contain information to confirm or deny the claim.
   - **Contradicted**: The evidence directly conflicts with the claim.
3. Assign an overall confidence score from 0.0 to 1.0 based on the following scale:
   - **0.9-1.0**: All claims fully supported; answer is a faithful synthesis of the evidence.
   - **0.7-0.89**: Most claims fully supported; minor details are partially supported or implicit.
   - **0.5-0.69**: Mixed support; some claims unsupported but none contradicted.
   - **0.3-0.49**: Significant unsupported claims; answer may be misleading.
   - **0.1-0.29**: Mostly unsupported or contains contradictions.
   - **0.0**: Completely fabricated or directly contradicted by evidence.
4. Write a confidence explanation that:
   - States the score.
   - Summarizes the evidence coverage (e.g., "3 of 5 claims fully supported").
   - Flags any unsupported or contradicted claims with specific examples.
   - Is suitable for display to an end user.

## CONSTRAINTS

- Do not introduce facts not present in the Source Evidence.
- If the Source Evidence is empty or irrelevant, the score must be 0.0.
- If the Generated Answer is empty or a refusal, the score is 1.0 with the explanation "No factual claims to verify."
- Be strict: partial support is not full support.

## OUTPUT FORMAT

Return a single JSON object with the following schema:

{
  "confidence_score": <float between 0.0 and 1.0>,
  "confidence_explanation": "<string suitable for end-user display>",
  "claim_verdicts": [
    {
      "claim": "<exact text of the claim>",
      "verdict": "fully_supported | partially_supported | unsupported | contradicted",
      "evidence_span": "<relevant excerpt from Source Evidence or null>"
    }
  ]
}

## RISK LEVEL

[RISK_LEVEL]

## EXAMPLES

[EXAMPLES]

To adapt this template, start by populating the required inputs. [USER_QUESTION] and [GENERATED_ANSWER] come from your RAG pipeline. [SOURCE_EVIDENCE] should be the exact retrieved passages used during generation, concatenated with document identifiers or chunk IDs for traceability. The [RISK_LEVEL] placeholder lets you inject domain-specific instructions—for example, in a healthcare context you might add: "If any claim is unsupported or contradicted, the confidence score must not exceed 0.3 and the explanation must recommend human review." The [EXAMPLES] placeholder is where you insert few-shot demonstrations of correct scoring behavior, which is critical for calibrating the model's judgment. Without examples, models tend to be overly generous with partial-support scores. Include at least one example where the answer is fully supported, one where it is partially supported, and one where it is contradicted.

Before shipping this prompt, validate that the output JSON matches the schema exactly. A common failure mode is the model producing a confidence_score as a string instead of a float, or omitting the claim_verdicts array when the answer is short. Add a post-processing validator that parses the JSON, checks field types, and clamps the score to the 0.0–1.0 range. For high-stakes deployments, run this prompt against a golden dataset of scored answer-evidence pairs and measure agreement with human raters. If agreement drops below your threshold, adjust the scoring scale descriptions or add more few-shot examples. Never surface a confidence score to users without first verifying that the model's calibration holds on your specific data distribution.

This prompt is designed to sit as a post-generation quality gate in a RAG pipeline, after an answer has been synthesized but before it reaches the end user. The harness should call the model with the generated answer, the retrieved context passages, and the original user query. The primary output is a structured JSON object containing a confidence_score (typically a float between 0.0 and 1.0 or a categorical label like HIGH/MEDIUM/LOW) and a confidence_explanation string suitable for display. The implementation must treat this as a synchronous or asynchronous validation step whose latency adds directly to the user's perceived response time, so model choice and timeout handling are critical.

Wire the prompt into your application by first constructing the input payload: the [GENERATED_ANSWER], the [RETRIEVED_CONTEXT] (as a list of passage objects with id and text fields), and the [USER_QUERY]. Use a fast, instruction-tuned model for this scoring task—GPT-4o-mini, Claude 3.5 Haiku, or a fine-tuned small model—since the prompt is a classification and explanation task, not a generation task. Implement a strict JSON schema validator on the response. If the model fails to return valid JSON matching the expected {confidence_score, confidence_explanation} schema, retry once with a stronger format constraint appended to the prompt. If the retry also fails, log the raw output and default to a conservative low-confidence score with a generic explanation to avoid blocking the pipeline.

Build a decision rule around the returned confidence_score. For example, if the score is below a defined threshold (e.g., 0.7 or MEDIUM), route the answer to a human review queue instead of delivering it directly. If the score is critically low (e.g., below 0.4 or LOW), suppress the answer entirely and return a 'not enough evidence' message to the user. Log every scoring result—including the input answer, context, score, explanation, and final routing decision—to an observability platform for offline analysis. This log becomes your dataset for calibrating the threshold over time. Avoid using this prompt on answers that are purely conversational, chitchat, or procedural instructions where factuality scoring is meaningless; gate the scoring step behind a classifier that only triggers it for factual, evidence-backed responses.