Evidence Confidence Calibration Prompt

This prompt is designed for verification system architects and pipeline engineers who need to move beyond binary true/false labels and produce calibrated confidence scores for claim-evidence pairs. The primary job-to-be-done is assigning a numeric confidence level (e.g., 0.0 to 1.0) that reflects the strength of the evidence relative to the claim, accompanied by explicit uncertainty factors that explain why the score is not higher or lower. This is essential for building production verification systems that require triage routing (e.g., auto-verify above 0.9, human review between 0.5 and 0.9, flag as unverified below 0.5), audit-ready outputs that document the reasoning behind each confidence decision, and threshold-based escalation logic. The ideal user is someone who has already built or is building a claim extraction and evidence retrieval pipeline and now needs to add a calibration layer that produces defensible, explainable scores rather than opaque model judgments.

Required upstream context: This prompt assumes you have already completed two prior steps in your verification pipeline. First, discrete, atomic claims have been extracted from source content using a claim extraction prompt. Second, candidate evidence passages have been retrieved from your knowledge base, document store, or provided source set using a retrieval step. This prompt does not perform extraction or retrieval itself—it operates on pre-extracted claims and pre-retrieved evidence. If you feed it raw documents or unextracted text, it will not function correctly. You must also have a defined output schema for confidence scores and uncertainty factors that your downstream systems can consume. The prompt expects structured input containing at minimum a claim text, evidence text, and source identifier for each pair to be scored.

When not to use this prompt: Do not use this prompt if your pipeline only needs a simple ternary label—supported, contradicted, or insufficient evidence. For that use case, a claim-to-evidence pairing prompt with a match type field is more appropriate and less computationally expensive. Do not use this prompt if you lack a calibration dataset of human-annotated claim-evidence pairs with ground-truth confidence scores; without calibration testing, the numeric scores this prompt produces will be unvalidated and potentially misleading. Do not use this prompt for real-time, latency-sensitive applications where the additional reasoning required for uncertainty articulation adds unacceptable delay—consider a simpler classifier in those cases. Finally, do not use this prompt as a substitute for human judgment in high-stakes domains (healthcare, legal, safety-critical systems) without building a human review step that uses the confidence score as a routing signal, not a final decision. The next step after implementing this prompt should be running calibration tests against your annotated dataset to measure whether the model's confidence scores correlate with actual evidence reliability, and adjusting your acceptance thresholds accordingly.

This prompt template instructs the model to act as a verification calibrator. Instead of returning a simple true/false or supported/unsupported label, it produces a structured confidence score accompanied by explicit uncertainty factors. This is critical for production verification pipelines where downstream systems need to make risk-aware decisions—such as auto-publishing, queuing for human review, or discarding—based on the quality and completeness of the evidence, not just its existence.

textCopy
You are an evidence calibration assistant. Your task is to evaluate a claim against provided evidence sources and produce a calibrated confidence score. Do not output a binary label. Instead, assess the degree to which the evidence supports, contradicts, or fails to address the claim, and enumerate the specific factors that contribute to uncertainty.

## CLAIM
[CLAIM]

## EVIDENCE SOURCES
[EVIDENCE_SOURCES]

## OUTPUT SCHEMA
Return a single JSON object with the following keys:
- `claim_summary`: A concise restatement of the claim being evaluated.
- `confidence_score`: A float between 0.0 and 1.0, where 1.0 represents the highest possible confidence that the claim is fully and unambiguously supported by the provided evidence, and 0.0 represents certainty that it is contradicted or entirely unsupported. A score of 0.5 indicates maximum uncertainty.
- `verdict`: One of `supported`, `contradicted`, `insufficient_evidence`, or `partially_supported`.
- `uncertainty_factors`: An array of strings, each describing a specific reason for uncertainty. Reference concrete gaps, ambiguities, or conflicts in the evidence. If confidence is high, this array may be empty.
- `evidence_assessment`: A brief narrative explaining how the evidence sources map to the claim, noting any direct quotes, contradictions, or missing information.

## CONSTRAINTS
- Base your score strictly on the provided evidence. Do not use external knowledge.
- If evidence is missing, conflicting, or vague, your confidence score must decrease and you must document the reason in `uncertainty_factors`.
- Do not hallucinate citations. Only reference sources by their provided identifiers.
- If no evidence is provided, set `confidence_score` to 0.5 and `verdict` to `insufficient_evidence`.

To adapt this template, replace [CLAIM] with the single, atomic assertion you need to verify. Replace [EVIDENCE_SOURCES] with a structured list of your retrieved documents, each with a unique identifier and its full text. For high-risk domains, add a [RISK_LEVEL] placeholder to adjust the strictness of the confidence scoring. After generating the output, always validate the JSON structure and confirm that the confidence_score logically aligns with the verdict and the listed uncertainty_factors. A common failure mode is a high score paired with a long list of unresolved uncertainty factors; your application harness should flag this for review or automatic retry.

The Evidence Confidence Calibration Prompt is not a standalone tool; it is a scoring node in a larger verification pipeline. To use it in production, you must wrap it in a harness that validates the output schema, enforces calibration thresholds, and routes low-confidence or malformed results to a human review queue. The prompt outputs a structured JSON object containing a confidence score, uncertainty factors, and an evidence sufficiency summary. Your harness must parse this output, check that the score falls within the expected 0.0–1.0 range, and confirm that all required fields are present before the result flows downstream to a decision engine or audit log.

A robust implementation begins with a schema validator that rejects any response missing the confidence_score, uncertainty_factors, or evidence_summary keys. Use a JSON Schema definition or a Pydantic model to enforce field types: confidence_score must be a float, uncertainty_factors must be a non-empty array of strings, and evidence_summary must be a string. If validation fails, implement a single retry by re-invoking the prompt with the original inputs and a prepended error message such as 'Your previous output was invalid. Ensure you return a JSON object with the exact fields: confidence_score, uncertainty_factors, evidence_summary.' Do not retry more than once for schema errors; instead, log the failure and route the claim to a human review queue with the raw model output attached. For score-range violations (e.g., a confidence_score of 1.5 or -0.2), apply the same retry-and-escalate pattern.

Calibration thresholds are the core operational decision in this harness. A raw confidence score of 0.7 means nothing without a calibrated threshold that maps scores to actions. Define three bands in your pipeline configuration: auto_accept (e.g., score ≥ 0.85), human_review (e.g., 0.5 ≤ score < 0.85), and auto_reject (e.g., score < 0.5). These thresholds must be tuned against a human-annotated calibration dataset specific to your domain. Run the prompt against a golden set of claims with known ground-truth verification outcomes, then plot a reliability diagram to check whether the model's confidence scores align with actual accuracy. If the model is overconfident (high scores on incorrect verifications), apply Platt scaling or isotonic regression on a held-out calibration set before setting production thresholds. Store calibration parameters in your pipeline configuration, not hardcoded in the prompt.

Logging and observability are non-negotiable for calibration workflows. For every invocation, log the input claim, the retrieved evidence snippets, the raw model response, the parsed confidence score, the uncertainty factors, the validation status, and the routing decision. Include a prompt_version and model_id in each log entry so you can trace score drift when the prompt or model changes. Emit these logs to your observability stack (e.g., a structured logging sink or an evaluation store) and set up a dashboard that tracks the distribution of confidence scores over time, the rate of schema validation failures, and the proportion of claims routed to human review. A sudden shift in the score distribution—such as a spike in 0.9+ scores—often signals prompt drift, evidence contamination, or a model update that requires recalibration.

Human-review routing must carry sufficient context for a reviewer to make a decision efficiently. When a claim falls into the human_review band or fails validation, assemble a review packet containing the original claim text, the retrieved evidence passages with source metadata, the model's raw confidence score and uncertainty factors, and a direct link to the full verification trace. Present this in a review interface that lets the reviewer accept, override, or reject the model's assessment with a single click and an optional note. Log the reviewer's decision as ground truth for future calibration updates. Avoid the trap of routing only low-confidence items to humans; periodically sample and review high-confidence auto-accepted claims to detect silent calibration drift before it causes downstream harm.