Confidence Calibration with Evidence Weighting Prompt Template

This prompt is designed for verification system architects and pipeline engineers who need to assign calibrated confidence scores to individual claims. It is intended for production pipelines where binary true/false labels are insufficient and where downstream routing decisions—such as auto-verify, human review, or reject—depend on nuanced confidence signals. The ideal user is someone integrating this prompt into a multi-step verification workflow, specifically after claim extraction and evidence retrieval are complete. The prompt expects structured evidence packets with source metadata and produces a confidence score with explicit evidence weights, source authority factors, and an uncertainty interval.

Do not use this prompt for real-time chat or conversational fact-checking where latency and context windows preclude structured evidence input. It is also unsuitable for scenarios where evidence has not yet been retrieved or where claims have not been decomposed into atomic, verifiable units. The prompt requires a specific input shape: a single claim string, a list of evidence objects each containing the evidence text, source identifier, source authority score, and publication date. If your pipeline cannot provide this structured input, you should first implement claim extraction and evidence matching prompts before attempting confidence calibration. Using this prompt with unstructured or incomplete evidence will produce unreliable confidence scores and misleading uncertainty intervals.

Before deploying this prompt, ensure you have a calibration set of human-labeled verification outcomes to validate the model's confidence scores against. The prompt includes instructions for the model to self-assess its calibration, but this is not a substitute for offline evaluation. Common failure modes include overconfidence on claims with superficially strong but shallow evidence, underconfidence when evidence is technically sufficient but poorly written, and miscalibration when source authority scores are inflated. Wire the output into a validation layer that checks for score range compliance, interval coherence, and evidence weight summation before routing decisions are made. If the claim domain is high-risk (legal, medical, financial), always route outputs through human review regardless of the confidence score.

This template is the core instruction set for a verification model. It forces the model to decompose its confidence decision into explicit factors—evidence weight, source authority, corroboration count, and contradiction presence—before producing a final score and an uncertainty interval. Replace every square-bracket placeholder with the actual values for your verification run before sending the prompt to the model. The template is designed to be stateless and idempotent: given the same inputs, it should produce a structurally identical output that can be validated programmatically.

textCopy
You are a claim verification specialist. Your task is to assign a calibrated confidence score to a single claim based on the provided evidence set. You must not rely on any knowledge outside the provided evidence. Produce a structured output that explains your confidence decision before stating the final score.

## CLAIM TO VERIFY
[CLAIM_TEXT]

## EVIDENCE SET
[EVIDENCE_ITEMS]

## SOURCE AUTHORITY CONTEXT
[SOURCE_AUTHORITY_MAP]

## OUTPUT SCHEMA
Return a single JSON object with the following fields:
- "claim_id": "[CLAIM_ID]"
- "confidence_score": number between 0.0 and 1.0
- "confidence_interval": {"lower": number, "upper": number}
- "evidence_weights": [{"evidence_id": string, "weight": number, "rationale": string}]
- "source_authority_factors": [{"source_id": string, "authority_score": number, "domain_relevance": string}]
- "corroboration_count": number
- "contradiction_flags": [{"evidence_id": string, "severity": "direct"|"partial"|"temporal"|"none", "description": string}]
- "uncertainty_rationale": string
- "calibration_note": string

## CONSTRAINTS
- Base confidence on evidence weights, source authority, corroboration, and contradictions.
- If no evidence supports the claim, confidence_score must be 0.0.
- If evidence is mixed, widen the confidence_interval proportionally.
- Do not invent evidence. If evidence is missing, state it in uncertainty_rationale.
- For high-risk domains ([RISK_LEVEL]), flag low-confidence scores for human review.

## EXAMPLES
[FEW_SHOT_EXAMPLES]

## INSTRUCTIONS
1. Weigh each evidence item for relevance and reliability.
2. Assess source authority per the provided authority map.
3. Count independent corroborating sources; discount redundant citations.
4. Identify and classify any contradictions.
5. Compute a confidence_score and a justified confidence_interval.
6. Explain your uncertainty and any calibration concerns.
7. Output only the JSON object. No markdown, no preamble.

Adaptation guidance: The [EVIDENCE_ITEMS] placeholder should contain a pre-retrieved, deduplicated list of evidence passages with unique IDs. The [SOURCE_AUTHORITY_MAP] should be a pre-computed mapping of source IDs to authority scores and domain relevance tags—do not ask the model to assess authority from scratch unless you have no prior authority model. The [FEW_SHOT_EXAMPLES] placeholder is critical for score calibration; include at least three examples spanning high-confidence, mixed-evidence, and no-evidence scenarios, each with the exact output schema. If you omit examples, expect score drift. For production pipelines, post-process the output with a JSON schema validator and a confidence interval sanity check (lower ≤ score ≤ upper). If the validator fails, retry once with the error message appended to the prompt. If [RISK_LEVEL] is "high" or "critical", route any output with confidence_score < 0.7 or contradiction_flags containing "direct" severity to a human review queue before the score is used downstream.

This prompt is designed to be a single step within a larger claim verification pipeline. It should not be called in isolation. The upstream system must first extract discrete, atomic claims from source content and retrieve a candidate evidence set for each claim. The output of this prompt—a calibrated confidence score with evidence weights, source authority factors, and an uncertainty interval—feeds into downstream routing logic that decides whether to auto-verify, escalate for human review, or reject the claim as unverifiable. Treat this prompt as a scoring microservice: it receives a structured claim-evidence pair and returns a structured score, nothing more.

Input Contract: The application layer must assemble the following variables before calling the model: [CLAIM_TEXT] (the exact claim string, not a summary), [CLAIM_DOMAIN] (a controlled vocabulary label like financial, medical, scientific, legal, news), [EVIDENCE_LIST] (a JSON array of evidence objects, each with source_id, source_text, source_type, publication_date, and authority_metadata), [CORROBORATION_COUNT] (integer), [CONTRADICTION_FLAGS] (array of contradiction descriptions if any), and [CALIBRATION_REFERENCE] (optional historical accuracy data for this domain and claim type). Output Contract: Parse the model response into a typed object with fields: confidence_score (float 0.0–1.0), confidence_interval_lower (float), confidence_interval_upper (float), evidence_weight_breakdown (array of objects mapping each evidence item to its weight and rationale), source_authority_factors (object with domain_relevance, recency, and credential_justification), uncertainty_drivers (array of strings explaining what lowered confidence), and calibration_note (string describing alignment with reference data if provided). Use a strict JSON schema validator on the model output before accepting it. If parsing fails, retry once with the validation error included in the retry prompt.

Model Choice and Latency: This prompt benefits from models with strong reasoning and calibration capabilities. GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro are suitable defaults. For high-throughput pipelines, consider routing low-complexity claims (single evidence source, no contradictions) to a faster model like Claude 3 Haiku or GPT-4o-mini, reserving the full prompt for multi-source, contradictory, or high-stakes claims. Retry and Fallback Logic: If the model returns a malformed response, retry once with the original prompt plus the parse error. If the second attempt fails, log the failure, flag the claim for human review, and emit a default low-confidence score (0.0) with an uncertainty_drivers entry of `

scoring_failure

.

Do not silently accept unparseable scores. Human Review Integration: When the output confidence_score falls below your configured threshold (start with 0.7 and tune based on domain risk tolerance)

route the claim to a human review queue. Package the full prompt input

the model's scored output

and the evidence list into a review packet. The reviewer should see exactly what the model saw

plus the model's reasoning

to make an efficient override decision.

Logging and Audit Trail: Log every scoring call with: claim_id, prompt_version, model_id, timestamp, input_evidence_ids, output_confidence_score, output_confidence_interval, parse_success (boolean), retry_count, and routing_decision. This log becomes the foundation for calibration analysis. Periodically compare scored confidence intervals against actual human verification outcomes to detect drift. If the model's confidence intervals consistently fail to contain the true outcome rate (e.g., 90% confidence intervals that contain the correct label only 70% of the time), the prompt needs recalibration—adjust the [CALIBRATION_REFERENCE] input with updated historical accuracy data or add few-shot examples of well-calibrated scores. What to Avoid: Do not use this prompt for claims where no evidence was retrieved—route those to the Unsupported Statement Flagging prompt instead. Do not bypass the output validator; unvalidated scores corrupt downstream routing. Do not treat the confidence score as a probability in the strict statistical sense without calibration evidence; treat it as a calibrated signal for triage decisions.