Confidence Score Assignment Prompt Template

This prompt is designed for AI engineering teams building agentic systems that must make threshold-based decisions under uncertainty. The core job is to assign a calibrated confidence score to a specific claim, plan step, or piece of extracted information, accompanied by an explicit justification and a list of known limitations. The ideal user is a developer or AI architect integrating this prompt into a planning pre-processor, a fact-verification pipeline, or an agent runtime that needs to decide whether to proceed, escalate, or gather more evidence before committing to an action.

Use this prompt when downstream logic depends on a numeric or categorical confidence signal that a model can produce more reliably than a binary yes/no. It is appropriate for workflows where the cost of a false-positive action (e.g., sending an incorrect customer communication, executing an irreversible database write, or publishing an unverified claim) is high enough to warrant an explicit uncertainty check. The prompt requires a specific [CLAIM] to evaluate, optional [CONTEXT] or evidence sources, and a defined [CONFIDENCE_SCHEMA] that specifies the expected output format, such as a 0.0–1.0 score, a categorical label (HIGH/MEDIUM/LOW), and required justification fields. Do not use this prompt for subjective preference tasks, creative generation where there is no ground truth, or latency-critical paths where the additional reasoning step would violate a strict timeout budget.

Before adopting this prompt, confirm that you have a calibration plan. The prompt includes a structured justification field, but the scores it produces are only useful if you periodically evaluate them against actual outcomes. If you cannot collect ground-truth labels for the claims being scored, the confidence numbers will drift into a stylistic convention rather than a decision-useful signal. In high-risk domains such as healthcare, legal, or finance, always route low-confidence outputs to a human reviewer and log the justification for audit. The next section provides the copy-ready template; after that, the implementation harness covers validation, retries, and calibration eval.

This prompt template is designed for agents and planning modules that need to produce calibrated confidence estimates on individual claims, facts, or execution steps. Unlike a simple high/medium/low tag, this template forces the model to articulate the evidence basis, enumerate specific limitations, and separate the confidence score from the action decision. The output is structured for downstream threshold logic, allowing an application harness to route low-confidence items for human review, trigger additional evidence collection, or halt irreversible actions.

textCopy
You are a confidence scoring engine. Your job is to assign a calibrated confidence score to each provided claim or step, with explicit justification and known limitations. Do not decide whether to proceed. Only produce the confidence annotation.

## INPUT
Claims to score:
[CLAIMS]

Available evidence:
[EVIDENCE]

Assumptions in effect:
[ASSUMPTIONS]

## OUTPUT SCHEMA
Return a JSON object with a "scores" array. Each element must have:
- "claim_id": string matching the input claim identifier
- "confidence": number between 0.0 and 1.0
- "evidence_basis": array of strings citing specific evidence sources used
- "assumptions_relied_on": array of strings listing assumptions this score depends on
- "limitations": array of strings describing what would reduce confidence further
- "unknowns": array of strings listing specific information that, if known, could change the score
- "scoring_rationale": string explaining how the score was derived from evidence and assumptions

## SCORING GUIDELINES
- 0.9-1.0: Multiple corroborating sources, no contradictory evidence, assumptions verified
- 0.7-0.89: Strong evidence but minor gaps or unverified assumptions
- 0.5-0.69: Partial evidence, significant assumptions, or some contradictory signals
- 0.3-0.49: Weak evidence, major assumptions unverified, or substantial contradictions
- 0.0-0.29: No direct evidence, speculation, or evidence contradicts the claim

## CONSTRAINTS
- Do not invent evidence. Only cite sources provided in [EVIDENCE].
- If no evidence exists for a claim, confidence must be ≤ 0.2 and limitations must state "no direct evidence available."
- Do not conflate confidence with importance or urgency.
- Flag any claim where assumptions are doing more work than evidence in the scoring_rationale.
- If a claim involves an irreversible action, note this in limitations even if confidence is high.

## EXAMPLES
[EXAMPLES]

## OUTPUT
Return only the JSON object. No preamble, no commentary.

To adapt this template for your workflow, replace the square-bracket placeholders with real data. [CLAIMS] should be a structured list of claim objects, each with an id and text field. [EVIDENCE] should contain the available source material, ideally with source identifiers that can be cited in evidence_basis. [ASSUMPTIONS] should list the active planning assumptions from your assumption inventory. [EXAMPLES] is critical for calibration: provide 2-3 scored examples that demonstrate the difference between high-confidence claims with strong evidence and low-confidence claims where assumptions dominate. Without examples, models tend to default to middling scores (0.6-0.8) that provide little signal for threshold decisions. After adapting, validate the output against your confidence thresholds before wiring it into execution logic. For high-risk domains, route any claim scoring below 0.7 for human review regardless of the action decision.

The confidence score assignment prompt is not a standalone utility; it is a component that feeds downstream decision logic. In production, the prompt output should be parsed into a structured object containing the claim, the assigned score, the justification, and the known limitations. This object becomes the input to threshold-based routers, escalation triggers, and evidence-gathering loops. The harness must treat the model's confidence estimate as a signal to be calibrated against actual outcomes, not as ground truth. Before integrating this prompt into any agent workflow, define the numeric scale (e.g., 0.0–1.0 or 1–5), the semantic anchors for each score level, and the action thresholds that will consume these scores.

The implementation should wrap the prompt call in a validation layer that checks for required fields (claim_id, confidence_score, justification, limitations), enforces score range constraints, and rejects outputs where the justification is circular or merely restates the claim. If the output fails validation, retry with a repair prompt that includes the specific validation error and requests a corrected response. Log every confidence assignment alongside the model version, prompt version, and any retrieved context used. For high-risk domains—such as healthcare, legal, or financial workflows—route all assignments below a configurable threshold (e.g., <0.7) to a human review queue. The review interface should display the original claim, the model's justification, the evidence available, and a simple accept/override control. Overrides must be recorded and fed back into a calibration dataset.

Calibration is the most important long-term harness concern. Maintain a table of confidence assignments paired with eventual ground-truth outcomes. Periodically run a calibration eval that compares predicted confidence against actual accuracy across score buckets. If the model is systematically overconfident (e.g., assigning 0.9 to claims that are correct only 70% of the time), adjust the system prompt to include calibration examples or apply a post-hoc Platt scaling layer. For agent architectures, wire the confidence output into the planning module so that low-confidence steps trigger evidence-gathering subroutines, clarification requests, or replanning. Avoid the common failure mode of treating confidence scores as static; they should decay when the underlying evidence ages or when new contradictory information arrives. Implement a confidence_staleness check that compares the timestamp of the evidence used against a configurable TTL, and downgrade scores on stale evidence before they reach downstream decision points.