Confidence Score Calibration Prompt Template

This prompt is designed for ML engineers and AI reliability teams who need a model's stated confidence to reflect its actual probability of being correct. Raw model confidence scores are often overconfident or poorly scaled, making them unreliable for downstream decision gates. The primary job-to-be-done is transforming a raw, uncalibrated model output into a trustworthy probability estimate by prompting the model to self-assess its reliability against a provided set of ground-truth calibration examples. Use this when you need confidence scores you can trust for automated escalation thresholds, uncertainty propagation in multi-step agent pipelines, or selective human review routing.

The ideal user has a held-out calibration set with known correct answers and needs to integrate a calibration step into an inference pipeline. This prompt assumes you are working in a batch or near-real-time setting where the additional round-trip latency for calibration is acceptable. It is not suitable for real-time, latency-sensitive applications where the calibration step adds unacceptable delay. You should also avoid this prompt when the calibration set is not representative of the production input distribution, as the model's self-assessment will be miscalibrated for out-of-distribution data. The prompt works best with instruction-following models that can process structured examples and produce a calibrated probability alongside diagnostic metrics.

Before using this prompt, ensure you have a representative calibration set of at least 50-100 examples with verified ground-truth labels. The prompt template expects you to provide these examples inline, so consider the context window budget. For production deployment, pair this prompt with a validation step that checks the output schema and logs the calibration diagnostics for drift monitoring. If the calibration error exceeds your tolerance, escalate to a human review queue or trigger a model retraining workflow rather than silently accepting miscalibrated scores.

This prompt template instructs the model to self-assess its confidence on a set of provided examples where the correct answer is known. By comparing the model's self-assigned probability to actual outcomes, you can measure calibration error and detect drift before deploying the prompt in a production decision gate. The template is designed to be run as a batch evaluation, not a single-turn chat.

textCopy
You are an expert calibration analyst. Your task is to review a set of completed tasks, each with a known correct answer. For each task, you must output a calibrated confidence score representing the probability that your own answer would be correct if you were to answer a similar unseen question.

## INPUT DATA
You will receive a list of examples. Each example contains:
- [EXAMPLE_ID]: A unique identifier for the example.
- [INPUT_QUERY]: The original user query or task input.
- [MODEL_OUTPUT]: The final answer the model produced for this query.
- [GROUND_TRUTH]: The known correct answer.

## CALIBRATION INSTRUCTIONS
For each example, analyze the query, your output, and the ground truth. Then, assign a confidence score from 0.0 to 1.0 based on this rubric:
- 0.0-0.2: The model output is factually incorrect or completely misaligned with the ground truth.
- 0.3-0.5: The output is partially correct but contains a significant error or omission.
- 0.6-0.8: The output is mostly correct with minor imprecision.
- 0.9-1.0: The output is fully correct and precisely matches the ground truth.

## CONSTRAINTS
- Do not use the ground truth to *change* the model output; only use it to *score* the output.
- Provide a brief, one-sentence justification for each score.
- If the model output was an abstention or refusal, score it as 0.0 and note the reason.

## OUTPUT SCHEMA
Return a valid JSON object with a single key "calibration_results" containing an array of objects. Each object must conform to this schema:
{
  "example_id": "string",
  "confidence_score": number (0.0 to 1.0),
  "is_correct": boolean,
  "justification": "string"
}

## EXAMPLES TO CALIBRATE
[EXAMPLES]

To adapt this template, replace the [EXAMPLES] placeholder with your batch of scored examples. For a production implementation, run this prompt against a held-out calibration dataset that mirrors your real traffic distribution. After collecting the confidence_score and is_correct fields, compute Expected Calibration Error (ECE) by binning predictions and comparing average confidence to actual accuracy within each bin. If ECE exceeds your threshold, the model is miscalibrated for this task, and you should not use its raw confidence scores to trigger automated actions without adjustment. For high-risk domains, always route decisions below a confidence of 0.9 to a human review queue, regardless of the calibration metrics.

The calibration prompt is not a one-shot evaluation tool; it is a component inside a continuous calibration loop. In production, you will call this prompt periodically—after model updates, data distribution shifts, or on a fixed schedule—using a held-out calibration dataset that the model has not seen during training. The prompt expects [GROUND_TRUTH_EXAMPLES] containing input-output pairs with known correct answers, and [TARGET_OUTPUTS] representing recent model responses that need confidence scores. The harness should log every calibration run's inputs, raw outputs, and parsed scores to enable drift analysis over time.

Validation and Parsing: The prompt instructs the model to return a JSON object with a confidence_scores array. Your harness must validate this structure immediately. Each element in the array must contain a score (float between 0.0 and 1.0), an is_calibrated boolean, and a calibration_error float. Reject any response where scores fall outside [0,1], where the array length mismatches the number of target outputs, or where required fields are missing. On validation failure, implement a single retry with the error message appended to the [CONSTRAINTS] field—do not retry more than once for calibration tasks, as a second failure indicates a deeper prompt or model issue that requires human review. Expected Calibration Error (ECE) Calculation: After parsing, compute ECE by binning predictions (10 equal-width bins is standard), calculating the difference between average confidence and accuracy per bin, and weighting by bin size. Log this metric alongside the raw scores.

Model Choice and Tool Integration: This prompt is designed for frontier models with strong instruction-following and structured output capabilities (GPT-4o, Claude 3.5 Sonnet, or equivalent). Do not use smaller or older models for calibration, as their self-assessment reliability is significantly lower. Wire the prompt into your MLOps pipeline as a post-inference evaluation step: after your primary model generates outputs, route a sample to this calibration prompt. If calibration error exceeds a predefined threshold (start with ECE > 0.1 as a warning, > 0.2 as critical), trigger an alert and block the model from serving high-stakes decisions until recalibration or human review occurs. For low-risk applications, you may use the calibration scores directly to adjust downstream decision thresholds without blocking.

Failure Modes and Monitoring: The most common production failure is calibration drift, where the model's self-assessed confidence becomes increasingly misaligned with actual accuracy over time. Monitor ECE trends, not just point values. A sudden ECE spike often indicates a data distribution shift or a model hotfix that altered behavior. Also watch for score collapse, where the model assigns nearly identical confidence to all outputs (e.g., everything at 0.85), which renders calibration useless. Log every calibration run's ECE, per-bin accuracy, and the raw prompt-response pair to your observability platform. If you use this prompt in a regulated domain, retain these logs as audit evidence and require human sign-off when calibration thresholds are breached before the primary model can resume autonomous operation.