Confidence Score Calibration Prompt Template

Use this prompt when you need to measure how well a model's self-reported confidence scores align with its actual accuracy. This is essential for any production system that relies on model confidence for routing, escalation, or decision thresholds. The prompt takes a set of model outputs, each with a predicted confidence score and a ground-truth correctness label, and returns a calibration analysis including Expected Calibration Error (ECE), a reliability diagram in tabular form, and identification of overconfidence and underconfidence regions. Use this before setting confidence thresholds in production, when auditing model behavior for safety-critical applications, or when comparing calibration quality across model versions. This prompt assumes you already have a labeled dataset with correctness judgments. It does not generate correctness labels or evaluate output quality itself.

This prompt is not a substitute for human evaluation of output quality. It measures the statistical relationship between confidence and correctness, not whether the outputs are good. Do not use it when you lack ground-truth labels, when your dataset is too small to produce meaningful calibration statistics (fewer than 100 samples per bin is a warning sign), or when confidence scores come from different models or prompting strategies that aren't comparable. The prompt also won't help if your confidence scores are binary or categorical rather than continuous probabilities—calibration analysis requires scores that can be binned into probability ranges. For high-risk domains like healthcare or legal applications, always supplement automated calibration analysis with human review of the most confidently wrong outputs.

Before running this prompt, ensure you have a dataset with at least 500 labeled examples spread across the confidence range. Skewed datasets where most examples cluster at high confidence will produce unreliable calibration curves. If your data is limited, consider using temperature scaling or isotonic regression on a held-out calibration set before running this analysis. The output will give you actionable thresholds—for example, 'below 0.7 confidence, accuracy drops below 80%'—that you can wire directly into your production routing logic. After calibration, re-run this analysis periodically as model behavior drifts or when you change prompting strategies.

This prompt template is designed to be dropped directly into your evaluation harness. It instructs the model to act as a calibration analyst, comparing its own confidence scores against ground-truth correctness labels. The output is a structured calibration report that you can parse programmatically to generate reliability diagrams, compute Expected Calibration Error (ECE), and identify regions where the model is overconfident or underconfident. Use this when you have a batch of scored predictions and need to measure how trustworthy those scores are before relying on them in production gating or human handoff decisions.

textCopy
You are a calibration analyst evaluating the reliability of confidence scores produced by an AI system. Your task is to compare predicted confidence scores against actual correctness labels and produce a calibration report.

## INPUT DATA
You will receive a JSON array of prediction records. Each record contains:
- `id`: unique identifier for the prediction
- `confidence`: a float between 0.0 and 1.0 representing the model's self-assessed confidence
- `correct`: a boolean indicating whether the prediction was actually correct (true) or incorrect (false)

[INPUT_DATA]

## OUTPUT SCHEMA
Return a JSON object with the following structure:
{
  "summary": {
    "total_predictions": <integer>,
    "accuracy": <float 0-1>,
    "average_confidence": <float 0-1>,
    "expected_calibration_error": <float 0-1>,
    "ece_interpretation": <string: "well-calibrated" | "moderately miscalibrated" | "severely miscalibrated">
  },
  "bins": [
    {
      "bin_range": "<lower>-<upper>",
      "count": <integer>,
      "average_confidence": <float>,
      "accuracy": <float>,
      "gap": <float>,
      "diagnosis": <string: "well-calibrated" | "overconfident" | "underconfident">
    }
  ],
  "overall_diagnosis": {
    "primary_issue": <string: "overconfidence" | "underconfidence" | "mixed" | "none">,
    "worst_bin": <string>,
    "recalibration_recommendation": <string>
  }
}

## BINNING RULES
- Create exactly 10 equal-width bins: 0.0-0.1, 0.1-0.2, ..., 0.9-1.0
- For each bin, compute: count of predictions, average confidence, accuracy (proportion correct), and gap (average_confidence minus accuracy)
- A positive gap indicates overconfidence; a negative gap indicates underconfidence
- Label bins with |gap| < 0.05 as "well-calibrated", gap >= 0.05 as "overconfident", gap <= -0.05 as "underconfident"

## ECE CALCULATION
Compute Expected Calibration Error as the weighted average of absolute gaps across all bins:
ECE = sum( (count_i / total) * |gap_i| ) for each bin i

## CONSTRAINTS
- Do not invent or modify input data
- If any confidence value is outside [0.0, 1.0], flag it in a `data_quality_issues` field
- If fewer than 50 predictions are provided, include a `low_sample_warning: true` flag
- Round all floats to 4 decimal places
- Return ONLY valid JSON, no markdown fences or commentary

## RISK LEVEL
[RISK_LEVEL]

## ADDITIONAL CONTEXT
[CONTEXT]

To adapt this template, replace [INPUT_DATA] with your actual prediction records serialized as a JSON array. Set [RISK_LEVEL] to guide the urgency of the recalibration recommendation—use values like low, medium, or high depending on whether these confidence scores gate automated actions, trigger human review, or are purely informational. The [CONTEXT] placeholder can include domain-specific notes about acceptable calibration tolerances. After running this prompt, validate the output JSON against the schema before computing downstream metrics. For high-stakes applications, run this calibration check across multiple random seeds and compare ECE stability before trusting a single run. If the sample size is below 200 predictions, treat the per-bin diagnostics as directional rather than definitive.

This prompt is designed to run inside an automated evaluation harness, not as a one-off chat interaction. The typical integration point is a CI/CD pipeline or a scheduled calibration job that runs after a new model version, prompt change, or fine-tuning checkpoint is produced. The harness feeds the prompt a set of model outputs paired with ground-truth correctness labels, collects the calibration analysis, and writes the results to a metrics store for comparison against prior runs.

Input assembly requires two structured arrays: [MODEL_OUTPUTS] containing the text or structured responses the model produced, and [GROUND_TRUTH_LABELS] containing binary correctness indicators (1 for correct, 0 for incorrect). These arrays must be aligned by index. The harness should validate that both arrays have identical length before calling the prompt, and reject any run where lengths mismatch. For high-stakes domains, include a [CONTEXT] field describing the task and acceptable correctness criteria so the prompt can interpret edge cases consistently.

Output validation must parse the returned JSON and verify the presence of required fields: calibration_curve, ece, overconfidence_regions, underconfidence_regions, and reliability_diagram_data. A schema validator should confirm that ece is a float between 0 and 1, that calibration_curve contains bin_edges, bin_centers, accuracy_per_bin, and confidence_per_bin arrays of equal length, and that overconfidence_regions and underconfidence_regions are arrays of objects with confidence_range and severity fields. Failed validation should trigger a single retry with the error message appended to the prompt as [PREVIOUS_ERROR].

Model choice matters for this task. Use a model with strong reasoning and JSON output capabilities. The prompt expects the model to compute calibration statistics internally, so avoid small or quantized models that may produce arithmetic errors. For production pipelines, log the raw prompt, the full response, validation status, and the extracted ece value to your observability platform. Compare ece against a predefined threshold (typically 0.05 for high-reliability systems) and trigger an alert if calibration degrades beyond acceptable limits.

Human review is required when the calibration report shows ece above threshold, when overconfidence_regions contain high-severity entries in safety-critical confidence bands, or when the calibration curve shows non-monotonic behavior (accuracy decreasing as confidence increases). Do not ship a model or prompt change that fails calibration checks without investigating whether the degradation is caused by distribution shift, prompt drift, or model behavior change. Wire the calibration report into your release gate so that failing calibration blocks promotion to production.