Confidence Calibration Check Prompt

This prompt is designed for MLOps engineers and AI reliability teams who need to validate that a model's confidence scores are calibrated—meaning a reported 90% confidence should translate to roughly 90% accuracy. The job-to-be-done is a pre-deployment calibration audit. You feed the prompt a batch of test cases where you already know the ground-truth outcome, along with the model's predicted class and its raw confidence score for each case. The prompt produces a structured calibration report that compares stated confidence to observed accuracy across the entire batch, flags regions where the model is systematically overconfident or underconfident, and computes standard calibration error metrics like Expected Calibration Error (ECE). This belongs in your evaluation pipeline immediately after fine-tuning, before a new model version is promoted to production, or as a recurring monitoring check to detect confidence drift over time.

Do not use this prompt for real-time inference gating or as a live guardrail. It is a batch audit tool, not a low-latency decision point. The required inputs are a dataset with at least several hundred examples to produce statistically meaningful calibration curves, each containing the model's input, the predicted class, the confidence score for that prediction, and the ground-truth label. The output is a report, not a binary pass/fail signal. You should expect to receive binned accuracy charts, overconfidence and underconfidence flags per confidence decile, and an ECE score. Wire this into a CI/CD pipeline for models where confidence scores drive downstream actions—such as routing, escalation, or autonomous decisions—so that a miscalibrated model is caught before it causes harm. If your use case involves high-stakes decisions in regulated domains, the calibration report should be reviewed by a human before the model is approved for production, and the report itself should be archived as audit evidence.

After running this prompt, compare the ECE score against your team's pre-defined threshold. If the model is overconfident in critical confidence ranges, you may need to apply post-hoc calibration techniques like Platt scaling or isotonic regression before deployment. If underconfidence is the issue, investigate whether your test set contains distribution shift or ambiguous examples that the model correctly hedges on. The next step is to integrate this prompt into your model release checklist and pair it with a confidence threshold decision prompt to define the operational thresholds your system will use in production.

This section provides the core prompt template for the Confidence Calibration Check. It is designed to be pasted directly into your evaluation harness. The prompt instructs the model to act as an MLOps calibration auditor, taking a batch of test cases—each with an input, the model's output, its self-assessed confidence score, and the ground-truth label—and producing a structured diagnostic report. The report's primary job is to surface miscalibration: regions where the model is systematically overconfident or underconfident, quantified by standard metrics like Expected Calibration Error (ECE).

textCopy
You are an MLOps calibration auditor. Your task is to analyze a batch of test cases and produce a calibration report comparing the model's stated confidence scores to its actual correctness.

## INPUT DATA
You will receive a JSON array of test cases. Each object has the following fields:
- `id`: a unique identifier for the test case.
- `input`: the original prompt or input text given to the model.
- `model_output`: the final text output produced by the model.
- `confidence_score`: a float between 0.0 and 1.0 representing the model's self-assessed confidence in its `model_output`.
- `ground_truth`: the correct or expected output for the given `input`.

[TEST_CASES]

## TASK
1. For each test case, determine if the `model_output` is semantically correct compared to the `ground_truth`. Set a boolean `is_correct` field.
2. Group the test cases into [NUM_BINS] confidence bins: [0.0-0.1), [0.1-0.2), ..., [0.9-1.0].
3. For each bin, calculate:
   - `count`: number of test cases in the bin.
   - `accuracy`: the proportion of correct test cases within the bin.
   - `avg_confidence`: the average `confidence_score` of test cases in the bin.
   - `gap`: `avg_confidence` minus `accuracy`. A positive gap indicates overconfidence; a negative gap indicates underconfidence.
4. Calculate the overall Expected Calibration Error (ECE) as the weighted average of the absolute `gap` for each bin, weighted by `count`.
5. Flag any bins where the absolute `gap` exceeds [OVERCONFIDENCE_THRESHOLD] as an "overconfident region" and any bins where the absolute `gap` exceeds [UNDERCONFIDENCE_THRESHOLD] as an "underconfident region".
6. Provide a brief diagnostic summary explaining the primary calibration failure modes observed.

## OUTPUT FORMAT
Return your analysis as a single valid JSON object with this exact schema:
{
  "ece": <float>,
  "bins": [
    {
      "bin_range": "<string, e.g., 0.5-0.6>",
      "count": <int>,
      "accuracy": <float>,
      "avg_confidence": <float>,
      "gap": <float>,
      "flag": "<string: 'overconfident', 'underconfident', or null>"
    }
  ],
  "diagnostic_summary": "<string>"
}

## CONSTRAINTS
- Do not include any text outside the JSON object.
- Ensure all calculations are precise.
- If a bin has zero test cases, omit it from the `bins` array.

To adapt this template, start by replacing the [TEST_CASES] placeholder with your structured evaluation data. The critical configuration parameters are [NUM_BINS], [OVERCONFIDENCE_THRESHOLD], and [UNDERCONFIDENCE_THRESHOLD]. Start with 10 bins and a threshold of 0.1 for both overconfidence and underconfidence flags. For high-stakes applications where miscalibration is costly, tighten the thresholds to 0.05. The is_correct determination relies on semantic comparison; for objective, single-token answers, you can add a strict string-match instruction. For complex, open-ended outputs, you may need to replace this step with a separate LLM-as-a-judge call before feeding the data into this calibration prompt. Always validate the output JSON against the schema before ingesting the report into monitoring dashboards.

The Confidence Calibration Check prompt is not a one-shot query; it is a batch evaluation tool designed to run over a labeled test set. The implementation harness must iterate over a dataset of [INPUT, MODEL_OUTPUT, MODEL_CONFIDENCE, GROUND_TRUTH_CORRECTNESS] records, send each record through the prompt, collect the structured calibration report, and aggregate the results. Because this prompt produces a structured JSON report per record, the harness must validate the output schema before aggregation. A single malformed JSON response should trigger a retry with the same input, not silently corrupt the aggregate Expected Calibration Error (ECE) calculation.

Wire the prompt into a Python or TypeScript evaluation script that loads a golden dataset, calls the LLM with response_format set to json_object (or the equivalent structured output mode for your model provider), and validates the returned JSON against a strict schema. The schema must require: stated_confidence (0-1 float), observed_correct (boolean), confidence_bucket (string, e.g., '0.0-0.1'), calibration_assessment (one of 'well_calibrated', 'overconfident', 'underconfident'), and explanation (string). Use a JSON Schema validator like jsonschema in Python or zod in TypeScript. If validation fails, retry up to two times with the same input and a repair instruction appended to the prompt. After three failures, log the record as unprocessable and exclude it from aggregate metrics, but flag it for human review. For model choice, prefer models with strong instruction-following and JSON mode support (e.g., GPT-4o, Claude 3.5 Sonnet). Avoid models known to hallucinate numeric scores or ignore output format constraints.

After collecting all valid per-record reports, compute aggregate calibration metrics in application code, not in the LLM. Calculate Expected Calibration Error (ECE) by bucketing predictions by stated_confidence (in 0.1-width bins), computing the accuracy per bucket, and taking the weighted average of the absolute difference between bucket accuracy and bucket mean confidence. Also compute the overconfidence rate (fraction of records where calibration_assessment is 'overconfident') and underconfidence rate. Log these metrics to your observability platform alongside the model version, dataset version, and timestamp. If ECE exceeds a pre-defined threshold (e.g., 0.1), trigger an alert and block the model from autonomous high-risk decisions until recalibration or human review is complete. Do not use the LLM to compute the aggregate ECE; the LLM's role is per-record assessment only. The harness is the source of truth for aggregate calibration quality.