Confidence Calibration Prompt for Extraction Models

This prompt is for ML engineers and data scientists who need to verify that an extraction model's self-reported confidence scores are a reliable signal for downstream automation gates. The core job-to-be-done is to compare predicted confidence bins against actual accuracy on a labeled dataset, producing a calibration curve, drift indicators, and recalibration suggestions. The ideal user has a set of extraction results with both model-assigned confidence scores and ground-truth labels, and they need to decide whether the current threshold logic is safe for production or requires tuning.

Use this prompt when you are evaluating a model before deployment, monitoring it in production for drift, or tuning the confidence thresholds that control human escalation. It is designed for structured extraction tasks where the model outputs a confidence score between 0.0 and 1.0 alongside each extracted field. The prompt expects a batch of scored predictions and their corresponding correctness labels, and it returns a calibration analysis rather than raw predictions. This is not a prompt for generating extractions, fixing extraction errors, or routing individual records—it is a meta-evaluation tool for the confidence layer itself.

Do not use this prompt when you lack ground-truth labels for comparison, as calibration analysis requires observed correctness data. It is also inappropriate for models that do not produce a numeric confidence score, or for single-record debugging where a batch analysis is unnecessary. If your goal is to flag individual low-confidence fields for human review, use the Low-Confidence Field Flagging Prompt instead. If you need to set initial thresholds based on business risk rather than empirical calibration, start with the Extraction Confidence Threshold Decision Prompt and return here once you have production data to validate against.

The output of this prompt should feed directly into your monitoring dashboards and threshold configuration. Expect a structured report with binned accuracy comparisons, an overconfidence score, and specific recalibration recommendations. Before relying on the analysis, validate that your labeled sample is representative of production traffic and large enough to populate each confidence bin with sufficient examples—sparse bins produce unreliable calibration estimates.

This template is designed to be dropped into an evaluation harness that feeds a batch of extraction results—each with a model-reported confidence score and a ground-truth correctness label—and returns a structured calibration analysis. The prompt instructs the model to act as a calibration auditor, not to re-extract data. It expects pre-computed inputs and focuses entirely on the relationship between confidence and accuracy.

textCopy
You are an extraction model calibration auditor. Your task is to analyze the alignment between a model's reported confidence scores and its actual extraction accuracy.

## INPUT
You will receive a JSON array of extraction records. Each record contains:
- `field_name`: the extracted field identifier
- `reported_confidence`: a float between 0.0 and 1.0
- `is_correct`: a boolean indicating whether the extraction matched ground truth

[EXTRACTION_RECORDS]

## TASK
1. Bin the records by confidence into the following buckets: [0.0-0.2), [0.2-0.4), [0.4-0.6), [0.6-0.8), [0.8-0.9), [0.9-1.0].
2. For each bin, calculate:
   - Count of records
   - Observed accuracy (fraction of `is_correct` = true)
   - Expected accuracy (midpoint of the bin range)
   - Calibration error (observed accuracy minus expected accuracy)
3. Identify bins where the absolute calibration error exceeds [CALIBRATION_THRESHOLD].
4. Flag whether the model exhibits **overconfidence** (observed accuracy significantly below expected) or **underconfidence** (observed accuracy significantly above expected) in any bin.
5. If overconfidence is detected, note the confidence bins where it occurs and the magnitude of the gap.
6. Provide a recalibration suggestion: a simple scaling or shifting recommendation that would bring observed accuracy closer to expected accuracy, if applicable.

## OUTPUT SCHEMA
Return a JSON object with this exact structure:
{
  "calibration_curve": [
    {
      "bin_range": "[0.0-0.2)",
      "count": 0,
      "observed_accuracy": 0.0,
      "expected_accuracy": 0.1,
      "calibration_error": 0.0
    }
  ],
  "overconfidence_detected": false,
  "overconfident_bins": [],
  "underconfidence_detected": false,
  "underconfident_bins": [],
  "recalibration_suggestion": "string or null",
  "summary": "string"
}

## CONSTRAINTS
- Do not invent records. Only analyze the provided [EXTRACTION_RECORDS].
- If a bin has zero records, set `observed_accuracy` to null and `calibration_error` to null.
- The `summary` field must be a concise, evidence-grounded paragraph describing the overall calibration quality.
- If no recalibration is needed, set `recalibration_suggestion` to null.
- Do not include any text outside the JSON response.

To adapt this template, replace [EXTRACTION_RECORDS] with your actual batch data. Adjust [CALIBRATION_THRESHOLD] based on your tolerance—0.10 is a common starting point for detecting meaningful miscalibration. If your extraction model uses a different confidence scale, normalize it to 0.0–1.0 before passing records in. For high-stakes domains, always pair this prompt with a human review of the calibration report before adjusting any production thresholds.

This prompt is not a one-shot query; it is a batch evaluation step in an ML pipeline. The primary integration point is a calibration script that loads a dataset of extraction results, each containing a model-reported confidence score and a ground-truth correctness label. The script assembles the prompt by injecting the dataset summary and per-bin statistics into the [EXTRACTION_DATASET] and [CONFIDENCE_BINS] placeholders, then sends the request to a capable reasoning model. The output is a structured calibration report, not a single extraction, so the harness must parse the JSON response and validate it against the expected schema before the report is used to adjust thresholds or trigger retraining.

Validation and retry logic is critical because a malformed calibration report can lead to incorrect threshold adjustments. After receiving the model response, validate that the calibration_curve array contains monotonically increasing bin boundaries, that expected_calibration_error is a float between 0 and 1, and that overconfidence_detected is a boolean. If validation fails, retry once with an explicit error message injected into the prompt context. If the second attempt also fails, log the raw response and escalate to a human reviewer rather than silently proceeding with a broken report. For high-stakes extraction pipelines where threshold changes affect compliance or financial outcomes, always require human approval before applying recalibration suggestions.

Model choice matters for this task. Use a model with strong reasoning and numerical analysis capabilities—GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro are appropriate. Avoid smaller or older models that may hallucinate statistical interpretations or produce invalid JSON. The prompt is designed for a single-turn request with no tool calls, but you can extend it by providing a calculate_ece tool if you want the model to verify its own expected calibration error computation. Logging should capture the full prompt, the raw response, the parsed calibration report, and any validation errors. This audit trail is essential when debugging why a pipeline's confidence thresholds drifted or when proving to auditors that calibration checks are performed regularly.

Integration pattern: run this prompt on a schedule (daily or per-batch) against a rolling window of recent extraction results. Feed the output into your threshold configuration system—if overconfidence_detected is true or expected_calibration_error exceeds your tolerance, automatically widen the confidence threshold for human review or pause automated ingestion. Do not use this prompt for real-time per-extraction decisions; it is a batch analysis tool. For per-extraction confidence handling, pair it with the Low-Confidence Field Flagging Prompt or the Extraction Confidence Threshold Decision Prompt in this playbook series.