Confidence Score Calibration Audit Prompt

ML engineers and data platform teams use this prompt when they need to audit whether an extraction model's self-reported confidence scores are reliable. A model might claim 0.95 confidence on a field it gets wrong 40% of the time. This prompt compares model-assigned confidence against ground-truth correctness across a batch of extraction results, producing calibration curves, overconfidence flags, and threshold recommendations. Use it before setting automated decision thresholds, during model evaluation, or as part of a recurring extraction quality monitoring pipeline.

The ideal user has a batch of extraction results where each record contains both the model's confidence score and a ground-truth correctness label. Without ground truth, calibration is impossible—you cannot audit confidence reliability from confidence scores alone. The prompt expects structured input: a JSON array of records with at minimum confidence (a float between 0 and 1) and is_correct (a boolean). Optional fields include field_name for per-field calibration, document_id for traceability, and model_version for multi-model comparisons. The prompt works best with at least 100 records per calibration bucket to produce statistically meaningful curves. Smaller batches will produce noisy results and should be flagged with wider confidence intervals.

Do not use this prompt when you lack ground-truth labels, when the extraction task is subjective and correctness cannot be defined binarily, or when confidence scores are generated by a different model than the one that performed the extraction. Also avoid using it on streaming data without batching—calibration requires aggregation. If you need to audit extraction completeness, schema drift, or hallucination rates instead of confidence calibration, use the sibling audit prompts for those specific concerns. After running this prompt, expect to set new confidence thresholds, flag overconfident field types for retraining, or integrate the calibration curve into your monitoring dashboard.

This prompt is designed to be pasted directly into your orchestration layer, such as a Python script using the OpenAI or Anthropic SDKs, a LangChain node, or a custom evaluation harness. It instructs the model to act as an ML audit system, comparing model-assigned confidence scores with human-verified ground-truth labels. The goal is to produce a structured calibration report that quantifies overconfidence, underconfidence, and the reliability of your extraction pipeline's self-assessment.

textCopy
You are an ML audit system evaluating the calibration of confidence scores from an extraction pipeline. Your task is to compare a batch of extraction results against ground-truth labels and produce a calibration report.

## INPUT DATA
You will receive a JSON array of extraction records. Each record contains:
- `record_id`: A unique identifier for the extracted record.
- `extracted_fields`: An object containing the fields extracted by the model.
- `model_confidence`: A float between 0.0 and 1.0 representing the model's self-assigned confidence in the entire extraction.
- `ground_truth`: An object containing the correct values for the extracted fields.
- `field_accuracy`: An object mapping each field name to a boolean indicating if the extraction was correct.

[BATCH_DATA]

## OUTPUT SCHEMA
Return a single JSON object with the following structure:
{
  "calibration_curve": [
    {
      "confidence_bucket": "0.0-0.1",
      "bucket_lower_bound": 0.0,
      "bucket_upper_bound": 0.1,
      "sample_count": 10,
      "observed_accuracy": 0.05
    }
    // ... for all buckets 0.0-0.1 through 0.9-1.0
  ],
  "expected_calibration_error_ECE": 0.15,
  "overconfidence_flags": [
    {
      "record_id": "rec_123",
      "model_confidence": 0.95,
      "is_correct": false,
      "severity": "high",
      "explanation": "Model was highly confident but the extraction was completely wrong."
    }
  ],
  "underconfidence_flags": [
    {
      "record_id": "rec_456",
      "model_confidence": 0.15,
      "is_correct": true,
      "severity": "medium",
      "explanation": "Model had very low confidence but the extraction was fully correct."
    }
  ],
  "threshold_recommendations": {
    "recommended_auto_accept_threshold": 0.85,
    "recommended_human_review_threshold": 0.60,
    "rationale": "At 0.85 confidence, observed accuracy is 0.92. Below 0.60, accuracy drops to 0.45, making human review cost-effective."
  },
  "summary_statistics": {
    "total_records": 500,
    "overall_accuracy": 0.78,
    "mean_confidence": 0.82,
    "confidence_accuracy_gap": 0.04
  }
}

## CONSTRAINTS
- Use exactly 10 confidence buckets: 0.0-0.1, 0.1-0.2, ..., 0.9-1.0.
- Calculate Expected Calibration Error (ECE) as the weighted average of |accuracy - confidence| across buckets.
- Flag any record where the absolute difference between confidence and correctness exceeds 0.5 as a `high` severity overconfidence or underconfidence flag.
- If a bucket has zero samples, set `observed_accuracy` to null and `sample_count` to 0.
- Do not invent or assume any data not present in the input batch.
- If the input batch is empty, return a valid JSON object with empty arrays and zero/null summary statistics.

To adapt this prompt, replace the [BATCH_DATA] placeholder with your serialized JSON array of extraction records. Ensure your application code handles the case where the model returns malformed JSON by implementing a retry loop with a repair prompt. For high-stakes pipelines, always log the raw model response and the parsed output for auditability. Before deploying, run this prompt against a golden dataset where you know the expected ECE and flag counts to validate the model's arithmetic and reasoning.

This prompt is designed to run as a batch audit step, not a real-time extraction component. It expects a structured payload containing model-assigned confidence scores paired with ground-truth correctness labels for each extraction. The typical integration point is inside an evaluation harness that runs after a batch extraction job completes and before results are accepted for downstream ingestion. You feed the prompt a JSON array of [confidence, is_correct] pairs, and it returns calibration curves, overconfidence flags, and threshold recommendations. The prompt is stateless and idempotent—running it twice with the same input should produce identical audit conclusions.

Input assembly is the critical pre-step. You must join two data sources: the extraction pipeline's confidence scores (typically from the model's log probabilities, token-level scores, or a separate confidence estimator) and the ground-truth correctness labels (from human review, golden datasets, or programmatic checks). Structure the input as a JSON array of objects with confidence (float 0–1), is_correct (boolean), and optionally field_name and document_id for per-field drill-down. If your batch exceeds ~500 records, split it into chunks of 200–400 records and run the prompt per chunk, then aggregate the results. Validate the input before calling the prompt: reject confidence values outside [0, 1], ensure is_correct is strictly boolean, and confirm the array is non-empty. A malformed input will produce garbage calibration output.

Model choice matters. This prompt performs statistical reasoning—binning, curve estimation, and threshold analysis. Use a model with strong quantitative reasoning (GPT-4o, Claude 3.5 Sonnet, or equivalent). Avoid smaller or older models that may hallucinate bin counts or miscalculate expected calibration error (ECE). Set temperature=0 and top_p=1 for deterministic output. The prompt includes an [OUTPUT_SCHEMA] placeholder where you should insert a strict JSON schema for the calibration report. Require fields like ece_score, overconfidence_ratio, calibration_curve (array of bin objects with bin_range, count, accuracy, mean_confidence), overconfident_bins (array of bin indices where confidence exceeds accuracy by more than a configurable margin), and recommended_threshold (the confidence value above which accuracy meets your target). Validate the output against this schema before accepting it.

Retry logic should be minimal. If the output fails schema validation, retry once with the same input and a stronger instruction to adhere to the schema. If it fails again, log the raw output and escalate for manual review—do not loop. Logging is essential for auditability. Capture the input batch ID, record count, model version, prompt version, timestamp, raw output, and parsed calibration metrics. Store these in your evaluation database alongside the extraction run metadata. This creates a traceable record for compliance and debugging. If the prompt flags overconfidence (e.g., ECE > 0.1 or overconfidence ratio > 0.2), trigger an alert to the ML engineering team and block automated ingestion until the calibration issue is investigated.

Human review integration is straightforward: the recommended_threshold output tells you where to set your confidence cutoff for automated acceptance. Records below this threshold should route to a human review queue. The overconfident_bins output tells reviewers which confidence ranges are most misleading—use this to prioritize re-labeling efforts or to adjust the confidence estimator. What to avoid: Do not use this prompt for real-time extraction gating on every document. It is a batch audit tool. Do not skip input validation—bad confidence values will corrupt the calibration math. Do not treat the model's threshold recommendation as final without verifying it against your business SLA for precision and recall. The prompt is an advisor, not an automated decision-maker.