Auto-Verification Confidence Calibration Prompt

This prompt is for ML engineers and verification pipeline operators who need to measure whether their auto-verification system's confidence scores actually predict correctness. The core job is calibration analysis: comparing predicted confidence values against ground-truth human reviewer outcomes to detect systematic overconfidence or underconfidence. Use this when you have a batch of claims that have been both auto-verified (with confidence scores) and independently reviewed by humans (with binary correct/incorrect labels). The output is a calibration report that surfaces confidence ranges where the model is unreliable, enabling you to set triage thresholds, adjust confidence outputs, or route uncertain claims to human review.

Do not use this prompt for real-time verification decisions or as a substitute for human review in high-stakes domains. It is an offline analysis tool, not a runtime safety net. It requires paired data: every claim in the input must have both a model-assigned confidence score and a human-assigned correctness label. If you lack human-reviewed ground truth, this prompt cannot produce meaningful calibration metrics. Similarly, if your auto-verification system outputs only binary true/false labels without confidence scores, you need a different evaluation approach. This prompt also assumes that human reviewer judgments are treated as ground truth for calibration purposes—if your reviewer pool has significant disagreement or inconsistency, calibrating against noisy labels will produce misleading results.

Before running this prompt, ensure your input dataset includes at least several hundred claim-review pairs to produce statistically meaningful calibration curves. Stratify your sample across confidence bins to avoid sparse regions that produce unreliable estimates. After receiving the calibration analysis, use the identified miscalibration ranges to set conservative auto-verification thresholds, flag claims in overconfident bins for mandatory human review, or apply Platt scaling or isotonic regression to recalibrate raw model scores. The next step is feeding these calibration findings into your Verification Triage Decision Prompt to update routing rules with evidence-backed confidence thresholds.

This prompt template is designed to analyze a batch of verification records where both an automated system and a human reviewer have assessed the same claims. It produces a calibration analysis that compares predicted confidence scores to actual correctness rates, identifies confidence ranges where the auto-verification system systematically overestimates or underestimates its reliability, and calculates Expected Calibration Error (ECE). Use this template when you have accumulated enough paired auto-verification and human-review outcomes to perform statistically meaningful calibration analysis. Do not use this prompt for individual claim triage decisions—it is a batch analysis tool for ML engineers tuning verification pipelines.

Below is the copy-ready prompt template. Replace each square-bracket placeholder with your specific data, schemas, and constraints before sending it to the model. The template expects a structured input of verification records, each containing the auto-verification confidence score and the human reviewer's correctness determination. The output schema is designed to be machine-readable for downstream monitoring dashboards or model retraining pipelines.

textCopy
You are a calibration analyst evaluating an auto-verification system. Your task is to compare predicted confidence scores against human reviewer outcomes and produce a calibration analysis.

## INPUT DATA
You will receive a JSON array of verification records. Each record contains:
- claim_id: unique identifier for the claim
- auto_confidence: the model's predicted confidence score (float between 0.0 and 1.0)
- human_correct: whether the human reviewer confirmed the auto-verification was correct (boolean)
- [ADDITIONAL_METADATA_FIELDS]

Records:
[VERIFICATION_RECORDS_JSON]

## OUTPUT SCHEMA
Return a JSON object with the following structure:
{
  "total_records": integer,
  "ece": float,
  "ece_interpretation": string,
  "confidence_buckets": [
    {
      "bucket_range": string,
      "count": integer,
      "expected_accuracy": float,
      "observed_accuracy": float,
      "calibration_error": float,
      "bias_direction": "overconfident" | "underconfident" | "calibrated"
    }
  ],
  "overall_bias": "overconfident" | "underconfident" | "calibrated",
  "problematic_ranges": [
    {
      "range": string,
      "issue": string,
      "recommendation": string
    }
  ],
  "summary": string,
  "recommendations": [string]
}

## CONSTRAINTS
- Use exactly 10 equal-width confidence buckets (0.0-0.1, 0.1-0.2, ..., 0.9-1.0).
- Calculate ECE as the weighted average of absolute differences between expected and observed accuracy per bucket.
- Flag any bucket where |expected_accuracy - observed_accuracy| > 0.10 as problematic.
- If a bucket has fewer than [MIN_SAMPLES_PER_BUCKET] records, mark it with a note about insufficient data rather than drawing strong conclusions.
- The summary should be no more than [MAX_SUMMARY_WORDS] words and highlight the most actionable finding.
- Do not extrapolate beyond the provided data. If confidence ranges are sparsely represented, note the limitation.

## RISK CONTEXT
This analysis will be used to adjust auto-verification thresholds and routing rules. [RISK_LEVEL] risk tolerance applies. [ADDITIONAL_RISK_CONTEXT]

## EXAMPLES
Example input:
[FEW_SHOT_EXAMPLES]

Example output:
[OUTPUT_EXAMPLES]

To adapt this template for your pipeline, start by replacing [VERIFICATION_RECORDS_JSON] with your actual paired records. Define [MIN_SAMPLES_PER_BUCKET] based on your statistical requirements—typically at least 20-30 records per bucket for reliable calibration estimates. Set [MAX_SUMMARY_WORDS] to fit your reporting format, such as 150 words for a dashboard widget or 500 for a detailed report. The [RISK_LEVEL] and [ADDITIONAL_RISK_CONTEXT] fields should reflect your domain's tolerance for miscalibration: high-risk domains like healthcare or finance may require tighter calibration thresholds and mandatory human review for problematic ranges. Include [FEW_SHOT_EXAMPLES] if your model benefits from seeing the expected output format, and provide [OUTPUT_EXAMPLES] to anchor the model's behavior. After generating the analysis, validate the output against your schema, verify that ECE is calculated correctly, and cross-check bucket-level statistics against your raw data before using the results to adjust production thresholds.

This prompt is not a runtime component of your verification pipeline; it is an offline evaluation tool for ML engineers. Its job is to compare predicted confidence scores against actual human reviewer outcomes to measure calibration error. The implementation harness must therefore treat this prompt as a batch analysis job: feed it a structured dataset of historical verification decisions, collect the calibration report, and use the output to adjust confidence thresholds in your production triage system. Do not call this prompt in the hot path of live verification requests.

The harness should assemble a JSONL input file where each record contains a claim, the auto-verification system's predicted confidence score, the human reviewer's ground-truth correctness label, and optional metadata such as claim domain or evidence count. Pass this file to the prompt along with a [CONFIDENCE_BIN_COUNT] parameter (typically 10 or 20) and an [OUTPUT_SCHEMA] that specifies the expected report structure. Validate the input before running: reject records with missing confidence scores, non-numeric scores, or labels that are not binary correctness indicators. The model call should be wrapped in a retry loop with a maximum of three attempts, checking that the returned JSON matches the schema and that bin counts, sample sizes, and ECE calculations are internally consistent. Log every run with a unique run ID, timestamp, model version, and input dataset hash for auditability.

After receiving the calibration report, the harness should programmatically extract the Expected Calibration Error (ECE) and the per-bin accuracy-vs-confidence table. Use these to identify confidence ranges where the auto-verification system systematically overestimates or underestimates reliability. For example, if the 0.8–0.9 confidence bin shows actual accuracy of only 0.62, the production triage threshold for auto-verification should be raised above 0.9 until the model is recalibrated. Store the report in your experiment tracking system alongside the input dataset reference. Do not use this prompt to make live routing decisions; its output informs threshold tuning and model retraining decisions made by engineers, not the runtime verification flow. The next step is to run this calibration analysis on a held-out test set before every model or prompt update to detect calibration drift before it affects production triage quality.