Judge Confidence Score Calibration Prompt Template

This playbook is for evaluation infrastructure teams who need to answer a critical question: when an LLM judge says it is 90% confident in a score, does that actually mean the score is correct 90% of the time? The Judge Confidence Score Calibration Prompt Template transforms raw judge outputs—scores paired with self-reported confidence levels—into a calibration analysis that reveals whether your judges are well-calibrated, systematically overconfident, or underconfident. The ideal user is an AI engineer or evaluation lead who already has a judge fleet producing scored outputs with confidence metadata and needs to validate those confidence signals before relying on them for automated decision-making, low-confidence escalation, or human-review routing.

Use this prompt when you have a batch of scored items where each item includes a judge-assigned score, a judge-reported confidence value, and a ground-truth correctness label. The prompt expects these inputs structured as a score matrix and produces calibration curves, Expected Calibration Error (ECE), and confidence-agreement correlation analysis. This is not a prompt for building a judge from scratch, nor is it for evaluating the quality of the outputs themselves—it assumes you already have a working judge and need to audit its confidence calibration. Do not use this prompt when you lack ground-truth labels, when confidence scores are absent or inferred rather than explicitly requested from the judge, or when your sample size is too small to produce statistically meaningful calibration bins.

The output of this prompt is diagnostic, not operational. It tells you whether your judges need recalibration, identifies confidence ranges where the judge is unreliable, and recommends escalation thresholds for low-confidence items. It does not automatically adjust judge behavior or retrain models. After running this analysis, expect to take follow-up actions: tightening your confidence elicitation instructions, adding calibration examples to your judge prompt, implementing confidence-based routing rules, or flagging specific judges for replacement. If your workflow involves high-stakes decisions—such as safety filtering, medical review, or legal compliance—always pair this analysis with human review of calibration outliers and never rely solely on self-reported confidence without external validation.

This prompt instructs an LLM to act as a calibration analyst. It takes a set of scored items—each containing a judge's confidence score and a ground-truth correctness label—and produces a calibration report. The report includes a calibration curve, Expected Calibration Error (ECE), and confidence-agreement correlation analysis. Use this prompt when you have a batch of judge outputs with known outcomes and need to determine whether the judge's confidence scores are trustworthy predictors of accuracy.

textCopy
You are a calibration analyst evaluating the reliability of confidence scores produced by an LLM judge.

You will receive a dataset where each record contains:
- A confidence score assigned by the judge (a float between 0.0 and 1.0).
- A ground-truth correctness label (1 for correct, 0 for incorrect).

Your task is to produce a calibration report in strict JSON format.

## INPUT DATA
[JUDGE_SCORE_DATA]

## OUTPUT SCHEMA
Return a JSON object with the following fields:
{
  "calibration_curve": [
    {
      "confidence_bin": "string (e.g., '0.0-0.1', '0.1-0.2', ... '0.9-1.0')",
      "bin_center": float,
      "sample_count": int,
      "accuracy_in_bin": float,
      "average_confidence_in_bin": float
    }
  ],
  "expected_calibration_error": float,
  "maximum_calibration_error": float,
  "confidence_accuracy_correlation": float,
  "overconfidence_score": float,
  "underconfidence_score": float,
  "reliability_diagram_description": "string summary of the calibration curve shape",
  "escalation_threshold_recommendation": {
    "low_confidence_threshold": float,
    "high_confidence_threshold": float,
    "rationale": "string explaining why these thresholds were chosen based on observed accuracy patterns"
  },
  "flagged_bins": [
    {
      "confidence_bin": "string",
      "issue": "overconfidence | underconfidence | low_sample_count",
      "severity": "low | medium | high",
      "detail": "string explanation"
    }
  ]
}

## CONSTRAINTS
- Use 10 equal-width bins from 0.0 to 1.0.
- If a bin has fewer than [MIN_SAMPLES_PER_BIN] samples, flag it with severity based on how far the bin count is from the minimum.
- ECE is the weighted average of |accuracy_in_bin - average_confidence_in_bin| across bins, weighted by sample_count.
- Overconfidence score is the weighted sum of positive differences (confidence > accuracy).
- Underconfidence score is the weighted sum of positive differences (accuracy > confidence).
- If the dataset has fewer than [MIN_TOTAL_SAMPLES] total samples, include a warning in the reliability_diagram_description and note that calibration metrics may be unreliable.
- Do not invent data. If a bin is empty, set sample_count to 0 and accuracy_in_bin to null.

## RISK LEVEL
[RISK_LEVEL]

## EXAMPLES
[EXAMPLES]

To adapt this prompt, replace [JUDGE_SCORE_DATA] with your actual scored dataset as a JSON array of {"confidence": float, "correct": int} objects. Set [MIN_SAMPLES_PER_BIN] to a reasonable floor (typically 5–20 depending on your total dataset size) and [MIN_TOTAL_SAMPLES] to at least 50 for meaningful calibration. The [RISK_LEVEL] placeholder should be set to "low", "medium", or "high" based on the downstream consequences of miscalibrated confidence—this controls how conservative the escalation thresholds should be. The [EXAMPLES] placeholder can contain one or two representative calibration curve examples to anchor the model's output style. After generation, validate the JSON structure against the schema and verify that ECE is computed correctly by spot-checking a few bins manually. For high-risk domains, route the calibration report to a human reviewer before adjusting any production escalation thresholds.

The calibration prompt is not a one-off analysis tool; it is a reusable component that should run automatically whenever a new judge is onboarded, after a prompt update, or on a scheduled cadence to detect drift. The harness must accept a structured dataset of scored items—each containing the judge's confidence score, the ground-truth correctness label, and optional metadata such as the item's difficulty tier or source. The prompt expects this data serialized as a JSON array of objects, which the harness injects into the [INPUT_DATA] placeholder. Because calibration is a statistical process, the harness should enforce a minimum sample size (typically at least 100 items per judge) before invoking the model, and it should log a warning if the sample is below the threshold for reliable Expected Calibration Error (ECE) computation.

Wire the prompt into a Python or TypeScript evaluation service that performs pre-validation, model invocation, and post-validation in sequence. Before calling the model, validate that the input data contains the required fields (judge_id, confidence_score, is_correct) and that confidence scores are numeric values between 0 and 1. After receiving the model's JSON response, validate the output schema: the calibration_curve must contain at least 5 bins, each with bin_range, count, accuracy, and average_confidence fields; the ece field must be a float between 0 and 1; and the overconfidence_detection section must include a flagged_bins array and a severity classification. If validation fails, retry once with the error message appended to the prompt as a correction instruction. If the retry also fails, log the failure and escalate to a human reviewer rather than silently accepting a malformed calibration report.

Model choice matters for this workflow. Use a model with strong quantitative reasoning capabilities and reliable JSON output (such as GPT-4o, Claude 3.5 Sonnet, or equivalent). Avoid smaller or older models that may miscalculate ECE or produce inconsistent bin boundaries. Set temperature=0 to maximize reproducibility, and configure a response format constraint (JSON mode or structured outputs) to reduce parsing failures. For high-stakes calibration decisions—such as determining whether a judge is safe to deploy to production—route the model's output to a human reviewer alongside a summary dashboard that highlights the ECE value, any overconfidence flags, and a comparison against the previous calibration run. The human gate should be mandatory when ECE exceeds 0.15 or when the overconfidence severity is classified as high.

Log every calibration run with a unique run ID, timestamp, judge identifier, model version, prompt version, and the full input/output pair. Store these in a queryable format (such as a database table or structured log sink) to enable trend analysis over time. The harness should also compute a simple agreement-correlation metric (Pearson or Spearman) between confidence scores and correctness labels outside the model call, and compare it against the model's reported correlation to detect hallucinated statistics. If the model's reported correlation deviates from the harness-computed value by more than 0.1, flag the run for review and do not automatically publish the results. Finally, expose the calibration output through an internal dashboard or API so that other services—such as judge routing, alerting, or model update pipelines—can consume the calibration status programmatically.