Pairwise Judge Calibration Prompt

This prompt is for evaluation platform teams who need to align an LLM judge's preferences against a known human baseline. The core job is to detect and measure bias in your automated judge before you trust it to evaluate production outputs. You use this when you have a calibration set—a collection of output pairs where human annotators have already established the correct preference—and you need to quantify how often your judge agrees with those labels. This is not a prompt for generating new preference data or for evaluating a single model's output in isolation. It is a diagnostic tool for your evaluation infrastructure itself.

Use this prompt when you are onboarding a new LLM judge, switching judge models, or monitoring for drift in an existing judge. It is also appropriate after major prompt changes to your evaluation harness. The prompt requires a structured calibration set as input, typically containing the two outputs under comparison, the known human preference, and the evaluation criteria that annotators used. Do not use this prompt when you lack a verified human baseline, when you are comparing more than two outputs at once, or when you are performing a live champion-challenger test against a production model. Those scenarios require different prompts from the pairwise comparison family, such as the Champion-Challenger Model Testing Prompt or the Blind A/B Output Evaluation Prompt.

The output of this prompt is not a preference decision for a single pair. It is a calibration report that compares the judge's decisions against the human baseline across the entire set, surfacing systematic disagreements, confidence misalignment, and per-category bias. Before running this prompt, ensure your calibration set is representative of your production distribution and that human labels were collected with clear, documented criteria. After running, you should log the agreement rate, Cohen's kappa, and per-category confusion matrix. If agreement falls below your threshold, do not deploy the judge. Instead, investigate the failure modes, refine your judge's instructions, or expand the calibration set to cover the gaps.

This template is the core instruction set for a calibration run. It presents the judge with a pair of outputs and the known human preference label, then asks the judge to produce its own decision and a structured explanation of any divergence. The goal is not to retrain the judge but to measure alignment, detect systematic bias, and produce a calibration report that can be fed into downstream normalization or drift monitoring systems.

codeCopy
SYSTEM:
You are an evaluation calibration assistant. Your task is to compare two model outputs for the same input and produce a preference decision. You will also be shown a known human preference label. Your job is to explain your own judgment and, if it differs from the human label, analyze the likely cause of the divergence.

INPUT:
[INPUT]

OUTPUT A:
[OUTPUT_A]

OUTPUT B:
[OUTPUT_B]

HUMAN PREFERENCE LABEL:
[HUMAN_LABEL]

EVALUATION CRITERIA:
[CRITERIA]

OUTPUT SCHEMA:
{
  "judge_preference": "A" | "B" | "TIE",
  "judge_confidence": 0.0-1.0,
  "judge_rationale": "string",
  "matches_human_label": true | false,
  "divergence_analysis": {
    "likely_cause": "CRITERIA_WEIGHTING" | "MISSED_DETAIL" | "SUBJECTIVE_DIMENSION" | "ANNOTATOR_ERROR" | "JUDGE_BIAS" | "NONE",
    "explanation": "string",
    "criteria_affected": ["string"]
  },
  "position_bias_check": {
    "swapped_order_decision": "A" | "B" | "TIE" | "NOT_CHECKED",
    "position_consistent": true | false | "NOT_CHECKED"
  }
}

CONSTRAINTS:
- You must produce a preference even if the difference is small.
- If you select TIE, you must explain why no meaningful distinction exists under the given criteria.
- The divergence_analysis field is required only when matches_human_label is false.
- Do not use the human label to guide your own judgment. Explain your independent assessment first, then compare.

To adapt this template, replace the square-bracket placeholders with your calibration set data. The [CRITERIA] field should contain the exact evaluation dimensions used by human annotators—do not paraphrase or simplify them, as criteria mismatch is a primary source of false divergence signals. The [HUMAN_LABEL] should be one of A, B, or TIE and must come from a validated annotation run with measured inter-annotator agreement. If your calibration set includes position-swapped duplicates, set the position_bias_check fields accordingly; otherwise, leave them as NOT_CHECKED. For high-stakes calibration (e.g., safety evaluations or regulatory-facing systems), always include a human review step on divergence cases before accepting the calibration report as ground truth for downstream normalization.

The calibration prompt is not a standalone utility; it is a measurement instrument that must be embedded in a controlled harness. The core workflow sends a calibration pair—two outputs where the correct human preference label is already known—to the LLM judge and compares the judge's decision against the ground-truth label. This comparison produces the raw signal for bias detection, score normalization, and drift monitoring. The harness must ensure that the judge never sees the ground-truth label during inference, that position bias is mitigated through randomized output ordering, and that every decision is logged with enough metadata to reconstruct the evaluation context later.

Implement the harness as a stateless function that accepts a calibration record and returns a structured judgment with alignment metrics. The function should: (1) randomize the order of Output A and Output B to mitigate position bias, recording the original positions for later analysis; (2) inject the randomized pair into the prompt template's [OUTPUT_A] and [OUTPUT_B] placeholders along with the [TASK_DESCRIPTION] and [EVALUATION_CRITERIA]; (3) call the judge model with a low temperature (0.0–0.2) for deterministic, reproducible decisions; (4) parse the structured output—preference verdict, justification, and confidence score—using a schema validator that rejects malformed responses; (5) compare the judge's preference against the known [GROUND_TRUTH_PREFERENCE] label to compute agreement; (6) log the full decision record including the randomized order, judge verdict, ground truth, agreement boolean, confidence score, and a timestamped model identifier. On validation failure or unparseable output, retry once with a simplified prompt that removes the justification field and requests only the verdict; if that also fails, log the failure and exclude the record from calibration metrics rather than guessing.

For production deployment, run calibration batches on a fixed schedule—daily or per-deployment—using a held-out calibration set that covers your evaluation dimensions (accuracy, safety, style, instruction following). Track agreement rate over time as a primary health metric. When agreement drops below a configured threshold (typically 80–85%, depending on task subjectivity), trigger an alert and pause automated evaluation pipelines that depend on that judge. Store calibration results in a queryable log with fields for judge model version, calibration set version, per-dimension agreement rates, position bias ratio (how often the judge prefers Output A when it appears first), and confidence calibration error. Use this log to detect judge drift before it corrupts downstream RLHF data or model selection decisions. Never use the same pairs for both calibration and judge fine-tuning; maintain strict separation between calibration data and training data to avoid circular evaluation.

Model choice matters. Use a capable model for the judge role—typically the same tier as your production evaluator—and pin the model version. Changing the judge model invalidates calibration baselines, so treat judge model updates as a migration event: run the full calibration set against the new judge, compare agreement rates against the previous judge, and establish new thresholds before switching. If your calibration set includes human-labeled pairs with inter-annotator agreement scores, weight calibration metrics by human agreement difficulty; a judge disagreeing with humans on low-agreement pairs is less concerning than disagreement on high-agreement pairs. Finally, expose a human review queue for calibration failures where the judge's confidence is high but the agreement is false—these are the highest-signal cases for improving either the judge prompt or the calibration set itself.