LLM Judge Calibration Against Human Ratings Prompt Template

This prompt is designed for evaluation platform teams who need to align an LLM judge's scoring behavior against a set of human-annotated ratings. The core job-to-be-done is producing a calibration mapping that reveals systematic over-scoring, under-scoring, or dimension-specific drift, so that automated evaluation scores can be trusted as a reliable proxy for human judgment. The ideal user is an MLOps engineer, evaluation lead, or AI quality architect who already has a batch of scored outputs with both LLM judge scores and human reference labels, and who needs to quantify the gap between them before shipping an automated evaluation pipeline.

Use this prompt when you have a stable rubric, a consistent judge prompt, and a representative sample of human-annotated examples spanning the full scoring range. It works best for Likert-scale, categorical, or numeric scoring dimensions where alignment can be measured through distribution comparison, mean offset, and agreement coefficients. The prompt expects per-dimension calibration metrics, so your input data must include dimension-level scores from both the judge and human raters. If you only have overall scores without dimension breakdowns, or if your human labels are binary pass/fail rather than graded, consider the Score Threshold Calibration Prompt instead.

Do not use this prompt when the judge prompt itself is still under active revision, when the human annotation set is too small to produce statistically meaningful alignment metrics (typically fewer than 50 examples per score bucket), or when the rubric definitions have shifted between the time of human annotation and judge scoring. In those cases, invest first in Calibration Set Construction or Judge Instruction Rewriting prompts. Also avoid this prompt for real-time score adjustment during inference; it produces an offline calibration analysis meant for judge improvement and monitoring, not a live scoring wrapper. For production score normalization at runtime, pair this analysis output with the Score Normalization Prompt Across Multiple LLM Judges.

This section provides the core prompt for aligning an LLM judge with human rating distributions. The template is designed to be dropped into an evaluation harness where it receives a batch of scored examples and produces a calibration analysis. It does not assume a specific scoring scale or rubric—those are injected via the [RUBRIC] and [SCORING_SCALE] placeholders. The prompt instructs the model to act as a calibration analyst, comparing its own hypothetical scores against provided human scores to identify systematic biases, per-dimension drift, and confidence intervals.

textCopy
You are a calibration analyst for an LLM-based evaluation system. Your task is to compare your own scoring behavior against a set of human-annotated examples and produce a calibration report.

## INPUT DATA
You will receive a set of examples. Each example contains:
- An input that was evaluated.
- A human rating with per-dimension scores.
- The human rater's brief justification.

[RATED_EXAMPLES]

## YOUR TASK
For each example, first independently score it according to the provided rubric. Then compare your score to the human score. After processing all examples, produce a calibration analysis.

## RUBRIC
[RUBRIC]

## SCORING SCALE
[SCORING_SCALE]

## OUTPUT FORMAT
Produce a JSON object with the following structure:
{
  "per_example_analysis": [
    {
      "example_id": "string",
      "my_scores": { "[DIMENSION_NAME]": number },
      "human_scores": { "[DIMENSION_NAME]": number },
      "score_delta": { "[DIMENSION_NAME]": number },
      "discrepancy_category": "judge_over_score" | "judge_under_score" | "aligned" | "rubric_ambiguity" | "edge_case",
      "explanation": "string"
    }
  ],
  "calibration_summary": {
    "overall_alignment": "strongly_aligned" | "moderately_aligned" | "weakly_aligned" | "misaligned",
    "mean_delta_per_dimension": { "[DIMENSION_NAME]": number },
    "systematic_bias_detected": boolean,
    "bias_description": "string or null",
    "confidence_interval_95_per_dimension": { "[DIMENSION_NAME]": [number, number] },
    "most_problematic_dimension": "string or null",
    "recommended_calibration_actions": ["string"]
  },
  "discrepancy_distribution": {
    "judge_over_score_count": number,
    "judge_under_score_count": number,
    "aligned_count": number,
    "rubric_ambiguity_count": number,
    "edge_case_count": number
  }
}

## CONSTRAINTS
- Score independently before comparing. Do not anchor on the human score.
- If the rubric is ambiguous for a specific example, flag it as "rubric_ambiguity" rather than forcing alignment.
- Report systematic bias only when a pattern is clear across multiple examples, not from a single discrepancy.
- Confidence intervals should reflect the variance in your score deltas, not a fixed assumption.
- If fewer than [MIN_EXAMPLES_FOR_CALIBRATION] examples are provided, set overall_alignment to "insufficient_data" and omit statistical fields.

To adapt this template, replace the square-bracket placeholders with your specific configuration. [RUBRIC] should contain the full evaluation criteria your judge uses, including dimension definitions and anchor examples. [SCORING_SCALE] must define the numeric range and what each value represents. [RATED_EXAMPLES] is the calibration dataset—inject it as structured text or JSON, ensuring each example has a stable example_id. [DIMENSION_NAME] placeholders in the output schema should be replaced with your actual dimension keys. [MIN_EXAMPLES_FOR_CALIBRATION] should be set to a threshold below which statistical claims are unreliable; 20–30 is a reasonable default for initial calibration. The output schema is designed for direct ingestion by a downstream calibration dashboard or automated threshold adjustment system. If your judge uses a different discrepancy taxonomy, modify the discrepancy_category enum before use.

The calibration prompt is not a one-shot report generator. It should sit inside a pipeline that feeds it batches of human-annotated examples paired with LLM judge outputs, collects the structured calibration results, and routes them to downstream consumers: dashboards, alerting systems, judge instruction updaters, and human review queues. The harness is responsible for data preparation, invocation, output validation, and action routing.

Input assembly. Before calling the prompt, assemble a batch of [NUM_EXAMPLES] records, each containing the original input, the LLM judge's score and justification, the human rating, and optional metadata such as difficulty tier, topic category, and annotator ID. Stratify the batch to cover score boundaries, edge cases, and underrepresented rating regions. If the calibration set construction prompt is available upstream, use its output to ensure coverage. Pass the batch as [CALIBRATION_EXAMPLES] in a structured format (JSON array) alongside [JUDGE_NAME], [EVALUATION_DIMENSIONS], and [CONFIDENCE_LEVEL] for interval calculations.

Invocation and retry logic. Call the prompt with a model that supports structured output (GPT-4o, Claude 3.5 Sonnet, or equivalent). Set response_format to json_schema with the expected output shape: per-dimension alignment metrics, systematic bias flags, confidence intervals, and a severity summary. If the output fails schema validation, retry once with the validation errors appended as [PREVIOUS_ERRORS]. If the second attempt fails, log the failure and escalate to a human reviewer rather than silently accepting a malformed calibration report. Do not retry more than twice—calibration data is too sensitive for guesswork.

Post-processing and routing. After validation, extract the per-dimension alignment metrics and compare them against [DRIFT_THRESHOLDS]. If any dimension exceeds the threshold, route the calibration result to the judge instruction rewriting prompt for targeted fixes. If systematic over-scoring or under-scoring is detected, flag the judge for severity analysis. Log every calibration run with a timestamp, batch ID, judge version, and alignment scores to enable drift monitoring over time windows. Store raw outputs for auditability.

Human review gates. Calibration is high-stakes evaluation infrastructure. If the confidence interval width on any dimension exceeds [MAX_ACCEPTABLE_WIDTH], or if the alignment metrics show a statistically significant shift from the previous calibration window, require human review before updating judge instructions or score thresholds. The human reviewer should see the calibration report, the raw example pairs, and the proposed actions. Automate the routing but not the decision.