Judge Alignment with Human Gold Standard Prompt Template

This prompt is for evaluation infrastructure teams who need to measure how well an LLM judge's scores align with human-annotated ground truth. Use it when you have a calibration dataset containing model outputs paired with human quality ratings, and you need a structured alignment report that quantifies agreement, identifies systematic bias, and flags annotation quality concerns. The core job-to-be-done is auditing the judge itself—not scoring individual outputs. Run this before deploying a judge into production, after rubric changes, or when human reviewers report that automated scores feel wrong.

The prompt requires a calibration dataset with at least 30 scored items per rubric criterion to produce statistically meaningful alignment metrics. Each item must include the original model output, the human-assigned score, and optional human rationale. The prompt template accepts placeholders for [CALIBRATION_DATA], [RUBRIC_DEFINITION], [SCORING_SCALE], and [CONFIDENCE_LEVEL]. It produces a structured report with agreement coefficients, per-criterion breakdowns, systematic bias flags, and annotation quality warnings. Do not use this prompt when you have fewer than 20 scored items, when human annotations lack inter-rater reliability checks, or when the rubric itself is still in draft form—alignment metrics on unstable foundations produce false confidence.

This is a meta-evaluation prompt, not a scoring prompt. It does not evaluate model outputs directly. It evaluates the evaluator. If you need to score individual outputs, use a standard judge prompt from the Rubric Design and Scoring Prompts group. If you need to compare two judges against each other without human ground truth, use the Inter-Rater Agreement Score Calculation Prompt Template. Reserve this prompt for the specific workflow where human-annotated gold data exists and you need to calibrate an automated judge against it before trusting its scores in production pipelines.

This prompt template is the core instruction set you send to a meta-evaluation model. Its job is to take a batch of items—each containing a human gold-standard score, an LLM judge's score, and the original content—and produce a structured alignment report. The report includes overall agreement metrics, per-criterion breakdowns, systematic bias flags, and a confidence assessment. You will calibrate this template by replacing each square-bracket placeholder with your specific data, rubrics, and constraints before execution.

textCopy
You are an evaluation auditor. Your task is to compare LLM judge scores against human gold-standard annotations and produce a structured alignment report.

## INPUT DATA
[JUDGE_SCORES_AND_GOLD_STANDARDS]

## RUBRIC
[RUBRIC_DEFINITION]

## OUTPUT SCHEMA
Return a single JSON object with the following structure:
{
  "overall_alignment": {
    "agreement_metric": "string (e.g., 'Cohen's Kappa', 'Percent Agreement', 'Weighted Kappa')",
    "value": number,
    "interpretation": "string (e.g., 'Substantial agreement', 'Moderate agreement')",
    "confidence_interval_95": [lower_bound, upper_bound]
  },
  "per_criterion_breakdown": [
    {
      "criterion_name": "string",
      "agreement_value": number,
      "divergence_direction": "string (e.g., 'Judge scores higher than human', 'Judge scores lower than human', 'No systematic divergence')",
      "sample_size": number
    }
  ],
  "systematic_bias_flags": [
    {
      "bias_type": "string (e.g., 'Leniency bias', 'Severity bias', 'Central tendency bias')",
      "affected_criteria": ["string"],
      "evidence": "string describing the pattern observed",
      "severity": "string (e.g., 'Low', 'Medium', 'High')"
    }
  ],
  "annotation_quality_notes": [
    {
      "concern": "string (e.g., 'Low inter-annotator agreement in gold standard', 'Possible label noise in criterion X')",
      "recommendation": "string"
    }
  ],
  "sample_size_sufficiency": {
    "is_sufficient": boolean,
    "rationale": "string",
    "minimum_recommended_samples": number
  },
  "overall_assessment": "string summarizing alignment quality and primary risks"
}

## CONSTRAINTS
[CONSTRAINTS]

## EVALUATION INSTRUCTIONS
1. Calculate the overall agreement metric specified in [CONSTRAINTS] using the paired human-judge scores.
2. For each criterion in [RUBRIC_DEFINITION], compute agreement separately and identify divergence direction.
3. Scan for systematic patterns: does the judge consistently score higher or lower than humans? Does the judge avoid extreme scores? Does agreement drop for specific criteria?
4. Assess the gold-standard data itself: are there signs of annotation noise, low inter-annotator agreement, or insufficient samples?
5. Flag any sample-size concerns that make the alignment metrics unreliable.
6. If [RISK_LEVEL] is 'high', include explicit human-review recommendations and do not state conclusions as definitive without confidence intervals.

## RISK LEVEL
[RISK_LEVEL]

To adapt this template, replace [JUDGE_SCORES_AND_GOLD_STANDARDS] with a structured array of items, each containing the original content, the human gold-standard score, and the LLM judge's score. Replace [RUBRIC_DEFINITION] with your evaluation rubric's criteria names, scale definitions, and scoring guidelines. Use [CONSTRAINTS] to specify the agreement metric you want computed, any domain-specific rules, and output formatting requirements. Set [RISK_LEVEL] to 'low', 'medium', or 'high' to control the conservatism of the analysis—high-risk contexts will produce more cautious language and demand confidence intervals. After generation, validate the JSON output against the schema before ingesting it into your evaluation dashboard. For high-stakes calibration, always route the output through a human reviewer before accepting alignment conclusions.

Wiring the Judge Alignment with Human Gold Standard prompt into production requires treating it as a measurement instrument, not a one-off script. The prompt compares LLM judge scores against human-annotated ground truth and produces alignment metrics, bias reports, and per-criterion agreement breakdowns. In an application harness, this prompt should be invoked as part of a scheduled calibration job or triggered when a new judge model is introduced, a rubric changes, or a human annotation batch is completed. The harness must enforce input validation before the prompt runs: verify that the human gold standard dataset contains enough samples per criterion (typically ≥30 for meaningful agreement statistics), that score distributions are not degenerate (e.g., all items scored identically), and that annotation metadata includes annotator IDs for inter-annotator agreement baselines. Without these checks, the alignment report will produce misleading confidence intervals or mask systematic annotation quality problems.

The implementation should wrap the prompt call in a retry layer with exponential backoff for transient model API failures, but must not retry on validation failures—malformed alignment reports indicate a prompt or model problem that requires debugging, not replay. After receiving the model output, parse the alignment report into a structured schema (JSON with fields for overall agreement score, per-criterion agreement, bias flags, confidence intervals, and sample size warnings). Validate that all required fields are present, that agreement scores fall within [0,1] or the expected scale, and that confidence intervals are properly ordered (lower < point estimate < upper). Log the raw prompt, raw response, parsed report, and validation results to an evaluation audit store for traceability. If the report flags annotation quality issues (e.g., low inter-annotator agreement on the human side), escalate to a human review queue rather than silently accepting the alignment score—the judge cannot be calibrated against unreliable ground truth.

Model choice matters for this workflow. Use a model with strong reasoning and structured output capabilities (e.g., GPT-4, Claude 3.5 Sonnet) because the prompt requires statistical interpretation, bias diagnosis, and multi-criterion comparison. Avoid lightweight models that may hallucinate agreement metrics or mishandle confidence interval calculations. If the alignment report will feed into automated decision-making (e.g., blocking a judge model from production), add a human approval gate before acting on the results. The harness should also store alignment reports over time to enable drift detection: compare current alignment scores against historical baselines and alert if agreement drops below a configurable threshold (e.g., Cohen's Kappa < 0.6). Finally, treat the human gold standard dataset as a versioned asset—when annotations are updated, re-run alignment and compare results to detect annotation drift or judge sensitivity to label changes.