LLM Judge Calibration Prompt Template

This prompt is for AI quality engineers and platform teams who need to calibrate an LLM judge against human evaluators. Use it when you are building an automated evaluation harness and must prove that your model-graded scores correlate with human judgment. The prompt produces a calibrated rubric with anchor examples, scoring guidelines, and a calibration report that quantifies inter-rater reliability. Do not use this prompt for one-off output grading or for tasks where a simple string match or schema validator is sufficient. This prompt assumes you already have a set of human-scored examples ready for calibration.

The ideal user is an engineer who owns the evaluation infrastructure and needs to defend their automated metrics to stakeholders. You should use this prompt when you are onboarding a new evaluation task, when you suspect judge drift after a model upgrade, or when you need to align multiple LLM judges to a single standard. The prompt requires a calibration dataset of at least 30 human-scored examples covering clear passes, clear failures, and ambiguous boundary cases. Without this dataset, the prompt will produce a rubric that looks plausible but has no empirical grounding. You must also define the scoring dimensions upfront—whether you are measuring correctness, helpfulness, safety, or a custom axis—because the calibration process aligns the judge to those specific dimensions.

Do not use this prompt as a substitute for defining what good looks like. If your team cannot agree on scoring criteria for human evaluators, an LLM judge will only amplify that ambiguity. Also avoid this prompt when your evaluation task has a deterministic ground truth, such as code compilation or schema validation, where unit tests are cheaper and more reliable. After running calibration, you must inspect the disagreement clusters manually. The prompt will flag where the judge and humans diverge, but it cannot fix fundamentally underspecified rubrics. The next step is to take the calibrated rubric and anchor examples and wire them into your continuous evaluation pipeline, re-running calibration whenever the task definition, model, or data distribution changes.

This prompt template is the core instruction set you will send to a 'judge' LLM. Its purpose is to calibrate that judge's scoring behavior against a known human standard. The template is designed to be filled with your specific evaluation criteria, a set of anchor examples that have already been scored by a human, and a batch of unlabeled outputs that need to be scored consistently. The output is a structured calibration report that quantifies alignment and flags specific areas of disagreement.

textCopy
You are an expert LLM evaluation calibrator. Your task is to align your scoring behavior with a provided human-generated rubric and set of anchor examples. You will then score a new set of outputs and produce a calibration report.

First, internalize the following evaluation rubric. This defines the scoring dimensions and the criteria for each score level.
[RUBRIC_DEFINITION]

Next, study the following anchor examples. Each example includes an input, an output, and the correct human-assigned score with a brief justification. These are your ground truth for calibration.
[ANCHOR_EXAMPLES]

Now, evaluate the following unlabeled outputs. For each one, apply the rubric exactly as demonstrated in the anchor examples.
[UNLABELED_OUTPUTS]

For each unlabeled output, you must produce a JSON object with the following keys:
- "id": The identifier of the output from the input list.
- "score": The score you assign, strictly adhering to the rubric's scale.
- "justification": A concise explanation referencing specific parts of the output and the rubric criteria that support your score.
- "calibration_confidence": A rating of HIGH, MEDIUM, or LOW indicating how confident you are that this score aligns with the human evaluator's intent, based on the anchor examples.

Finally, produce a "calibration_summary" JSON object with the following keys:
- "overall_alignment_assessment": A qualitative summary (e.g., "Strong alignment on factual accuracy, minor drift on conciseness").
- "identified_drift_areas": A list of specific rubric dimensions or score levels where your judgments may systematically differ from the human evaluator, with evidence from your justifications.
- "recommended_rubric_clarifications": Suggestions for how the rubric text or anchor examples could be improved to resolve the identified drift.

Your entire response must be a single, valid JSON object with two top-level keys: "evaluations" (an array of output evaluation objects) and "calibration_summary". Do not include any text outside this JSON structure.

To adapt this template, start by replacing the [RUBRIC_DEFINITION] with your specific scoring dimensions, levels, and descriptive criteria. The [ANCHOR_EXAMPLES] placeholder is critical; populate it with at least 5-10 diverse examples that have been carefully scored by a human to establish a clear standard. The [UNLABELED_OUTPUTS] should be a batch of new items for the judge to score. After running the prompt, you must validate the output JSON against a strict schema. If the judge's calibration confidence is LOW on many items or the identified_drift_areas list is long, this is a signal to refine your rubric and anchor examples before using this judge in an automated eval harness. For high-stakes evaluations, always have a human review the initial calibration report to confirm the alignment assessment before promoting the judge to production use.

The calibration prompt template is not a one-off script; it is a component of an eval harness that must run deterministically, produce auditable artifacts, and detect its own degradation over time. Wire this prompt into a pipeline that accepts a set of human-annotated examples (the calibration dataset), runs the prompt to produce a rubric and scoring guidelines, then immediately validates the output against a strict schema. The harness should treat any schema violation—missing anchor examples, unscored dimensions, or ambiguous threshold language—as a calibration failure that blocks downstream use of the judge.

Build the harness with three stages: generation, validation, and self-test. In the generation stage, call the calibration prompt with a curated set of 20–50 human-scored examples covering clear passes, clear failures, and boundary cases. In the validation stage, parse the output and check that every required field exists, every anchor example includes a score and justification, and every scoring guideline references observable output characteristics rather than vague intent. In the self-test stage, run the newly calibrated judge against a held-out set of 10 examples and compare its scores to human scores. If agreement (e.g., quadratic-weighted kappa) falls below a configured threshold—typically 0.7 for moderate-stakes tasks—block the judge from promotion and flag for human review. Log the raw prompt, the full output, the validation errors, and the agreement metrics as an immutable calibration record.

For production use, store the calibrated rubric as a versioned artifact in your eval registry alongside the calibration dataset hash, the model version used for calibration, and the agreement score. Schedule recalibration when the underlying model changes, when the task definition shifts, or when drift detection signals that judge scores are diverging from human spot checks. Never deploy a judge whose calibration record is missing or whose self-test agreement fell below threshold. If the calibration prompt itself produces inconsistent rubrics across multiple runs with the same inputs, investigate temperature settings (use temperature=0 for calibration), prompt ambiguity, or insufficient diversity in the calibration dataset before escalating to a human rubric designer.