Judge Replacement Candidate Evaluation Prompt Template

This prompt is designed for evaluation infrastructure teams who need to replace an existing LLM judge with a new candidate. The replacement could be a different model, an updated prompt, a new rubric, or a reconfigured judge instance. The core job is to produce an equivalence report that compares score distributions, measures agreement with the existing judge fleet, and identifies edge-case handling gaps before the switch goes live. Use this when the cost of a silent score break is high and when downstream systems depend on stable, reproducible evaluation signals.

The ideal user is an AI engineer or evaluation lead who already has a production judge fleet generating scores, a candidate judge they want to introduce, and a representative sample of items that both judges can score. You need access to historical score distributions from the existing judge, the candidate's scores on the same items, and any per-item metadata that might explain divergence (item difficulty, content length, domain category). Without this data, the prompt cannot produce a meaningful equivalence report. The prompt assumes you have already run both judges on the same evaluation set and collected their outputs—it does not orchestrate the scoring itself.

Do not use this prompt when you are designing a new evaluation rubric from scratch, when you lack historical judge data for comparison, or when the candidate judge operates on a fundamentally different scoring scale that cannot be meaningfully compared. This prompt is also inappropriate for one-off quality checks or ad-hoc spot comparisons—it is built for structured replacement decisions where the cost of getting it wrong includes broken CI/CD gates, misleading model selection decisions, or silent regression in content quality metrics. If your sample size is too small to detect meaningful differences, run a sample-size sufficiency check before invoking this prompt. For high-stakes domains like healthcare, legal, or finance evaluation, always route the equivalence report through human review before approving the judge switch.

This template is designed to be pasted directly into your evaluation harness. It instructs an LLM to act as a meta-evaluator, comparing the scoring behavior of a new candidate judge against a baseline incumbent judge across a shared set of test items. The prompt produces a structured equivalence report, not a single pass/fail score. Populate every square-bracket placeholder before each run. The prompt assumes you have already collected paired scores from both judges on the same inputs.

textCopy
You are an evaluation auditor responsible for assessing whether a candidate LLM judge can replace an incumbent judge in a production evaluation pipeline. Your task is to produce a structured equivalence report comparing the two judges.

## INPUT DATA
- Incumbent Judge Scores: [INCUMBENT_SCORES]
- Candidate Judge Scores: [CANDIDATE_SCORES]
- Score Scale: [SCORE_SCALE]
- Evaluation Rubric Used: [RUBRIC_TEXT]
- Test Item Metadata (optional): [ITEM_METADATA]

## OUTPUT SCHEMA
Return a JSON object with the following structure:
{
  "agreement_metrics": {
    "percent_agreement": number,
    "cohens_kappa": number,
    "pearson_correlation": number,
    "mean_absolute_error": number
  },
  "distribution_comparison": {
    "incumbent_mean": number,
    "candidate_mean": number,
    "mean_shift_detected": boolean,
    "variance_shift_detected": boolean,
    "distribution_notes": string
  },
  "edge_case_analysis": {
    "high_disagreement_items": [
      {
        "item_index": number,
        "incumbent_score": number,
        "candidate_score": number,
        "possible_cause": string
      }
    ],
    "systematic_bias_flags": [string]
  },
  "replacement_risk_assessment": {
    "risk_level": "LOW" | "MEDIUM" | "HIGH" | "CRITICAL",
    "backward_compatibility_concerns": [string],
    "recommended_actions": [string]
  },
  "confidence_in_assessment": "LOW" | "MEDIUM" | "HIGH",
  "requires_human_review": boolean
}

## CONSTRAINTS
- Do not invent scores or fabricate agreement metrics. Use only the provided input data.
- If the sample size is fewer than 30 paired items, set confidence_in_assessment to "LOW" and flag this in recommended_actions.
- Flag any systematic bias where the candidate consistently scores higher or lower than the incumbent by more than [BIAS_THRESHOLD] points on average.
- For high_disagreement_items, include only items where the absolute score difference exceeds [DISAGREEMENT_THRESHOLD].
- If the rubric contains ordinal scales, use Cohen's Kappa with quadratic weighting. If continuous, use Pearson correlation.
- Set requires_human_review to true if risk_level is "HIGH" or "CRITICAL", or if any systematic_bias_flags are present.

## EVALUATION INSTRUCTIONS
1. Compute all agreement metrics from the paired scores.
2. Compare score distributions for mean shift and variance shift.
3. Identify items with the largest score discrepancies and hypothesize causes based on the rubric and item metadata.
4. Assess replacement risk considering: score continuity, backward compatibility with historical evaluations, and edge-case handling parity.
5. If [HISTORICAL_BASELINE_SCORES] are provided, compare the candidate's score distribution against the historical baseline to detect regression risk.

## RISK LEVEL
[RISK_LEVEL]

Adaptation guidance: Replace [INCUMBENT_SCORES] and [CANDIDATE_SCORES] with arrays of paired numeric scores. The [SCORE_SCALE] placeholder should describe the range and type (e.g., '1-5 Likert' or '0.0-1.0 continuous'). If you have historical score distributions from the incumbent judge across a larger dataset, populate [HISTORICAL_BASELINE_SCORES] to enable regression detection. Set [BIAS_THRESHOLD] and [DISAGREEMENT_THRESHOLD] based on your tolerance—typical starting values are 0.5 for Likert scales and 0.15 for continuous scales. For high-stakes evaluation pipelines where score continuity is critical, set [RISK_LEVEL] to "HIGH" to instruct the model to apply stricter thresholds and require human review for any flagged divergence.

Before deploying this prompt into an automated pipeline, validate the output schema against your downstream consumers. A common failure mode is the model returning risk_level as a lowercase string or adding extra fields not in the schema. Implement a JSON schema validator in your harness that rejects malformed responses and triggers a retry with the error message appended. For production use, log every equivalence report alongside the input score pairs so you can audit replacement decisions later. If the prompt flags requires_human_review: true, route the report to a human evaluator before proceeding with judge replacement.

This prompt is designed to be a gate in your CI/CD or evaluation pipeline, not a one-off manual check. The core workflow involves collecting a representative sample of scored items from your production judge fleet, running both the incumbent judge and the replacement candidate on the same items, and then feeding the paired score distributions into this prompt for analysis. The output is a structured equivalence report that can be parsed by downstream automation to either approve the replacement, flag it for human review, or reject it outright based on configurable thresholds for score continuity, agreement, and edge-case parity.

To implement this in a pipeline, first define a sampling strategy that captures your production distribution: stratify by score range, input length, topic cluster, and known edge cases. Collect at least 200-500 paired judgments to achieve statistical significance for the agreement metrics. Store the raw scores, rationales, and metadata in a structured format (JSON Lines works well) so the prompt's [SCORE_DATASET] placeholder can be populated programmatically. Before invoking the LLM, run a pre-validation step to ensure the dataset meets minimum size requirements, contains both judge columns, and has no missing scores that would skew the comparison. If the dataset fails pre-validation, abort and alert the pipeline operator rather than sending incomplete data to the model.

The prompt expects a [REGRESSION_THRESHOLD] parameter—this is your team's policy decision, not a statistical output. Set this threshold based on your risk tolerance: a strict threshold (e.g., 0.95 agreement, <0.05 mean score drift) is appropriate for high-stakes evaluation surfaces like safety grading or compliance checks; a looser threshold (e.g., 0.85 agreement) may be acceptable for internal quality scoring where some drift is tolerable. Wire this threshold into your pipeline configuration so it can be adjusted per use case without modifying the prompt template. After the LLM returns the equivalence report, run a post-validation parser that extracts the structured fields—equivalence_decision, agreement_score, drift_magnitude, edge_case_parity, and risk_flags—and compares them against your threshold. If the decision is CONDITIONAL or REJECT, automatically escalate to a human review queue with the full report attached.

For retry and error handling, implement a simple retry loop: if the LLM response fails schema validation (missing required fields, malformed JSON, or incoherent decision values), retry up to two times with a slightly lower temperature setting (0.1 → 0.0) to reduce output variance. If all retries fail, log the raw response and escalate for manual triage. For observability, log every invocation with the prompt version hash, model used, dataset size, threshold value, and the final decision. This creates an audit trail that lets you answer 'why was this judge replaced?' months later. Finally, integrate this prompt into your regression test suite: before any judge replacement goes live, run a back-compatibility check where the new judge's scores on a golden dataset of 50-100 historically scored items are compared against the original scores, and only proceed if the equivalence report passes the same threshold gates.

Avoid running this prompt on tiny sample sizes (under 50 items) or on datasets with severe class imbalance where one score dominates—the agreement metrics will be misleadingly high due to prevalence. Also avoid treating the LLM's equivalence decision as the final word without human review when the risk_flags field contains DISTRIBUTION_SHIFT or EDGE_CASE_REVERSAL warnings. These flags indicate that even if aggregate metrics look acceptable, the replacement judge is systematically different on important subsets, and a human should inspect those specific cases before approving the swap.