Cross-Model Judge Agreement Comparison Prompt Template

This prompt is for evaluation infrastructure teams who need to determine whether scores from different LLM judges—such as GPT-4, Claude, Gemini, or open-weight models—are interchangeable. The primary job-to-be-done is producing a statistical agreement analysis that answers one question: 'If I swap Judge A for Judge B, will my evaluation results mean the same thing?' Use this prompt when you are migrating between judge models, running a multi-model evaluation fleet, or auditing whether your judge ensemble produces consistent results. The prompt assumes you have already collected raw scores from each judge on a shared set of evaluation items. It does not help you design the scoring rubric, calibrate a single judge, or evaluate the underlying task outputs directly; those belong to separate calibration and scoring playbooks.

The ideal user is an AI engineer or evaluation lead who already has a score matrix—rows of evaluation items, columns of judges, cells containing numeric or categorical scores—and needs to quantify agreement before making a production change. The prompt requires you to supply the raw score data, judge identifiers, score scale metadata (e.g., 1-5 Likert, binary pass/fail, categorical labels), and any known judge characteristics such as model version or temperature settings. The output is not a single number but a structured agreement report containing a pairwise agreement matrix, model-specific bias profiles, and a migration risk assessment that flags which judge pairs are safe to swap and which require further calibration. This is a diagnostic tool, not a decision automator; the migration risk assessment should inform human judgment, not replace it.

Do not use this prompt when you have only one judge, when you are evaluating the quality of the task outputs themselves, or when your judges use incompatible scoring scales without normalization. If your judges produce free-text rationales without structured scores, use the Judge Rationale Consistency Check prompt first to extract structured scores before running agreement analysis. If you suspect a specific judge is an outlier, pair this prompt with the Outlier Judge Detection prompt to isolate problematic judges before computing fleet-wide agreement. For production environments where agreement is monitored continuously, integrate the output schema from this prompt into the Continuous Monitoring Pipeline prompt to automate agreement threshold alerts. The next section provides the copy-ready prompt template you can adapt to your judge fleet.

This prompt template is designed to be deployed inside an evaluation harness that sends identical scoring requests to multiple LLM judges—such as GPT-4, Claude, Gemini, or open-weight models—and collects their outputs for comparison. The template expects a set of items to evaluate, a scoring rubric, and a list of judge model identifiers. It returns a structured agreement matrix, per-judge bias profiles, and a migration risk assessment that tells you whether swapping one judge for another would break your score continuity. Use this when you are building or maintaining an automated evaluation pipeline and need to know whether your judge fleet produces interchangeable results.

textCopy
You are an evaluation meta-analyst. Your task is to compare scores assigned by multiple LLM judges to the same set of evaluation items and produce a structured agreement analysis.

## INPUTS
- Evaluation Items: [ITEMS]
  Each item includes:
  - item_id: string
  - input: the original prompt or conversation that was evaluated
  - output: the model-generated response that was scored
  - reference_answer: (optional) ground-truth or reference answer if available
- Judge Scores: [JUDGE_SCORES]
  A mapping of judge_model_id -> list of score objects, where each score object contains:
  - item_id: string
  - score: number or categorical value
  - rationale: string (the judge's explanation for the score)
  - confidence: number (0.0 to 1.0, optional)
- Scoring Rubric: [RUBRIC]
  Description of the scoring scale, criteria, and what each score level means.
- Comparison Configuration: [COMPARISON_CONFIG]
  - agreement_metric: one of ["cohens_kappa", "fleiss_kappa", "percent_agreement", "krippendorff_alpha", "all"]
  - agreement_threshold: number (minimum acceptable agreement, e.g., 0.7)
  - bias_checks: list of bias dimensions to analyze, from ["leniency_severity", "position_bias", "length_bias", "confidence_calibration"]
  - migration_risk_threshold: number (maximum acceptable score distribution shift for judge replacement, e.g., 0.1)

## OUTPUT SCHEMA
Return a JSON object with this exact structure:
{
  "agreement_matrix": {
    "metric_used": string,
    "pairwise_agreement": {
      "[judge_a]_vs_[judge_b]": {
        "agreement_score": number,
        "interpretation": "high" | "moderate" | "low" | "poor",
        "sample_size": number,
        "confidence_interval_95": [number, number]
      }
    },
    "overall_fleet_agreement": number,
    "agreement_pass": boolean
  },
  "bias_profiles": [
    {
      "judge_model_id": string,
      "leniency_score": number,
      "severity_score": number,
      "position_bias_detected": boolean,
      "length_bias_detected": boolean,
      "confidence_calibration_error": number | null,
      "bias_flags": [string]
    }
  ],
  "migration_risk_assessment": {
    "replacement_candidates": [
      {
        "source_judge": string,
        "target_judge": string,
        "score_distribution_shift": number,
        "risk_level": "low" | "medium" | "high" | "critical",
        "recommendation": string,
        "requires_recalibration": boolean
      }
    ],
    "overall_migration_readiness": "ready" | "caution" | "not_ready"
  },
  "divergence_diagnostics": [
    {
      "item_id": string,
      "judge_pair": string,
      "score_difference": number,
      "possible_cause": string,
      "evidence_excerpts": [string]
    }
  ],
  "summary": string
}

## CONSTRAINTS
- Do not fabricate agreement scores. Only compute from the provided [JUDGE_SCORES] data.
- If fewer than 10 items are provided, flag this in the summary and note that agreement metrics have wide confidence intervals.
- If any judge is missing scores for items that other judges scored, note the sparsity pattern in divergence_diagnostics.
- For categorical rubrics, use appropriate agreement metrics (e.g., Cohen's Kappa for two judges, Fleiss' Kappa for three or more).
- If confidence scores are not provided by all judges, skip confidence_calibration_error and note the omission.
- Flag any judge whose leniency_score or severity_score deviates more than 0.2 from the fleet mean.
- For migration_risk_assessment, a score_distribution_shift above [MIGRATION_RISK_THRESHOLD] must produce a risk_level of "high" or "critical".
- Include specific item_ids and evidence excerpts in divergence_diagnostics for any item where judges disagree by more than 1 standard deviation of the score distribution.

## EVALUATION CHECKS (run after output generation)
1. Verify that every judge_model_id in [JUDGE_SCORES] appears in bias_profiles.
2. Verify that pairwise_agreement covers all unique judge pairs.
3. Verify that agreement_pass is true only if overall_fleet_agreement >= [AGREEMENT_THRESHOLD].
4. Verify that no fabricated item_ids appear in divergence_diagnostics.
5. If sample_size < 30, verify that confidence_interval_95 is appropriately wide.

To adapt this template, replace the square-bracket placeholders with your actual evaluation data. The [ITEMS] array should contain the prompts and outputs your judges scored. The [JUDGE_SCORES] mapping should include every judge you want to compare, with their scores and rationales aligned by item_id. Set [RUBRIC] to the exact scoring instructions your judges received—this context helps the meta-analyst interpret divergence. Configure [COMPARISON_CONFIG] with your preferred agreement metric and thresholds. After running the prompt, apply the evaluation checks listed at the bottom of the template to validate the output before trusting the agreement matrix. For high-stakes evaluation pipelines where judge agreement directly affects product decisions, always run this analysis on a held-out calibration set before promoting any judge replacement.

This harness transforms the cross-model judge agreement prompt from a one-off experiment into a repeatable, auditable pipeline. The core workflow collects scores from multiple LLM judges on the same set of evaluation items, then feeds those scores into the agreement analysis prompt. The harness must handle model invocation, response parsing, score matrix construction, and statistical validation before the agreement prompt ever runs. Without this scaffolding, teams risk comparing scores collected under different conditions, with different parsing rules, or after silent model failures—producing agreement metrics that look precise but are meaningless.

The implementation requires a score collection loop that iterates over evaluation items and target judge models. For each (item, model) pair, call the judge with a consistent rubric prompt, parse the numeric score from the response using a strict schema validator, and log the raw response alongside the parsed score. Use a retry wrapper with exponential backoff (max 3 attempts) for transient failures, but escalate to a dead-letter queue if the model returns unparseable output after retries. Store results in a structured format: {item_id, judge_model, score, timestamp, raw_response, parse_success}. Before constructing the agreement matrix, run a completeness check—if any judge has missing scores for more than 5% of items, abort and alert rather than computing agreement on sparse data. For the agreement prompt itself, assemble the score matrix as a JSON input, include metadata about which models were compared and how many items were evaluated, and pass it to a single high-capability model (GPT-4 or Claude 3.5 Sonnet) for the analysis. Validate the output against an expected schema containing agreement_matrix, bias_profiles, migration_risk_assessment, and statistical_equivalence_flags.

After receiving the agreement analysis, run statistical sanity checks outside the LLM: compute Cohen's Kappa and percent agreement independently using a deterministic library, and compare against the LLM's reported values. If the LLM's numbers deviate by more than 0.05, flag the analysis for human review—the model may have hallucinated the statistics. Log every run with a unique comparison_id, the prompt version hash, the score matrix, and the full analysis output. For high-stakes decisions like judge replacement or evaluation pipeline migration, require human sign-off on the agreement report before acting on the LLM's recommendations. Avoid running this pipeline on fewer than 50 evaluation items; small samples produce unstable agreement estimates that can mislead teams into trusting or distrusting judges based on noise.