Inter-Rater Disagreement Root Cause Analysis Prompt Template

This prompt is a diagnostic instrument for evaluation platform teams who operate multiple LLM judges and have already observed unacceptable disagreement in their scoring pipeline. The job-to-be-done is not to calculate agreement metrics—you should have already computed Cohen's Kappa, pairwise agreement percentages, or Krippendorff's Alpha before reaching for this prompt. The job is to explain why the judges disagree. You need a structured root cause analysis that tells you whether the problem lives in the rubric (ambiguous criteria, missing edge-case guidance), the judges (model-specific bias, inconsistent calibration, position effects), or the data (genuinely ambiguous items, adversarial examples, distribution shift from the rubric's design range). The ideal user is an evaluation infrastructure engineer or ML engineer who owns judge quality and can act on calibration recommendations.

Use this prompt after a batch scoring run where at least two judges have produced scores on the same items and your agreement metrics have crossed a concerning threshold—typically Cohen's Kappa below 0.6 or pairwise agreement below 80%, though your team's tolerance may vary. Feed the prompt the items, the scores from each judge, the rubric that was used, and any relevant metadata about the judges (model version, temperature, system prompt). The output you should expect is a disagreement taxonomy that classifies each divergent case into categories such as rubric ambiguity, judge leniency bias, genuine edge case, or position effect, followed by calibration recommendations like rubric clarification, judge re-anchoring, or exclusion rules for ambiguous items. This is an offline diagnostic workflow, not a real-time scoring component. Do not wire this prompt into a production scoring pipeline; it is designed for periodic evaluation health checks, perhaps run weekly or after any judge model or rubric update.

Do not use this prompt when you have only a single judge, when you haven't yet computed agreement statistics, or when you need a real-time arbitration mechanism for conflicting scores. This prompt analyzes patterns across a batch; it is not a tiebreaker for individual items. Also avoid it when your judges use fundamentally different scoring scales or rubrics—the analysis assumes a shared evaluation framework. If you skip the prerequisite agreement computation, you risk feeding the prompt data where disagreement is actually within acceptable bounds, wasting analysis tokens on noise. Before running this prompt, ensure you have: (1) a batch of scored items with judge identifiers, (2) the exact rubric text each judge was given, (3) computed agreement metrics to confirm a problem exists, and (4) a human reviewer available to act on the calibration recommendations. The next step after receiving the output is to prioritize the most frequent disagreement category and implement the corresponding fix, then re-run the scoring batch and re-measure agreement to verify improvement.

This prompt template is designed for evaluation platform teams who need to move beyond simple agreement metrics and understand why judges diverge. It accepts the original input, the outputs from two judges, their scores, and the evaluation rubric. The model is instructed to classify the disagreement into a structured taxonomy—rubric ambiguity, judge bias, genuine edge case, or noise—and to produce actionable calibration recommendations. Use this when pairwise agreement drops below a threshold or when you are onboarding a new judge and need to align it with an existing baseline.

textCopy
You are an expert evaluation auditor. Your task is to analyze a scoring disagreement between two LLM judges and determine the root cause.

## INPUT DATA
- **Original User Prompt:** [ORIGINAL_PROMPT]
- **Model Output Under Evaluation:** [MODEL_OUTPUT]
- **Evaluation Rubric:** [RUBRIC_TEXT]
- **Judge A Score and Justification:** [JUDGE_A_OUTPUT]
- **Judge B Score and Justification:** [JUDGE_B_OUTPUT]

## TASK
1.  **Classify the Disagreement:** Assign the primary root cause from the taxonomy below.
    - `RUBRIC_AMBIGUITY`: The rubric is vague, contradictory, or missing criteria for this case.
    - `JUDGE_BIAS`: One judge exhibits a systematic bias (e.g., length bias, stylistic preference, position bias).
    - `GENUINE_EDGE_CASE`: The model output is a legitimate borderline case where reasonable raters can disagree.
    - `JUDGE_NOISE`: The disagreement appears random, inconsistent, or unjustified by either the rubric or the output.
2.  **Provide Evidence:** Quote the specific text from the rubric, justifications, or model output that supports your classification.
3.  **Recommend Calibration:** Suggest a concrete action to resolve the disagreement.
    - For `RUBRIC_AMBIGUITY`: Propose a clarification to the rubric text.
    - For `JUDGE_BIAS`: Describe the bias pattern and suggest a countermeasure (e.g., de-biasing instruction, position swapping).
    - For `GENUINE_EDGE_CASE`: Recommend a tie-breaking policy or a human review flag.
    - For `JUDGE_NOISE`: Suggest a temperature adjustment or a consistency check (e.g., majority vote with a third judge).

## OUTPUT FORMAT
Return a single JSON object with the following keys:
- `disagreement_category` (string): One of the four taxonomy values.
- `evidence_summary` (string): A concise explanation with direct quotes from the input data.
- `calibration_recommendation` (string): A specific, actionable step to improve alignment.
- `confidence_score` (float): Your confidence in this root cause analysis, from 0.0 to 1.0.

## CONSTRAINTS
- Do not re-score the model output. Your job is to analyze the *judges' behavior*.
- If the rubric is missing entirely, classify as `RUBRIC_AMBIGUITY`.
- Base your analysis strictly on the provided text. Do not introduce external knowledge about the domain.

To adapt this template, replace the square-bracket placeholders with your actual data. The [RUBRIC_TEXT] should be the exact instructions given to your judges, as even small wording differences are often the root cause of divergence. The [JUDGE_A_OUTPUT] and [JUDGE_B_OUTPUT] fields must include both the numeric or categorical score and the judge's written justification, as the evidence for bias or noise is typically found in the justification, not the score itself. For high-stakes evaluation pipelines, always route outputs classified as GENUINE_EDGE_CASE to a human review queue rather than accepting an automated tie-break. Before deploying this prompt, run it against a golden dataset of known disagreement causes to verify that the taxonomy classification matches your team's internal definitions.

This prompt is designed to be called programmatically as part of an automated evaluation debugging pipeline, not as a one-off chat interaction. The typical trigger is a scheduled job or a webhook that fires when inter-rater agreement metrics (such as Cohen's Kappa or raw agreement percentage) drop below a configured threshold for a specific rubric or judge pair. The harness should collect the raw scoring data, format it into the [DISAGREEMENT_DATA] input block, and call the LLM. The output is a structured disagreement taxonomy that downstream systems can use to flag rubric items for revision, quarantine a biased judge, or escalate edge cases for human review.

Input assembly: The [DISAGREEMENT_DATA] placeholder expects a JSON array of scoring records, each containing item_id, judge_a_score, judge_b_score, rubric_item, item_text, and optional judge_a_rationale and judge_b_rationale. The harness should query your eval database for all items where two judges disagreed within the target time window. Model choice: Use a model with strong reasoning capabilities (such as GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro) because the task requires comparing rationales and inferring root causes. Avoid smaller or faster models for this analysis; misclassification of disagreement types will erode trust in the entire judge fleet. Validation: After receiving the output, validate that the JSON conforms to the expected schema—particularly that every disagreement_category value is from the allowed enum (rubric_ambiguity, judge_bias, genuine_edge_case, scale_misalignment, rationale_contradiction, unknown). Reject and retry once if the schema is invalid or if the confidence field is missing for any entry.

Logging and observability: Log every analysis run as an event with the input disagreement count, the output taxonomy distribution, the model used, and the latency. Attach the full prompt and response to your trace store (e.g., LangSmith, Braintrust, or your internal logging system). This is critical because disagreement root cause analysis is itself a judgment call—you need to be able to audit the analyzer's decisions later. Human review gate: For any disagreement classified as genuine_edge_case with confidence below 0.8, route the item to a human review queue. These are the cases most likely to reveal gaps in your evaluation rubric or genuine ambiguity in your task definition. Do not auto-apply calibration adjustments based solely on the LLM's output; always require human sign-off before modifying judge prompts, retraining judge models, or updating rubric definitions.

Integration points: Wire the output into your rubric management system to flag items with high rubric_ambiguity rates for revision. Feed judge_bias signals into your judge calibration dashboard. Use scale_misalignment detections to trigger a score normalization pass. The recommended_action field in each output record should map to an automated workflow—for example, 'revise_rubric' creates a ticket in your rubric backlog, 'recalibrate_judge' triggers a calibration run against human-annotated samples, and 'escalate_for_review' opens a human review task. Avoid the temptation to close the loop fully automatically; disagreement analysis is diagnostic, not prescriptive, and acting on incorrect root cause labels will compound errors across your evaluation pipeline.