Pairwise Judge Inconsistency Root Cause Prompt Template

This prompt is for RLHF data pipeline engineers and evaluation platform teams who observe inconsistent pairwise judgments—cases where the same two outputs receive different preference labels from the same judge or across judges. The job-to-be-done is not just detecting inconsistency but attributing it to a root cause: position bias, length bias, stylistic preference leakage, rubric ambiguity, or genuine output ambiguity. You need this when your preference data quality is blocking model training or when inter-rater agreement metrics fall below your release threshold. The ideal user has access to the original prompt, both outputs, the judge's rubric, and the inconsistent judgment records.

Do not use this prompt when you only need a simple agreement score. If your goal is to measure Cohen's kappa or raw agreement percentage, use a metrics calculation script instead. This prompt is for diagnosis, not measurement. It is also inappropriate when you lack access to the underlying judgment artifacts—the prompt needs the full judgment context, not just the final labels. For high-stakes pipelines where preference data directly shapes model behavior, always pair this prompt's output with human review of the attributed root causes before accepting the taxonomy as ground truth.

Before running this prompt, gather the pairwise judgment records that show inconsistency, the original comparison prompt, both candidate outputs, and the judge's scoring rubric. The prompt expects these as structured inputs and will produce an inconsistency taxonomy report with evidence pointers. After receiving the report, validate the attributed causes against a sample of cases with human review, then use the findings to harden your rubric, add bias mitigations, or flag genuinely ambiguous examples for exclusion from training data.

This prompt template is designed to be copied directly into your evaluation pipeline or debugging notebook. It accepts a pair of model outputs, the original evaluation criteria, and the conflicting judgments from two or more LLM judges. The template instructs the model to produce a structured root cause analysis rather than another preference vote. Use it when you observe judge flips between runs, low inter-rater agreement scores, or unexpected champion-challenger results that don't match human intuition.

textCopy
You are an evaluation auditor analyzing inconsistent pairwise judgments. Your job is to diagnose why two or more LLM judges produced different preference decisions for the same pair of outputs. Do not cast another preference vote. Instead, identify the root cause of the disagreement.

## INPUT

**Evaluation Criteria:**
[EVALUATION_CRITERIA]

**Output A:**
[OUTPUT_A]

**Output B:**
[OUTPUT_B]

**Judge 1 Decision:** [JUDGE_1_DECISION]
**Judge 1 Rationale:** [JUDGE_1_RATIONALE]

**Judge 2 Decision:** [JUDGE_2_DECISION]
**Judge 2 Rationale:** [JUDGE_2_RATIONALE]

[ADDITIONAL_JUDGES]

## TASK

Analyze the disagreement and classify the root cause into one or more of the following categories. For each category that applies, provide specific evidence from the outputs and judge rationales.

### Inconsistency Taxonomy

1. **Position Bias:** One or both judges systematically favored the first or second output regardless of quality. Evidence: judge rationale ignores content and references position, or decisions flip when output order is swapped in prior tests.

2. **Length Bias:** One or both judges equated longer output with better quality without evaluating substance. Evidence: rationale mentions length, detail, or thoroughness as primary factor without engaging with correctness or relevance.

3. **Stylistic Preference Leakage:** A judge rewarded stylistic qualities (tone, formatting, verbosity, enthusiasm) that are not part of the evaluation criteria. Evidence: rationale emphasizes style attributes absent from the criteria.

4. **Criteria Interpretation Divergence:** Judges interpreted the same evaluation criteria differently. Evidence: rationales apply different standards or weight criteria dimensions inconsistently.

5. **Genuine Ambiguity:** The two outputs are of genuinely comparable quality and reasonable evaluators could disagree. Evidence: both rationales are well-reasoned, criteria-aligned, and identify real trade-offs.

6. **Judge Error or Hallucination:** One judge made a factual error, misread an output, or hallucinated content that isn't present. Evidence: rationale references non-existent content or contradicts the provided outputs.

7. **Context or Instruction Misalignment:** One judge applied implicit assumptions, external knowledge, or unstated requirements beyond the provided criteria. Evidence: rationale introduces standards not present in the evaluation criteria.

## OUTPUT SCHEMA

Return a JSON object with this exact structure:

{
  "inconsistency_id": "string - unique identifier for this analysis",
  "primary_root_cause": "string - the single most impactful category from the taxonomy",
  "contributing_factors": [
    {
      "category": "string - taxonomy category name",
      "confidence": "number - 0.0 to 1.0",
      "evidence": "string - specific quotes or observations from judge rationales and outputs",
      "impact_assessment": "string - how this factor contributed to the disagreement"
    }
  ],
  "position_bias_indicators": {
    "detected": "boolean",
    "evidence": "string or null"
  },
  "length_bias_indicators": {
    "detected": "boolean",
    "output_a_length": "number - character or word count",
    "output_b_length": "number - character or word count",
    "evidence": "string or null"
  },
  "criteria_clarity_assessment": "string - assessment of whether the evaluation criteria itself contributed to ambiguity",
  "recommended_action": "string - one of: recalibrate_judges, clarify_criteria, add_position_randomization, accept_ambiguity, investigate_judge_error, normalize_output_length, or other",
  "action_rationale": "string - why this action addresses the root cause",
  "inter_judge_agreement_note": "string - qualitative note on whether this disagreement is systemic or isolated"
}

## CONSTRAINTS

- Do not express a preference between Output A and Output B.
- Cite specific text from judge rationales when providing evidence.
- If you cannot determine a root cause with confidence, set confidence scores low and recommend human review.
- Do not invent content not present in the inputs.
- If position bias is suspected but order wasn't randomized in this comparison, note this as a limitation.

To adapt this template for your pipeline, replace each square-bracket placeholder with live data from your evaluation runs. The [ADDITIONAL_JUDGES] placeholder can be repeated for three or more judges. If you only have two judges, remove that line. The output schema is designed to be machine-readable for automated dashboards, but the evidence and impact_assessment fields should remain human-readable so on-call engineers can quickly understand what went wrong without re-reading full outputs. For high-stakes evaluation pipelines where judge inconsistency could mask real model regressions, route cases with confidence below 0.7 or primary_root_cause of "Judge Error or Hallucination" to a human review queue before accepting the diagnosis.

This prompt is designed to operate as a diagnostic step inside an existing evaluation pipeline, not as a standalone chat interface. It expects structured input—a list of inconsistent pairwise judgments with their metadata—and produces a structured taxonomy report. The implementation harness must validate both the input shape before calling the model and the output shape before the report enters any downstream dashboard or alerting system. Because the prompt analyzes judge behavior that may itself be used to gate model releases or training data quality, the harness should treat the output as high-signal but not automatically authoritative without review.

Wire the prompt as a single-turn call with a model that supports structured output or JSON mode. Before calling the model, validate that the [INCONSISTENT_PAIRS] input contains required fields: pair_id, output_a, output_b, judge_a_preference, judge_b_preference, judge_a_rationale, judge_b_rationale, and input_context. Reject malformed inputs with a clear error before consuming tokens. After receiving the model response, validate the output against the expected schema: an inconsistency_taxonomy array where each entry has pair_id, root_cause_category (one of position_bias, length_bias, stylistic_preference_leakage, genuine_ambiguity, other), confidence_score (0.0–1.0), evidence_from_rationales, and recommended_action. If the output fails schema validation, retry once with the validation errors appended to the prompt as feedback. If the retry also fails, log the failure and escalate for human review rather than silently accepting a partial report.

Log every invocation with the input pair count, the model version, the output taxonomy distribution, and any validation failures. This logging is essential for detecting judge drift over time—if the root cause distribution suddenly shifts toward genuine_ambiguity across many runs, the problem may be in the upstream judges rather than the diagnosis prompt. For high-stakes pipelines where pairwise judgments directly affect model training data, route all low confidence diagnoses (confidence below 0.7) and any other category results to a human review queue. The prompt includes an inter_judge_agreement_metrics section in its output schema; use this to set automated alerts when agreement drops below a configured threshold. Do not use this prompt to automatically override or relabel preference data—it is a diagnostic tool, not a corrective one.