Score Discrepancy Root Cause Analysis Prompt

Use this prompt when an LLM judge produces a score that diverges from an expected reference score, and you need to understand why. This playbook is for evaluation platform engineers, MLOps teams, and AI quality leads who operate automated scoring pipelines. The prompt performs a structured root cause analysis, categorizing the discrepancy into one of four causes: judge error, rubric ambiguity, input edge case, or reference label noise. It extracts supporting evidence from the judge's own justification and the input context, producing a machine-readable diagnosis you can log, aggregate, and act on.

This prompt is a diagnostic tool for offline debugging and continuous improvement of your evaluation infrastructure. It expects you to provide the original input that was scored, the LLM judge's score and full justification, the expected reference score, and the evaluation rubric that was used. The output is a structured JSON object containing the root cause category, a confidence score, extracted evidence snippets, and a concise explanation. Wire this into your evaluation pipeline as a post-scoring analysis step that runs only when score deltas exceed a configured threshold. Log the structured output to your observability platform so you can track recurring failure patterns across judges, rubrics, and input types over time.

Do not use this prompt for real-time scoring decisions or as a substitute for judge calibration. It is not designed to correct scores on the fly or to override a judge's output in production. It is also not a replacement for inter-rater reliability metrics or systematic calibration workflows. If you find that a particular root cause category dominates your diagnostics, the correct next step is to address the systemic issue—retrain the judge, clarify the rubric, expand your edge-case test suite, or audit your reference labels—rather than running this prompt on every sample.

This template is designed to be pasted directly into your evaluation debugging harness. It forces the model to act as a diagnostic analyst rather than a judge, focusing on explaining the gap between two scores rather than re-evaluating the output. Replace every square-bracket placeholder with the specific instance you are investigating. The prompt is structured to extract concrete evidence from the judge's own justification and the input context, preventing the common failure mode where the analysis itself becomes a new, equally opaque judgment.

textCopy
You are a diagnostic analyst investigating a scoring discrepancy. Your job is to explain why an LLM judge's score differs from an expected score, not to re-score the output. You must ground every finding in specific evidence from the provided materials.

## INPUTS
- **Original Prompt:** [ORIGINAL_PROMPT]
- **Model Output Under Evaluation:** [MODEL_OUTPUT]
- **Evaluation Rubric:** [RUBRIC]
- **LLM Judge Score:** [JUDGE_SCORE]
- **LLM Judge Justification:** [JUDGE_JUSTIFICATION]
- **Expected Score:** [EXPECTED_SCORE]
- **Expected Score Rationale (optional):** [EXPECTED_RATIONALE]
- **Input Metadata (optional):** [INPUT_METADATA]

## OUTPUT SCHEMA
Produce a JSON object with the following structure:
{
  "discrepancy_summary": {
    "judge_score": number,
    "expected_score": number,
    "absolute_difference": number,
    "direction": "judge_higher" | "judge_lower" | "match",
    "significant": boolean
  },
  "root_cause_categories": [
    {
      "category": "judge_error" | "rubric_ambiguity" | "input_edge_case" | "reference_label_noise" | "other",
      "confidence": "high" | "medium" | "low",
      "evidence": ["string of specific evidence from judge justification or input context"],
      "explanation": "Clear reasoning linking evidence to this category."
    }
  ],
  "evidence_analysis": {
    "judge_justification_key_claims": ["extracted claims the judge made"],
    "claims_contradicting_rubric": ["claims that appear to misapply the rubric"],
    "claims_contradicting_expected_rationale": ["claims that conflict with expected reasoning"],
    "input_factors_judge_may_have_missed": ["context elements the judge overlooked"]
  },
  "rubric_ambiguity_analysis": {
    "ambiguous_criteria": ["rubric dimensions open to interpretation"],
    "suggested_clarifications": ["concrete wording changes to reduce ambiguity"]
  },
  "recommended_action": {
    "primary_action": "fix_judge_instructions" | "clarify_rubric" | "add_edge_case_example" | "review_reference_label" | "investigate_further" | "no_action",
    "rationale": "Why this action addresses the most likely root cause.",
    "urgency": "critical" | "high" | "medium" | "low"
  }
}

## CONSTRAINTS
- Do NOT produce a new score for the model output.
- Every root cause category MUST cite specific evidence from the judge justification or input context.
- If the judge justification is missing or empty, flag this as a judge_error with high confidence.
- Treat the expected score as ground truth for this analysis; do not question it unless you have direct evidence of label noise.
- If multiple root causes are present, list them in order of likely contribution to the discrepancy.
- If the scores match (difference of 0), set significant to false and explain why no analysis is needed.

To adapt this template, start by populating the required fields from your evaluation logs. The [JUDGE_JUSTIFICATION] field is critical—without it, the analysis defaults to flagging a judge error. If you have multiple judges or a human rationale for the expected score, include those in [EXPECTED_RATIONALE] to enable richer contradiction analysis. For production pipelines, wrap this prompt in a validation layer that checks the output JSON schema before accepting the analysis. If the model returns a recommended_action.urgency of critical or high, route the case for human review before applying any automated fixes to your judge instructions or rubric. Avoid using this prompt on cases where the score difference is trivially small unless you are specifically investigating cumulative drift across many low-difference cases.

This prompt is designed to sit inside an evaluation debugging loop, not as a standalone chat interaction. When an LLM judge produces a score that diverges from an expected value—whether that expected value comes from a human annotator, a golden dataset, or a peer judge—this prompt receives the judge's output, justification, input context, and the expected score, then returns a structured root cause classification. The typical integration point is a post-scoring hook in your evaluation pipeline: after scores are collected, any discrepancy exceeding a configured threshold triggers this analysis prompt before the result is written to your evaluation store or surfaced to an analyst.

Wire the prompt into your application with a discrepancy detection gate first. Compare the judge's score against the reference score using a tolerance window (e.g., absolute difference > 1 on a 5-point scale, or categorical mismatch for pass/fail). Only invoke this prompt when a discrepancy is detected—running it on aligned scores wastes tokens and adds noise. Pass the full judge response object, the input that was scored, the reference score and its source (human label, golden answer, peer judge), and any rubric or scoring guidelines that were in effect. The prompt returns a JSON object with root_cause_category (one of judge_error, rubric_ambiguity, input_edge_case, reference_label_noise, or multi_factor), a confidence score, extracted evidence from the judge's justification, and a brief diagnostic narrative. Parse this JSON in your application code—do not treat the output as display text.

Validation and retry logic is critical because this prompt itself can fail. Validate that the returned JSON contains a recognized root_cause_category enum value. If the category is missing, malformed, or the confidence is below your threshold (we recommend 0.7 as a starting floor), retry once with the same inputs. If the second attempt also fails validation, escalate to a human review queue with the raw judge output and discrepancy details attached. Log every invocation—input hash, judge ID, discrepancy magnitude, root cause category, confidence, and whether human review was required—so you can track whether the root cause analyzer itself drifts over time or shows systematic blind spots.

Model choice matters. This prompt requires careful reasoning over multiple hypotheses, so prefer models with strong instruction-following and structured output capabilities. Claude 3.5 Sonnet, GPT-4o, and Gemini 1.5 Pro are all suitable. Avoid smaller or faster models (Claude Haiku, GPT-4o-mini, Gemini Flash) for this task unless you've validated their root cause accuracy against a labeled discrepancy dataset—they tend to default to rubric_ambiguity without sufficient evidence extraction. If latency is a concern, run this prompt asynchronously after scores are collected rather than blocking the scoring pipeline. The output should be written to your evaluation database alongside the original score for later aggregation and trend analysis.

When to escalate instead of trusting the prompt: if the discrepancy magnitude is extreme (e.g., judge scored 1, reference is 5), if the input contains sensitive or regulated content, or if the root cause analysis will trigger an automated action like judge instruction rewrites or calibration set changes, require human review of the analysis before acting on it. This prompt is a diagnostic aid, not an automated remediation system. Use its output to prioritize which judges, rubrics, or input types need human attention—don't close the loop automatically without verifying that the diagnosis is correct across a sample of cases.