LLM Judge Score Inconsistency Investigation Prompt

This prompt is for eval engineers and AI reliability teams who have detected that their LLM judge—the model used to automatically score outputs against a rubric—is producing inconsistent, contradictory, or uncalibrated scores. The job-to-be-done is not to fix a single failing output, but to diagnose the judge itself. You need to determine if the judge is misinterpreting the rubric, drifting over time, or failing on specific input patterns. This is a meta-evaluation task: you are evaluating the evaluator. The required context includes a sample of scored outputs, the evaluation rubric, and, critically, a set of human-verified ground-truth scores for the same sample. Without human review as a baseline, you cannot measure judge inconsistency, only judge confidence.

Use this prompt when you have quantitative evidence of a problem, such as a sudden drop in judge-human score correlation, high variance in scores for similar inputs, or a specific failure mode like the judge consistently over-penalizing safe refusals. The prompt is designed to produce a structured Judge Reliability Report, not a simple pass/fail verdict. It will guide the analysis to compare judge scores against human scores, identify specific rubric criteria where interpretation drift is occurring, and flag individual cases that require recalibration. Do not use this prompt for a one-off output review, for evaluating the application's performance directly, or when you lack a human-reviewed baseline. It is a diagnostic tool for the evaluation pipeline itself, and its output should feed directly into a rubric refinement or judge recalibration workflow.

The workflow is high-risk because decisions based on a faulty judge can hide real regressions or trigger false alarms. Therefore, the prompt's output must be treated as a hypothesis, not a final verdict. The implementation harness should log the full report and require a human to review the flagged cases and the proposed interpretation drift before any changes are made to the evaluation rubric or the judge's system prompt. The next step after generating this report is to take the specific, flagged examples and use them to rewrite the ambiguous parts of the rubric or to add few-shot calibration examples to the judge's configuration.

This prompt template is designed to be copied directly into your evaluation pipeline or a manual review interface. It instructs the model to act as a meta-evaluator, systematically comparing the original LLM judge's scores and reasoning against a human reviewer's assessment and the intended evaluation rubric. The goal is to produce a structured report that identifies specific inconsistencies, hypothesizes root causes such as rubric misinterpretation or scale drift, and flags cases where the judge requires recalibration.

textCopy
You are an expert evaluation auditor. Your task is to diagnose inconsistencies between an LLM judge's evaluation and a human reviewer's assessment for a given AI-generated output.

## INPUT DATA

**Original Output (to be evaluated):**
[OUTPUT_TEXT]

**Evaluation Rubric:**
[RUBRIC_DEFINITION]

**LLM Judge's Evaluation:**
Score: [JUDGE_SCORE]
Reasoning: [JUDGE_REASONING]

**Human Reviewer's Evaluation:**
Score: [HUMAN_SCORE]
Reasoning: [HUMAN_REASONING]

## TASK

1.  **Score Comparison:** Quantify the score delta. Is the discrepancy within an acceptable tolerance defined by [TOLERANCE_THRESHOLD]?
2.  **Reasoning Alignment:** Compare the judge's reasoning with the human's reasoning. Identify specific factual disagreements, omissions, or weighting differences.
3.  **Rubric Adherence Check:** For both the judge and the human, check if their score and reasoning are logically consistent with the provided rubric's criteria. Note any misinterpretations.
4.  **Root-Cause Hypothesis:** If an inconsistency exists, formulate a hypothesis for its cause. Choose from the following categories:
    - **Rubric Interpretation Drift:** The judge consistently applies a stricter or more lenient interpretation of a specific rubric criterion.
    - **Scale Calibration Error:** The judge's score does not map correctly to the rubric's scale (e.g., a '3' on a 1-5 scale is used for a passing grade when it should be a '4').
    - **Factual Oversight:** The judge missed a key fact in the output that the human caught, or vice versa.
    - **Ambiguous Rubric Criterion:** The rubric criterion itself is vague, leading to valid but divergent interpretations.
    - **No Significant Inconsistency:** The discrepancy is minor and attributable to subjective judgment within acceptable bounds.
5.  **Recalibration Flag:** Set a boolean flag indicating whether this case suggests the LLM judge needs recalibration or the rubric needs revision.

## OUTPUT FORMAT

Return your analysis as a single JSON object with the following schema:
{
  "score_delta": number,
  "within_tolerance": boolean,
  "reasoning_alignment_issues": [string],
  "rubric_adherence_notes": {
    "judge": string,
    "human": string
  },
  "root_cause_hypothesis": string,
  "recalibration_needed": boolean,
  "recommended_action": string
}

## CONSTRAINTS
- Base your analysis strictly on the provided text, rubric, and evaluations. Do not introduce outside knowledge.
- If the rubric is ambiguous, state this explicitly in the root cause hypothesis.
- The recommended action must be concrete, such as "Add a clarifying example to rubric criterion X" or "Adjust judge's system prompt to enforce stricter adherence to the score scale."

To adapt this prompt, replace the square-bracket placeholders with data from your evaluation store. [OUTPUT_TEXT] is the AI-generated content under review. [RUBRIC_DEFINITION] should contain the full text of the evaluation criteria given to the LLM judge. [TOLERANCE_THRESHOLD] is a numeric value representing the maximum acceptable score difference before an investigation is triggered (e.g., 0.5 on a 1-5 scale). For high-stakes domains like healthcare or finance, always route the final recalibration_needed flag and recommended_action for human approval before modifying any production evaluation pipeline.

This prompt is designed to run as a batch investigation step, not a real-time eval component. Wire it into your eval pipeline after a scheduled judge run produces a batch of scores. The trigger should be a statistical threshold: when the judge's score distribution shifts beyond a configured tolerance, or when a human spot-check sample reveals disagreement above a set rate, queue the inconsistent traces for this investigation prompt. Do not run this prompt on every eval—it is a diagnostic tool for when the judge itself is suspect.

The implementation requires three inputs fed into the [JUDGE_TRACES] placeholder: (1) the judge's score and reasoning for each case, (2) the human reviewer's score and notes for the same cases, and (3) the original model output and eval rubric used. Structure these as a JSON array of objects with fields case_id, judge_score, judge_reasoning, human_score, human_notes, model_output, and rubric. Before calling the LLM, validate that every case has both a judge and human score present—missing pairs will produce unreliable analysis. Log the input payload hash so you can trace which batch triggered the investigation.

Model choice matters. Use a model with strong reasoning capabilities and a large context window (e.g., GPT-4o, Claude 3.5 Sonnet) because the prompt processes multiple full-length outputs and reasoning chains simultaneously. Set temperature=0 for deterministic analysis. Implement a retry with backoff if the output fails to parse as valid JSON matching the expected schema. If the model returns findings that don't map to the provided case IDs, flag the entire run for human review rather than silently accepting hallucinated case references.

Human review is mandatory before acting on the output. The prompt produces a judge_reliability_report with severity classifications and recalibration recommendations, but these are hypotheses—not automated decisions. Route critical findings (judge reliability score below threshold, systematic rubric misinterpretation) to an eval engineer for manual verification. Only after human confirmation should you update the judge's rubric, adjust score calibration, or exclude the judge from specific eval dimensions. Store the full investigation report alongside the original traces for auditability.

What to avoid: Do not feed this prompt live production traffic. Do not use it to automatically override judge scores—it diagnoses the judge, it does not replace it. Do not skip the input validation step; malformed or incomplete judge-human pairs will produce confident-sounding but incorrect analysis. If your human review sample is smaller than 20 cases, the statistical claims in the output will be unreliable—flag this limitation in your logging and treat the report as directional rather than conclusive.