Human Rating Reconciliation Prompt for LLM Judge Tuning

This prompt is for annotation teams and evaluation engineers who have identified a disagreement between one or more human raters and an LLM judge on a specific scored item. Instead of treating the disagreement as a simple error to discard, this prompt performs a structured reconciliation analysis. It determines whether the gap originated from a judge error, a human rater error, rubric ambiguity, or an edge case in the input. The output includes actionable rubric clarification suggestions and concrete judge instruction patches that can be applied immediately. Use this prompt when you need to diagnose and fix systematic misalignment, not just log disagreement rates.

The ideal user is an evaluation lead or senior annotator who understands the scoring rubric deeply and can verify the reconciliation output before applying changes. Required context includes the original input that was scored, the human rater's score and justification, the LLM judge's score and justification, the full scoring rubric, and the current judge instructions. Without all of these inputs, the reconciliation analysis will be speculative rather than diagnostic. The prompt works best on disagreements that recur across multiple items—single anomalies may not justify the analysis cost.

Do not use this prompt for real-time scoring pipelines or high-throughput evaluation where latency is critical. This is a diagnostic and tuning tool, not a replacement for your primary judge. It is also not suitable when the disagreement stems from fundamentally different scoring philosophies that require stakeholder alignment rather than prompt engineering. If your human raters disagree with each other at high rates, fix inter-rater reliability before attempting human-judge reconciliation. For ongoing monitoring, pair this prompt with the Score Drift Detection Prompt and the Judge Instruction Rewriting Prompt to close the loop from detection to repair.

This prompt template is designed to be pasted directly into your reconciliation workflow. It instructs the model to analyze a specific disagreement between a human rater and an LLM judge, systematically diagnosing whether the gap stems from judge error, human error, or rubric ambiguity. The output includes concrete suggestions for clarifying the rubric and patching the judge's instructions, making it a critical tool for evaluation platform teams who need to maintain alignment at scale.

textCopy
You are an expert evaluation auditor. Your task is to reconcile a disagreement between a human rater and an LLM judge for a specific evaluation sample.

## INPUT

**Evaluation Rubric:**
[RUBRIC_DEFINITION]

**LLM Judge Instructions:**
[JUDGE_INSTRUCTIONS]

**Sample to Evaluate:**
[SAMPLE_INPUT]

**Model Output Being Evaluated:**
[MODEL_OUTPUT]

**Human Rater Score and Justification:**
Score: [HUMAN_SCORE]
Justification: [HUMAN_JUSTIFICATION]

**LLM Judge Score and Justification:**
Score: [JUDGE_SCORE]
Justification: [JUDGE_JUSTIFICATION]

## TASK

Analyze the disagreement and produce a structured reconciliation report. Your analysis must:
1. Identify the root cause category for the disagreement: "Judge Error", "Human Error", "Rubric Ambiguity", or "Input Edge Case".
2. Extract the specific evidence from both justifications that supports your root cause determination.
3. If rubric ambiguity is detected, propose a concrete clarification to the rubric text.
4. If judge error is detected, propose a specific patch to the judge instructions that would prevent this error in the future.
5. Assign a confidence level (High, Medium, Low) to your root cause determination.

## OUTPUT SCHEMA

Return a JSON object with the following structure:
{
  "reconciliation_id": "string",
  "root_cause_category": "Judge Error" | "Human Error" | "Rubric Ambiguity" | "Input Edge Case",
  "root_cause_confidence": "High" | "Medium" | "Low",
  "disagreement_summary": "A one-sentence description of the core disagreement.",
  "evidence_analysis": {
    "human_rater_key_points": ["string"],
    "llm_judge_key_points": ["string"],
    "divergence_point": "The specific claim or interpretation where the two diverge."
  },
  "rubric_clarification": {
    "needed": true | false,
    "original_text": "The ambiguous rubric text, if applicable.",
    "proposed_revision": "The clarified rubric text, if applicable."
  },
  "judge_instruction_patch": {
    "needed": true | false,
    "original_instruction": "The instruction that led to the error, if applicable.",
    "proposed_patch": "The revised instruction, if applicable.",
    "rationale": "Why this patch addresses the root cause."
  },
  "recommended_action": "A clear next step for the evaluation team."
}

## CONSTRAINTS

- Do not assume the LLM judge is always wrong. Human raters make mistakes, and rubrics can be genuinely ambiguous.
- Cite specific text from the justifications, rubric, and sample to ground your analysis.
- If you cannot determine the root cause with confidence, say so and recommend gathering additional human annotations.
- The rubric clarification and judge instruction patch must be actionable and specific. Do not suggest vague improvements like "be more careful."

Adaptation Guidance: Replace all square-bracket placeholders with actual values before sending to the model. The [RUBRIC_DEFINITION] should include the full scoring scale and dimension descriptions. The [JUDGE_INSTRUCTIONS] should be the exact system prompt or instructions given to the LLM judge. For batch reconciliation, iterate over a set of disagreements and aggregate the root_cause_category and recommended_action fields to identify systematic issues. If your evaluation pipeline uses multiple judges, include the judge identifier in the reconciliation_id field for traceability.

Validation and Risk Controls: Before applying any proposed_patch to production judge instructions, run the patched judge against a held-out calibration set to verify that the fix does not regress other scoring dimensions. For high-stakes evaluation pipelines, require a human review of the reconciliation report before accepting the root cause determination. Log all reconciliation outputs with timestamps and judge versions to maintain an audit trail of alignment decisions over time.

The Human Rating Reconciliation Prompt is not a one-shot analysis tool—it is a pipeline component that sits between annotation rounds and judge tuning cycles. The typical integration pattern: after a batch of items receives both human ratings and LLM judge scores, the reconciliation prompt processes each disagreement above a configurable threshold (e.g., score delta > 1 on a 5-point scale). The prompt output feeds directly into rubric revision decisions and judge instruction patches. Wire this into an existing annotation platform by triggering reconciliation runs when inter-rater agreement between human and judge falls below a target threshold, or on a scheduled cadence (weekly for active evaluation pipelines, per-release for CI/CD-gated systems).

Input assembly requires three data sources per reconciliation item: the original content or model output being evaluated, the human rating with annotator ID and optional justification, and the LLM judge score with its full reasoning trace. Package these into the [INPUT] placeholder as a structured JSON block rather than free text to prevent parsing ambiguity. The [RUBRIC] placeholder should contain the exact scoring rubric the judge was instructed to follow, including dimension definitions and score-level descriptions. The [JUDGE_INSTRUCTIONS] placeholder needs the complete system prompt or instruction set the judge received—partial instructions will cause the reconciliation analysis to miss instruction-rooted gaps. Validation layer: before calling the model, validate that all required fields are present, that score values are within the rubric's defined range, and that the human and judge scores actually disagree. If they agree, skip the reconciliation call and log a no-op to avoid wasting inference on aligned ratings.

Model choice and retry strategy: Use a model with strong reasoning capabilities (GPT-4, Claude 3.5 Sonnet, or equivalent) because the task requires multi-hypothesis analysis and structured root cause attribution. Set temperature to 0 or near-zero for deterministic output. Implement a retry wrapper that catches malformed JSON output, missing required fields in the reconciliation schema, or root cause categories outside the allowed enum. Maximum two retries with the error message appended to the prompt as feedback. After two failures, flag the item for human review rather than silently accepting a partial analysis. Logging and audit trail: persist the full reconciliation output alongside the original ratings, the judge's reasoning, and the model's confidence scores. This creates an auditable record for rubric change decisions and helps detect if the reconciliation prompt itself drifts over time.

Human review integration: The reconciliation output includes a requires_human_review flag and a recommended_action field. Route items where recommended_action is escalate_to_annotation_lead or ambiguous_root_cause to a review queue. For items marked rubric_clarification_needed, aggregate them across batches and present to the rubric owner as a prioritized list of ambiguous criteria with suggested wording changes. Downstream automation: when the reconciliation identifies a clear judge instruction gap (root_cause: judge_instruction_error), the suggested_instruction_patch field can feed directly into a judge instruction update workflow—but always require human approval before applying patches to production judges. A single misapplied instruction patch can silently degrade evaluation quality across the entire pipeline. What to avoid: do not use this prompt to automatically override human ratings or judge scores. The reconciliation analysis informs tuning decisions; it does not make them. Do not run reconciliation on fewer than 10 disagreement examples per batch—small samples produce unreliable root cause patterns. Do not skip the validation layer; malformed inputs produce reconciliation outputs that look plausible but attribute errors to the wrong source.