Disagreement Resolution Protocol Prompt Template

This prompt is designed for evaluation infrastructure teams who have already deployed multiple LLM judges and are now facing a production reality: judges disagree. When inter-rater reliability metrics like Cohen's Kappa or percent agreement fall below a defined threshold on a specific item, you need an automated, auditable way to resolve that conflict. The job-to-be-done is not just to pick a winner, but to execute a structured resolution protocol that classifies the severity of the disagreement, invokes the correct escalation path (arbitration judge, human review, or tie-breaking rule), and produces a documented rationale suitable for an audit trail. The ideal user is an MLOps engineer or evaluation lead wiring this prompt into an automated scoring pipeline where manual review of every disagreement is impossible at scale.

Use this prompt when you have a fleet of judges producing scores on a shared rubric, and you need a programmatic way to handle divergence. This is appropriate for high-stakes evaluation workflows—model release gates, RLHF data quality pipelines, or compliance checks—where unresolved disagreements can corrupt downstream metrics. The prompt requires several structured inputs: the original item being evaluated, the conflicting scores and rationales from each judge, the evaluation rubric, and a pre-defined disagreement severity classification schema. Do not use this prompt as a substitute for fixing a broken rubric or a poorly calibrated judge. If your judges systematically disagree on entire categories of items, the root cause is likely rubric ambiguity or judge bias, and you should use the Disagreement Root Cause Analysis or Rubric Clarity Diagnostic prompts first. This protocol is for resolving individual disputes within an otherwise healthy evaluation system.

The prompt is designed to be one step in a larger automated harness. It expects that you have already detected a disagreement via a separate monitoring step (such as the Inter-Rater Agreement Score Calculation prompt). The output is a structured resolution action, not a casual recommendation. Before deploying, you must define clear severity thresholds that map to actions: a minor disagreement might trigger an arbitration judge, while a critical disagreement on a safety-related item must force human review. The prompt includes placeholders for these thresholds and for the identity of the arbitration judge. After implementing, validate the protocol by running it against a golden dataset of known disagreements with expected resolutions, and log every resolution action with the full input context to maintain an audit trail for governance reviews.

This template is the core resolution engine for your disagreement protocol. It accepts a set of conflicting judge scores, their rationales, and the item under evaluation, then produces a structured resolution action. The prompt is designed to be invoked automatically when inter-rater agreement falls below your configured threshold, making it suitable for direct wiring into an evaluation harness rather than manual copy-paste use.

textCopy
You are an arbitration judge resolving scoring disagreements between multiple LLM evaluators. Your task is to review conflicting scores, analyze the divergence, and produce a binding resolution with clear rationale.

## INPUT
- Item Under Evaluation: [ITEM_TO_EVALUATE]
- Evaluation Criteria: [EVALUATION_CRITERIA]
- Scoring Rubric: [SCORING_RUBRIC]
- Judge Scores and Rationales:
[JUDGE_SCORES_AND_RATIONALES]
- Agreement Threshold: [AGREEMENT_THRESHOLD]
- Current Agreement Level: [CURRENT_AGREEMENT_LEVEL]

## TASK
1. Classify the disagreement severity as LOW, MEDIUM, HIGH, or CRITICAL based on score spread, criteria importance, and agreement gap.
2. Determine the resolution action: ARBITRATE, ESCALATE_TO_HUMAN, INVOKE_TIEBREAKER, or FLAG_FOR_RUBRIC_REVIEW.
3. If ARBITRATE, produce a final score with detailed justification addressing each judge's position.
4. If ESCALATE_TO_HUMAN, specify which criteria require human judgment and why automated resolution is insufficient.
5. If INVOKE_TIEBREAKER, specify the tie-breaking rule applied and why it resolves the specific disagreement pattern.
6. If FLAG_FOR_RUBRIC_REVIEW, identify the ambiguous rubric sections causing divergence with suggested clarifications.

## CONSTRAINTS
- Do not default to averaging scores unless the rubric explicitly supports numeric averaging.
- When judges cite different evidence from the item, verify each citation before weighting.
- If any judge's rationale contradicts the item content, flag it as a potential hallucination.
- For HIGH or CRITICAL severity, include a confidence score (0.0-1.0) for your resolution.
- Preserve all original judge scores and rationales in the output for audit trail integrity.

## OUTPUT SCHEMA
Return valid JSON matching this structure:
{
  "resolution_id": "string",
  "severity_classification": "LOW|MEDIUM|HIGH|CRITICAL",
  "severity_rationale": "string explaining the classification",
  "resolution_action": "ARBITRATE|ESCALATE_TO_HUMAN|INVOKE_TIEBREAKER|FLAG_FOR_RUBRIC_REVIEW",
  "resolution_details": {
    "final_score": "number or null if not arbitrated",
    "final_score_justification": "string or null",
    "escalation_criteria": ["array of criteria needing human review or null"],
    "tiebreaker_rule_applied": "string or null",
    "rubric_ambiguities": ["array of problematic rubric sections or null"]
  },
  "judge_position_analysis": [
    {
      "judge_id": "string",
      "original_score": "number",
      "position_assessment": "string",
      "hallucination_flag": "boolean"
    }
  ],
  "resolution_confidence": "number between 0.0 and 1.0 or null",
  "audit_trail": {
    "original_scores_preserved": "boolean",
    "resolution_timestamp": "string",
    "arbitration_judge_version": "[ARBITRATION_JUDGE_VERSION]"
  }
}

## EXAMPLES
[FEW_SHOT_EXAMPLES]

## RISK LEVEL
[RISK_LEVEL]

Adapt this template by replacing each square-bracket placeholder with values from your evaluation pipeline. The [JUDGE_SCORES_AND_RATIONALES] placeholder expects structured input containing each judge's ID, score, and full rationale text—format this as a JSON array or markdown table before insertion. The [FEW_SHOT_EXAMPLES] section is critical for production reliability: include at least two examples showing correct ARBITRATE resolutions and one example each for ESCALATE_TO_HUMAN and FLAG_FOR_RUBRIC_REVIEW paths. For high-stakes domains, set [RISK_LEVEL] to HIGH, which should trigger additional validation of the arbitration judge's output before it becomes binding. The [ARBITRATION_JUDGE_VERSION] token should be populated from your model registry to maintain traceability across resolution events.

This prompt is designed to be called as a secondary arbitration step, not a standalone workflow. It should only be invoked when a primary inter-rater agreement check (such as Cohen's Kappa or Fleiss' Kappa) falls below a configured threshold for a specific evaluation item. The harness must supply the original item, the scores and rationales from each disagreeing judge, the evaluation rubric, and the acceptable agreement threshold that was violated. Without this context, the protocol cannot produce a meaningful resolution action.

Wire the prompt into an evaluation pipeline as a post-scoring hook. After collecting judge scores, compute pairwise or fleet-wide agreement. For items where agreement < [AGREEMENT_THRESHOLD], call this prompt with the disagreement payload. Validate the output against a strict schema: resolution_action must be one of arbitration_judge, human_review, majority_vote, or tie_break; severity must be low, medium, high, or critical; rationale must be a non-empty string; and audit_trail must include the original scores, the disagreement metric, and the resolution logic. If the output fails schema validation, retry once with a stricter format reminder. If it fails again, escalate to human review and log the failure for prompt debugging.

Log every resolution to an audit table with the item ID, judge IDs, original scores, disagreement metric, resolution action, severity, and the full resolution rationale. This audit trail is critical for downstream trust—teams need to trace why an item was escalated or auto-resolved. For high-severity disagreements, always require human review regardless of the model's suggested action. The prompt can recommend human_review, but the harness should enforce it for high and critical severity classifications. Avoid using this protocol for real-time, user-facing decisions without a human-in-the-loop gate; it is designed for offline evaluation pipelines where latency is tolerable and correctness matters more than speed.