LLM Arbitration Judge Prompt Template

This prompt is for system designers and platform engineers who need a final arbiter when two or more specialized agents produce conflicting outputs. Instead of picking the first response or averaging results, this prompt instructs an LLM to act as a judge: it reviews agent positions, weighs evidence, calibrates confidence, and produces a reasoned selection with a full audit trail. Use this when agent disagreements are high-stakes, when downstream systems require a single authoritative answer, or when you need to measure and correct for judge bias over time.

The ideal deployment scenario involves asynchronous, non-real-time workflows where correctness and traceability outweigh latency. For example, a multi-agent legal document review pipeline where one agent flags a clause as high-risk and another classifies it as standard. The arbitration judge should be invoked as a post-processing step, consuming the full context of each agent's output, the shared evidence, and a predefined scoring rubric. The output must be logged immutably, including the judge's reasoning chain, so that a human auditor can later review the decision. This prompt is not a replacement for deterministic conflict resolution logic; it is a tool for semantic disagreements that cannot be resolved by schema alignment or simple voting.

Do not use this prompt for low-latency real-time flows where arbitration overhead is unacceptable, or when agent outputs are trivially mergeable without conflict. Avoid it in safety-critical control loops where a hallucinated arbitration could trigger an irreversible physical action. If the cost of a wrong arbitration is higher than the cost of escalating to a human, you should bypass this prompt entirely and route directly to a human-in-the-loop review queue. Before deploying, you must calibrate the judge against a golden dataset of human-resolved conflicts to measure accuracy, bias toward specific agent positions, and failure modes like false equivalence between strong and weak evidence.

This is the core arbitration prompt you copy into your agent orchestration layer. It accepts conflicting agent outputs, shared evidence, and a decision rubric, then produces a reasoned selection with an audit trail. The prompt is designed to be stateless and idempotent—given the same inputs, it should produce the same arbitration decision. Replace every square-bracket placeholder with live data from your agent pipeline before sending the request.

textCopy
You are an Arbitration Judge in a multi-agent system. Your job is to resolve disagreements between agent outputs and select the final answer.

## INPUTS

### Conflicting Agent Outputs
[AGENT_OUTPUTS]
<!-- Format: JSON array of objects, each with agent_id, output, confidence_score, and rationale -->

### Shared Evidence
[SHARED_EVIDENCE]
<!-- Format: JSON array of evidence objects with source_id, content, and authority_weight -->

### Decision Rubric
[DECISION_RUBRIC]
<!-- Format: JSON object with weighted criteria, evidence requirements, and decision thresholds -->

### Risk Level
[RISK_LEVEL]
<!-- One of: low, medium, high, critical -->

## OUTPUT SCHEMA
Return a JSON object with these fields:
{
  "selected_output": "The chosen agent output or a synthesized version",
  "selected_agent_id": "The agent whose output was selected, or 'synthesized'",
  "confidence": 0.0_to_1.0,
  "reasoning": {
    "criteria_scores": {"criterion_name": score},
    "evidence_alignment": "How well the selected output aligns with shared evidence",
    "dissenting_points": ["Points where other agents disagreed and why they were overruled"],
    "key_factors": ["The decisive factors that led to this selection"]
  },
  "escalation_required": true_or_false,
  "escalation_reason": "If escalation is required, explain why arbitration cannot resolve this",
  "audit_trail": [
    {"step": "Step description", "decision": "What was decided", "evidence_used": ["source_ids"]}
  ]
}

## CONSTRAINTS
[CONSTRAINTS]
<!-- Format: JSON array of constraint strings, e.g., "Prefer outputs grounded in shared evidence", "Do not select outputs with confidence below 0.6", "Escalate if agents disagree on safety-critical facts" -->

## INSTRUCTIONS
1. Parse all agent outputs and identify points of agreement and disagreement.
2. For each disagreement, consult the shared evidence to determine which agent is better supported.
3. Apply the decision rubric criteria in order of weight.
4. If no agent meets the minimum threshold from the rubric, set escalation_required to true.
5. If risk_level is 'critical', require higher evidence thresholds and prefer escalation when uncertain.
6. Document every decision step in the audit_trail.
7. Calibrate your confidence score based on evidence strength, agent agreement, and rubric alignment.
8. If you synthesize a new output from multiple agents, set selected_agent_id to 'synthesized' and explain which parts came from which agent.

Return only the JSON object. Do not include explanations outside the JSON.

Adaptation notes: Replace [AGENT_OUTPUTS] with the actual outputs from your conflicting agents, including their self-reported confidence scores and rationales. The [SHARED_EVIDENCE] placeholder should contain the evidence both agents were supposed to use—this is critical for detecting hallucinations and unsupported claims. The [DECISION_RUBRIC] defines what "good" looks like; weight criteria like factual accuracy, schema compliance, and safety alignment according to your product's priorities. The [CONSTRAINTS] array lets you inject domain-specific rules without rewriting the prompt. For high-risk domains, add constraints like "Escalate if outputs disagree on patient medication dosage" or "Reject outputs that introduce new legal obligations not present in the evidence."

What to do next: After copying this template, build a validation layer that checks the JSON schema before the arbitration result enters your application. Test the prompt with known conflict cases where you already know the correct answer. Run calibration checks to ensure the confidence scores correlate with actual correctness. If the judge consistently overrates or underrates certain agents, adjust the rubric weights or add bias-detection evals before shipping.

The arbitration judge prompt is not a standalone chatbot. It is a decision function inside a larger agent orchestration layer. In production, the judge receives structured conflict records from an upstream conflict detector, not raw agent chat logs. The harness must validate that all required fields—agent outputs, evidence references, confidence scores, and the specific points of disagreement—are present before the judge prompt is assembled. Missing context is the most common cause of arbitration failure, so the harness should reject incomplete conflict records and request resubmission from the upstream detector rather than allowing the judge to guess.

Wire the judge as a synchronous decision step with a strict timeout and a single-attempt retry on validation failure. After the judge returns a verdict, the harness must validate the output schema: a selected_output field, a reasoning block, confidence_calibration, and an audit_trail array. If the output fails schema validation, retry once with the validation errors injected into the [CONSTRAINTS] block. If the retry also fails, escalate to a human review queue with the raw conflict record and both failed judge outputs attached. For high-risk domains—healthcare, finance, legal—configure the harness to require human approval on any verdict where the judge's confidence falls below the [RISK_LEVEL] threshold, regardless of schema validity.

Log every arbitration decision with the full input conflict record, the judge's output, the schema validation result, and the final action taken (accepted, retried, escalated). This audit trail is essential for judge bias detection and calibration against human preference data. Run periodic eval batches where human reviewers score a sample of judge decisions against the same conflict records. Track agreement rate, overconfidence patterns, and any drift in the judge's selection behavior. If the judge consistently favors one agent class or one output style, recalibrate the [CONSTRAINTS] and [EXAMPLES] blocks rather than tuning the model. The harness should make these blocks configurable without changing the core prompt template.