Arbitration Prompt for Conflicting Sub-Task Outputs

This prompt is designed for evaluation engineers and orchestration system builders who face a specific failure mode in multi-agent pipelines: two or more sub-agents produce outputs that individually pass validation but are mutually incompatible. The prompt acts as an arbitrator, comparing evidence, weighing confidence scores, applying tie-breaking rules, and producing a single reconciled output with a traceable decision record. Use this when your orchestrator detects a conflict between validated sub-task results and needs a structured resolution before passing data to downstream agents or end users.

The ideal deployment context is a production pipeline where sub-agents operate under strict input/output contracts and each output has already passed its own validation gate. The orchestrator should invoke this arbitration prompt only after detecting a semantic conflict—not a formatting or schema mismatch. For example, if Agent A extracts a contract effective date of '2024-03-15' with 0.92 confidence and Agent B extracts '2024-03-22' with 0.88 confidence, both pass date-format validation but cannot both be correct. The arbitration prompt receives both outputs, their evidence chains, confidence scores, and any tie-breaking rules (e.g., prefer the agent with access to the original signed document over the one using OCR text). It must produce a single reconciled date and a decision record explaining why one was chosen over the other.

Do not use this prompt for simple majority voting across more than two agents—that pattern is better served by a lightweight aggregation function. Do not use it for conflicts that should escalate to a human immediately, such as safety-critical determinations, regulatory findings, or decisions where the cost of a wrong arbitration exceeds the cost of re-executing both sub-tasks with different instructions. Do not use it when the conflict stems from ambiguous or missing input data rather than agent disagreement; in that case, a clarification request to the user or upstream system is the correct path. If your pipeline lacks per-output confidence scores or evidence traces, add those before attempting automated arbitration, because the prompt's reasoning quality depends entirely on the quality of the evidence it receives.

This prompt template is designed to be injected into your orchestrator's arbitration step when two or more sub-task outputs pass individual validation but produce incompatible results. The template forces the model to compare evidence, weight confidence, apply tie-breaking rules, and produce a single reconciled output rather than simply picking one answer or averaging incompatible values. Use it as the core of your conflict resolution harness, populating the placeholders with the specific conflicting outputs, their validation results, and your domain's resolution rules.

textCopy
You are an arbitration agent responsible for resolving conflicts between sub-task outputs that have both passed individual validation but produce incompatible results.

## CONFLICTING OUTPUTS
[OUTPUT_A]

[OUTPUT_B]

## VALIDATION RESULTS
Output A validation: [VALIDATION_RESULT_A]
Output B validation: [VALIDATION_RESULT_B]

## EVIDENCE AND SOURCES
Output A evidence: [EVIDENCE_A]
Output B evidence: [EVIDENCE_B]

## DOMAIN CONTEXT
[DOMAIN_CONTEXT]

## TIE-BREAKING RULES
Apply these rules in order when evidence and confidence are equal:
1. [TIE_BREAK_RULE_1]
2. [TIE_BREAK_RULE_2]
3. [TIE_BREAK_RULE_3]

## RESOLUTION CONSTRAINTS
- Do not average or merge incompatible values unless explicitly allowed by [MERGE_POLICY]
- If neither output can be trusted, produce an explicit abstention with reasoning
- Flag any output that appears correct but relies on hallucinated or unverifiable evidence
- Preserve uncertainty: if the resolution is low-confidence, state that explicitly

## REQUIRED OUTPUT SCHEMA
Return a JSON object with exactly these fields:
{
  "reconciled_output": <the single resolved result, or null if abstention>,
  "selected_source": "output_a" | "output_b" | "merged" | "neither",
  "confidence": <float 0.0 to 1.0>,
  "evidence_comparison": {
    "output_a_strengths": [<list of specific strengths with source references>],
    "output_b_strengths": [<list of specific strengths with source references>],
    "output_a_weaknesses": [<list of specific weaknesses or missing evidence>],
    "output_b_weaknesses": [<list of specific weaknesses or missing evidence>]
  },
  "tie_breaker_applied": <which rule was used, or null>,
  "unresolved_issues": [<list of remaining uncertainties or conflicts>],
  "recommendation": <one-sentence guidance for downstream consumers>
}

## RISK LEVEL
[RISK_LEVEL]

## ADDITIONAL INSTRUCTIONS
[CONSTRAINTS]

Adapt this template by tightening the tie-breaking rules to match your domain's actual resolution logic. If your system has access to ground truth or authoritative sources, add a [GROUND_TRUTH] placeholder and require the model to compare both outputs against it. For high-risk domains, add a human_review_required boolean to the output schema and set it to true when confidence falls below your threshold. Wire the output into a post-arbitration validator that checks schema compliance, confidence thresholds, and whether the selected source aligns with the evidence comparison before the reconciled output reaches downstream consumers.

The arbitration prompt is not a standalone module; it is a decision node inside a multi-agent orchestrator. It fires only when two or more sub-task outputs pass their individual validation contracts but produce incompatible results. The orchestrator must first confirm that each output is individually valid—schema checks pass, required fields are present, and quality thresholds are met—before invoking arbitration. If any output fails its own validation, the orchestrator should retry or escalate that sub-task instead of arbitrating. This gate prevents the arbitrator from wasting tokens on outputs that should have been rejected earlier.

Wire the prompt into an arbitration_step function that accepts a list of conflicting outputs, their associated confidence scores, the original task context, and any ground-truth or reference data available. Before calling the model, run a pre-check: verify that at least two outputs exist, that they differ on a material field (not just formatting), and that each output's sub-task ID and agent role are traceable. After the model returns a reconciled output, apply a post-validation layer: confirm the output conforms to the expected schema, that the evidence_comparison field references specific claims from each input, and that the tie_breaking_rule field explains why one output was preferred. If the arbitrator's own confidence score falls below a configured threshold (e.g., 0.7), route to human review with the full conflict package. Log the arbitration decision, the inputs, the reconciled output, and the confidence score for audit trails and regression testing.

Model choice matters here. Arbitration requires careful reasoning over conflicting evidence, so prefer models with strong comparative reasoning (e.g., Claude 3.5 Sonnet, GPT-4o). Avoid lightweight models for high-stakes domains like finance or healthcare. Set temperature low (0.0–0.2) to reduce variance in tie-breaking. Implement a retry budget: if post-validation fails (malformed JSON, missing required fields), retry once with the validation error injected into the prompt as [PREVIOUS_FAILURE]. If the second attempt fails, escalate to a human operator with the raw inputs and failure context. Never loop more than twice—silent failures in arbitration erode trust faster than a clean escalation. For production systems, pair this prompt with an eval harness that compares arbitration decisions against known ground-truth resolutions to measure arbitration accuracy over time.