Agent Arbitration with Human-in-the-Loop Escalation Prompt Template

This prompt is designed for operations teams and platform engineers who need to insert a structured human review step when multiple agents produce conflicting, ambiguous, or high-risk outputs that automated arbitration cannot resolve with sufficient confidence. The job-to-be-done is producing a complete, self-contained handoff package that a human reviewer can act on without reading raw agent logs, reconstructing conversation context, or querying external systems. The ideal user is an SRE, ops engineer, or product manager responsible for a production agent pipeline where the cost of an incorrect autonomous decision—such as a wrong financial transaction, a misleading customer communication, or a safety-critical misclassification—exceeds the cost of human review latency. Required context includes the original user request, the conflicting agent outputs, any shared evidence or source documents the agents used, the arbitration attempt and its confidence score, and the specific decision the human must make. Without this context, the handoff package forces the reviewer to hunt for information, defeating the purpose of structured escalation.

Use this prompt when automated arbitration returns a confidence score below your production threshold, when the conflict involves a high-risk domain (finance, healthcare, safety, legal), when the disagreement is structural rather than superficial (e.g., agents propose incompatible schemas or mutually exclusive actions), or when your system's policy requires human approval for certain action types regardless of confidence. The prompt is also appropriate when the arbitration judge itself flags uncertainty, when the cost of reversal is high, or when you need an audit trail showing that a human reviewed the decision. Do not use this prompt for low-stakes disagreements where a simple vote or confidence-weighted merge is sufficient, when the review SLA is shorter than the time required for a human to read the package, or when the conflict is purely cosmetic (e.g., wording differences with no operational impact). Over-escalating creates reviewer fatigue and slows down the pipeline without improving outcomes.

Before wiring this prompt into production, define your escalation thresholds explicitly: minimum arbitration confidence, maximum acceptable risk level for autonomous decisions, and the list of action types that always require human review. Instrument the handoff with latency tracking, reviewer decision logging, and periodic audits to check whether escalated decisions actually change outcomes—if human reviewers consistently rubber-stamp the top agent's output, your thresholds may be too conservative or your handoff package may not surface the real decision tension. The next step after reading this section is to review the prompt template and adapt the escalation criteria to match your system's risk policy and operational SLAs.

This prompt template is designed to be the final step in your automated arbitration pipeline. When your primary arbitration logic fails to resolve a conflict between agents—whether due to low confidence scores, irreconcilable factual differences, or a risk level that exceeds your autonomous threshold—this prompt assembles a structured handoff package. It takes the raw, conflicting outputs and transforms them into a concise brief for a human reviewer, ensuring they have the context needed to make a fast, informed decision without digging through logs.

textCopy
You are an escalation handler for a multi-agent system. Your task is to prepare a structured review package for a human operator when automated arbitration has failed. Do not make a decision. Synthesize the conflict clearly and neutrally.

[CONTEXT]

[CONFLICT_SUMMARY]

[AGENT_OUTPUTS]

[EVIDENCE_ALIGNMENT]

[CONSTRAINTS]
- Do not resolve the conflict. Your role is to prepare the information for a human reviewer.
- Preserve all direct quotes and citations from the source agents.
- Explicitly state where agents agree and where they diverge.
- If any agent output is missing required fields, flag it as incomplete.
- Adhere strictly to the [OUTPUT_SCHEMA].

[OUTPUT_SCHEMA]
{
  "escalation_id": "string, unique identifier for this review instance",
  "conflict_type": "string, classification of the disagreement (e.g., factual, structural, policy)",
  "risk_level": "string, one of: low, medium, high, critical",
  "summary": "string, a 2-3 sentence neutral summary of the conflict and why it requires human review",
  "agent_positions": [
    {
      "agent_id": "string",
      "position": "string, the core claim or output of this agent",
      "confidence": "number, the agent's self-reported confidence score",
      "key_evidence": ["string, a cited piece of evidence supporting this position"]
    }
  ],
  "points_of_agreement": ["string, a fact or output field where all agents concur"],
  "points_of_contention": ["string, a specific fact or field where agents disagree"],
  "pending_decision": "string, the single, clear question the human reviewer must answer",
  "decision_options": ["string, a discrete option the reviewer can choose"],
  "irreconcilable_flag": "boolean, true if no automated resolution path remains"
}

[RISK_LEVEL]

To adapt this template, wire it into your escalation handler as a fallback. The [CONFLICT_SUMMARY] should be a pre-processed, concise description of the disagreement generated by your conflict detection step. The [AGENT_OUTPUTS] placeholder should contain the raw, unmodified outputs from each conflicting agent. The [EVIDENCE_ALIGNMENT] section is critical for high-stakes domains; it should map each agent's claims to the specific source documents or data they used. Before sending the final package to a human, validate the generated JSON against the [OUTPUT_SCHEMA] to ensure no required fields are missing, which would defeat the purpose of a structured handoff.

This prompt is designed to be the final step in an agent arbitration pipeline before a human is involved. It should only be triggered when upstream conflict detection and automated arbitration have failed to produce a high-confidence resolution. The orchestration layer must catch the arbitration_failed or low_confidence signal from the LLM Arbitration Judge and route it here. The primary job of this harness is to transform raw agent logs and conflicting outputs into a structured, actionable review packet for a human operator, ensuring they have everything they need to make a decision without reading raw trace logs.

The implementation requires a pre-processing step to assemble the [CONFLICT_SUMMARY], [AGENT_POSITIONS], and [SHARED_EVIDENCE] variables. The [CONFLICT_SUMMARY] should be a concise, machine-generated summary of the disagreement, including the specific claims or fields in conflict. The [AGENT_POSITIONS] must be a structured list, mapping each agent's name to its final output, self-reported confidence score, and the key evidence it cited. The [SHARED_EVIDENCE] block should contain the raw source text or data that all agents had access to, deduplicated and formatted for human readability. After the LLM generates the handoff package, a strict post-processing validator must parse the output to confirm it contains all required sections: decision_required, conflict_summary, agent_positions, evidence_alignment, and reviewer_prompt. If the validator fails, the system should retry the prompt once with a more explicit formatting instruction before escalating to a human with a raw data dump.

For high-stakes domains, the human review interface should log the reviewer's identity, decision timestamp, and final selection. This audit trail must be stored alongside the generated handoff package. Avoid using this prompt for low-severity, reversible decisions where the cost of human latency outweighs the risk of an automated fallback. In those cases, a safer default action (e.g., choosing the most conservative agent output) should be taken automatically, and this escalation path should be reserved for actions that are high-risk, irreversible, or regulated. The model choice for this prompt should favor high-instruction-following models with strong structured output capabilities, as format compliance is critical for the downstream review queue.