Agent Escalation Confidence Threshold Prompt Template

This prompt is for reliability engineers and AI platform builders who need autonomous agents to make structured, auditable decisions about when to escalate a task instead of proceeding with low-confidence output. The core job-to-be-done is producing a machine-readable escalation decision that includes a numeric confidence score, a comparison against a configurable threshold, and a recommended action (proceed, escalate to a specialized agent, or hand off to a human). This is not a general-purpose 'should I trust the AI' prompt—it is a narrow, operational component designed to sit inside an agent execution loop where every autonomous action carries a risk of silent failure.

Use this prompt when you have an agent that produces a result plus some internal signal of uncertainty, and you need a consistent policy enforcement point before that result is surfaced to a user, written to a database, or passed to another agent. The ideal user is someone who can wire the prompt into an agent harness with access to the agent's raw output, any tool-call results, and the original user request. Required context includes the agent's proposed output, a structured representation of the evidence or reasoning chain, and a pre-defined confidence threshold calibrated to your application's tolerance for false positives. Do not use this prompt when the underlying model cannot produce meaningful confidence estimates, when the cost of escalation exceeds the cost of an error, or when the decision requires real-time human judgment that cannot be pre-encoded in a threshold.

Before adopting this prompt, you must commit to threshold calibration and ongoing evaluation. A static threshold that is never tested against production outcomes will drift. You should plan to log every escalation decision, compare it against ground-truth outcomes where available, and adjust the threshold based on observed false-positive and false-negative rates. If your application domain is regulated, high-stakes, or safety-critical, this prompt is a necessary but insufficient control—pair it with human review for all escalated cases and maintain an audit trail of every threshold comparison. The next section provides the copy-ready template you will adapt and embed in your agent harness.

This prompt template is the core instruction set you will inject into an agent's system message or a dedicated escalation step. It is designed to be stateless and deterministic: given the same [INPUT], [CONTEXT], and [CONSTRAINTS], it should produce a consistent JSON decision. The template forces the model to separate its raw confidence from the organizational threshold, preventing the common failure mode where an agent either overestimates its ability or escalates prematurely due to vague uncertainty language.

textCopy
You are an escalation decision agent. Your only job is to decide whether the current task should be escalated based on a confidence threshold. You do not perform the task itself.

## INPUT
[INPUT]

## CONTEXT
[CONTEXT]

## CONSTRAINTS
- Confidence Threshold: [CONFIDENCE_THRESHOLD] (a float between 0.0 and 1.0)
- Risk Level: [RISK_LEVEL] (LOW, MEDIUM, HIGH, CRITICAL)
- Available Escalation Targets: [ESCALATION_TARGETS]
- Max Retries Before Escalation: [MAX_RETRIES]

## OUTPUT_SCHEMA
You must respond with a single JSON object matching this schema:
{
  "decision": "ESCALATE" | "PROCEED" | "RETRY",
  "confidence_score": <float 0.0-1.0>,
  "threshold": <float>,
  "rationale": "<one sentence explaining the score>",
  "escalation_target": "<target name or null if PROCEED/RETRY>",
  "risk_factors": ["<list of specific risk factors considered>"]
}

## DECISION RULES
1. If confidence_score < threshold, decision must be "ESCALATE" or "RETRY".
2. If risk_level is "CRITICAL" and confidence_score < 0.95, decision must be "ESCALATE".
3. If decision is "RETRY", you must specify what additional information or tool output would increase confidence.
4. Never invent confidence. If the input is ambiguous or context is missing, reflect that in a lower score.
5. If you cannot determine a reasonable confidence_score, set it to 0.0 and escalate.

## EXAMPLES
[EXAMPLES]

## TOOLS AVAILABLE
[TOOLS]

To adapt this template, start by replacing the placeholders with your production values. The [CONFIDENCE_THRESHOLD] should come from your evaluation harness, not a guess—run calibration tests against a golden dataset of known-correct escalations to find the threshold that balances false positives and false negatives for your specific agent and task. The [EXAMPLES] block is critical: include at least three few-shot examples showing a clear PROCEED, a clear ESCALATE, and a borderline RETRY case. Without these, the model will default to its own internal calibration, which may not match your operational risk tolerance. Wire the output into a validator that checks the JSON schema, confirms the decision rule logic (e.g., if risk is CRITICAL and score is below 0.95, decision must be ESCALATE), and logs any mismatch for review.

This prompt is designed to sit inside a decision node within an agent orchestration framework. When an agent completes a task and produces an output, the system calls this prompt with the agent's output, its self-reported confidence, the task context, and the configured escalation threshold. The prompt returns a structured decision: escalate or proceed. The harness must treat this as a deterministic gate, not an advisory note. If the decision is escalate, the harness must block the output from reaching the user or downstream agent and route it to the designated escalation target immediately.

Validation and retry logic is critical. Parse the model's JSON output and validate that confidence_score is a float between 0.0 and 1.0, threshold matches the value you passed in, and decision is exactly escalate or proceed. If parsing fails or fields are missing, retry once with the same input and a stricter instruction appended: You MUST return valid JSON matching the schema exactly. If the retry also fails, default to escalate and log the failure as an escalation_harness_parse_error. Never default to proceed on a harness failure—this is a safety-critical path. For high-risk domains, route parse-failure escalations to a human review queue with the raw model output attached.

Model choice and latency matter. This prompt is a classification task with structured output. Use a fast, cost-effective model (e.g., GPT-4o-mini, Claude Haiku, or a fine-tuned small model) rather than a large reasoning model. The prompt should complete in under 500ms at P95. If latency exceeds this, consider caching the system prompt prefix or moving to a local model for this specific gate. Logging must capture: the agent ID, task ID, confidence score, threshold, decision, model used, latency, and whether the harness overrode the decision due to a parse failure. These logs feed directly into the threshold calibration eval loop described in the Evaluation Criteria section.

Human review integration depends on your risk posture. For low-risk workflows, escalate may route to a fallback agent or a different model. For high-risk workflows (healthcare, finance, safety), escalate must create a review task in your human-in-the-loop queue with the full task context, agent output, confidence score, and the reason string from the prompt. The harness should track time-to-review and escalate again if the human review SLA is breached. Do not use this prompt as a substitute for domain-specific safety classifiers; it is a general confidence gate, not a content safety filter. Combine it with separate policy-enforcement prompts where compliance is required.