Automated Output Escalation Decision Prompt Template

This prompt is for operations and platform teams who need to programmatically decide whether a model's output can be delivered directly to a user or downstream system, or whether it must be routed to a human review queue. It acts as a decision layer that consumes quality signals, confidence scores, and business rules to produce a structured escalation decision. Use it when you have already generated an output and assessed its quality, and you now need a consistent, auditable routing decision that accounts for risk tolerance, reviewer capacity, and output criticality. Do not use this prompt for the initial quality assessment itself; pair it with a quality gate evaluation or confidence extraction prompt that produces the input scores this template requires.

The ideal user is an infrastructure or platform engineer integrating this prompt into an AI pipeline's post-processing stage. Required context includes a pre-assessed quality score, a confidence metric, the output's criticality classification, and current reviewer queue depth. Without these inputs, the prompt will hallucinate a decision based on incomplete signals. You must also define your escalation policy upfront: what risk levels trigger review, what latency budget exists for human intervention, and what constitutes an acceptable false-negative rate for auto-approvals. Wire this prompt after your quality gate but before your delivery layer, and log every decision with the full input context for auditability.

Avoid using this prompt when the quality assessment itself is unreliable or when the cost of a wrong auto-approval exceeds the cost of reviewing every output. In high-stakes domains such as clinical documentation, legal filings, or safety-critical operations, this prompt should default to escalation unless confidence exceeds a calibrated high threshold. Do not treat this as a one-size-fits-all router; tune the risk thresholds per output type and monitor escalation rates weekly. If your escalation rate exceeds 30% of total outputs, investigate whether your upstream quality assessment is too noisy or your thresholds are too conservative. Next, pair this prompt with the Confidence Score Extraction and Normalization Prompt Template to ensure the scores feeding into this decision layer are calibrated and comparable across models.

This prompt template is designed to be the decision engine inside an automated quality gate. When a model output falls below a confidence threshold or fails validation, this prompt takes the failed output, the failure reason, and your operational context to produce a structured escalation decision. The decision includes a routing target (which queue or team), a priority level, and a concise context summary so the human reviewer can immediately understand what went wrong without re-reading the entire conversation or source material. Replace every square-bracket placeholder with live data from your quality gate, queue monitor, and business rules before sending the prompt to the model.

textCopy
You are an automated escalation router for a production AI pipeline. Your job is to decide whether a failed model output should be escalated for human review, and if so, where it should go, at what priority, and what context the reviewer needs.

## Input

[OUTPUT_TO_REVIEW]
(The full model output that failed quality checks.)

[FAILURE_REASONS]
(A structured list of what failed: schema violations, low confidence scores, safety flags, missing citations, etc. Include specific field names, scores, and validator messages.)

[SOURCE_CONTEXT]
(The original user input, retrieved documents, or conversation history that led to this output. Include only what the reviewer needs to understand the task.)

[QUALITY_GATE_RESULT]
(The complete quality gate assessment, including pass/fail/review decision, confidence scores, and any dimensional scores.)

[AVAILABLE_QUEUES]
(A list of available review queues with descriptions, e.g., "safety_review: For content flagged for policy violations", "factuality_review: For outputs with low grounding scores", "general_review: For all other escalations".)

[ESCALATION_POLICY]
(The rules for when to escalate vs. auto-reject vs. silently discard. Include priority definitions, e.g., "P0: safety violation, must review within 15 minutes", "P1: factuality failure, review within 2 hours", "P2: formatting issue, review within 24 hours".)

[CURRENT_QUEUE_DEPTH]
(Optional: current load on each review queue to inform routing decisions and avoid overwhelming a single team.)

## Task

Analyze the failed output, the failure reasons, and the operational context. Produce a structured escalation decision.

## Output Schema

Return a single JSON object with these fields:

- `escalate` (boolean): Whether this output should be escalated for human review. Set to false only if the failure is trivial, the output can be safely discarded, or auto-retry is more appropriate.
- `routing_target` (string or null): The queue name from AVAILABLE_QUEUES where this should be routed. Null if escalate is false.
- `priority` (string or null): The priority level from ESCALATION_POLICY. Null if escalate is false.
- `context_summary` (string): A 2-4 sentence summary for the reviewer explaining what the model was trying to do, what went wrong, and what the reviewer should focus on. Write this as if speaking to a colleague who needs to act quickly.
- `failure_highlight` (array of strings): The 1-3 most critical failure points the reviewer must address. Be specific: cite field names, scores, or exact validator messages.
- `suggested_action` (string): One of "review_and_fix", "review_and_approve", "review_and_reject", "discard", or "retry".
- `retry_viability` (string or null): If retry is possible, explain what should change before retrying (e.g., "fix the schema mismatch and resubmit", "provide more source context"). Null if retry is not recommended.
- `estimated_review_effort` (string): One of "quick" (under 2 minutes), "moderate" (2-10 minutes), or "substantial" (over 10 minutes).

## Constraints

- Do not escalate outputs that can be safely discarded (e.g., empty responses, complete gibberish with no user impact).
- Do not escalate if the failure is a known, reproducible schema error that an auto-retry with a repair prompt can fix.
- Always escalate safety violations regardless of confidence score.
- If multiple queues could apply, choose the one that addresses the most severe failure mode.
- If CURRENT_QUEUE_DEPTH is provided, avoid routing to queues that are already overloaded unless the priority demands it.
- The context_summary must not include PII, credentials, or sensitive data from the original output. Redact if necessary.
- If the failure reasons are contradictory or unclear, escalate to general_review with priority P1 and note the ambiguity in context_summary.

## Examples

### Example 1: Safety Escalation
Input: Output contains a policy violation flagged by the safety gate. Confidence score is high but content is disallowed.
Output:
{
  "escalate": true,
  "routing_target": "safety_review",
  "priority": "P0",
  "context_summary": "Model generated a response to a user question about [topic] that included disallowed content in the third paragraph. The safety gate flagged a policy violation with severity HIGH. The rest of the response was factually accurate but the violating segment must be reviewed before any release.",
  "failure_highlight": ["Policy violation: [specific policy] in paragraph 3", "Severity: HIGH"],
  "suggested_action": "review_and_fix",
  "retry_viability": "Retry with stricter safety instructions and content filtering enabled.",
  "estimated_review_effort": "moderate"
}

### Example 2: Low-Confidence Factuality Issue
Input: Output has a grounding score of 0.3. Retrieved documents support some claims but not others.
Output:
{
  "escalate": true,
  "routing_target": "factuality_review",
  "priority": "P1",
  "context_summary": "Model answered a question about [subject] with claims that have low grounding scores. Two of five factual claims could not be matched to the provided source documents. The unsupported claims involve specific numbers and dates that may be hallucinated.",
  "failure_highlight": ["Grounding score: 0.3 (threshold: 0.7)", "Unsupported claim: [specific claim about numbers]", "Unsupported claim: [specific claim about dates]"],
  "suggested_action": "review_and_fix",
  "retry_viability": "Retry with stricter citation requirements and a narrower source set.",
  "estimated_review_effort": "moderate"
}

### Example 3: Trivial Formatting Issue
Input: Output is valid JSON but has an extra whitespace character that broke a parser. Content is correct.
Output:
{
  "escalate": false,
  "routing_target": null,
  "priority": null,
  "context_summary": "Model produced valid JSON with correct content but included a trailing whitespace character that caused a parsing error in the downstream system. This is a known formatting issue that the sanitization layer can fix automatically.",
  "failure_highlight": ["Trailing whitespace in JSON output"],
  "suggested_action": "retry",
  "retry_viability": "Apply whitespace sanitization and resubmit. No content changes needed.",
  "estimated_review_effort": "quick"
}

After pasting this template, the most important adaptation is calibrating the ESCALATION_POLICY and AVAILABLE_QUEUES to match your actual operational reality. Generic policies produce generic escalations. Define your queues based on who actually reviews what: if your safety team and your factuality team are the same people, don't create separate queues. Define priority levels based on your actual SLAs, not aspirational ones. If no one is on call at 3 AM, don't set a 15-minute P0 target. The CURRENT_QUEUE_DEPTH field is optional but powerful—wiring it to your actual queue metrics prevents the model from routing every escalation to the same overloaded team. If you don't have queue depth data, remove that field from the prompt entirely rather than leaving it empty, which can cause the model to invent queue states.

Before deploying this prompt to production, run it against a labeled dataset of at least 50 known escalation cases with ground-truth routing decisions. Measure routing accuracy, priority agreement, and whether the context_summary contains enough information for a reviewer to act without opening the original output. The most common failure mode is over-escalation: the model escalates trivial formatting issues because it's safer to escalate than to discard. Counter this by adding explicit examples of non-escalation cases and by monitoring your escalation rate. If more than 20% of outputs are being escalated, your thresholds or your examples need adjustment. The second most common failure is vague context summaries that force reviewers to re-inspect the full output, defeating the purpose of the escalation prompt. Test summaries by giving them to a colleague without the original output and asking them to describe what went wrong.

The escalation decision prompt is not a standalone artifact—it is a gating function inside a larger output pipeline. Wire it after your primary generation step and after any automated repair or validation passes have failed. The prompt receives the original output, the validation failure reasons, and any available confidence scores. Its job is to decide whether this output should be routed to a human review queue or discarded with a logged reason. Do not call this prompt on every output; only invoke it when automated quality gates have already rejected the payload. This keeps latency and cost predictable and prevents the escalation prompt from becoming a bottleneck.

Build a thin harness around the prompt call that enforces several operational constraints. First, validate the escalation decision schema strictly—the response must contain decision (enum: ESCALATE, DISCARD, RETRY), priority (enum: LOW, MEDIUM, HIGH, CRITICAL), target_queue (string matching a known queue ID), and reviewer_context (a concise summary of what failed and why). Reject malformed responses and retry once with a stricter schema reminder before falling back to a default ESCALATE to manual-review with MEDIUM priority. Second, enforce an escalation rate threshold: if more than N% of outputs in a rolling window trigger escalation, pause the pipeline and alert the operations team—this usually indicates a systemic model failure, not individual output issues. Third, deduplicate escalation requests by content hash to avoid flooding reviewers with near-identical failures. Fourth, attach the full validation trace, original input, and model output to the review ticket so the human reviewer has complete context without hunting through logs.

Model choice matters here. This prompt benefits from a capable instruction-following model with strong reasoning (GPT-4o, Claude 3.5 Sonnet, or equivalent). Avoid using a smaller or faster model for escalation decisions—false negatives here mean bad outputs reach users, and false positives waste reviewer time. Set temperature=0 or very low to keep decisions consistent. If your system processes outputs in batches, consider batching escalation decisions together so the model can see patterns across multiple failures and adjust priority accordingly. Log every decision with the prompt version, model ID, timestamp, and decision rationale. These logs become your audit trail for escalation rate analysis, reviewer workload forecasting, and prompt improvement over time.

Before deploying, test the harness with a golden set of known-bad outputs that should escalate and known-borderline outputs that should not. Measure escalation precision (did we escalate the right items?) and recall (did we catch all items that needed human review?). Also measure reviewer workload impact: if the escalation prompt sends too many low-priority items, reviewers will ignore or rush through the queue, defeating the purpose. Set a target maximum escalation rate (e.g., 5% of all outputs) and tune the prompt's decision threshold through few-shot examples and constraint language until the rate stabilizes. Finally, build a feedback loop: when reviewers resolve escalated tickets, capture their actual disposition (approved, rejected, corrected) and use that data to evaluate whether the escalation prompt is making correct routing decisions over time.