Multi-Modal Escalation Trigger Prompt for Human Review

This prompt is designed for operations engineers and platform teams who need a deterministic, auditable gate between an AI classifier and a human review queue. Its job is not to classify content, but to consume structured classification metadata—including confidence scores, risk flags, and modality-specific uncertainty signals—and produce a binary decision: process automatically or escalate for human review. The ideal user is someone integrating this prompt into a production pipeline where the cost of a missed high-risk input (e.g., a policy violation, a PII leak, or a safety-critical misclassification) far outweighs the operational cost of human review. The prompt assumes you already have an upstream classification step that outputs a well-defined schema; it acts as the policy enforcement layer that translates classification results into a consistent, explainable escalation action with an explicit reason code and a priority assignment for your review queue.

You should use this prompt when your escalation logic needs to be resistant to two common failure modes: over-escalation, which floods your human review queue with false positives and burns out your ops team, and under-escalation, which allows harmful or non-compliant content to reach end-users or downstream systems. The prompt template is structured to weigh multiple signals—such as a high-risk classification with low confidence versus a low-risk classification with high confidence—and to handle modality conflicts, like an image flagged for abuse while its accompanying text is clean. It is particularly valuable in multimodal pipelines where a single confidence threshold is insufficient because uncertainty can be concentrated in one modality. For example, a scanned document might have high-confidence text extraction but low-confidence handwriting detection, and the escalation decision must reflect that specific ambiguity rather than a blended score.

Do not use this prompt as a standalone safety classifier or as a replacement for the upstream classification step. It has no ability to analyze raw pixels, audio waveforms, or unstructured text; it only reasons over the metadata you provide. If you feed it poorly calibrated confidence scores or incomplete risk flags, it will make wrong decisions with high confidence. This prompt is also not a substitute for a queue management system—it assigns a priority level, but your application layer must handle queue insertion, reviewer assignment, and SLA tracking. Before deploying, you must calibrate the escalation thresholds against your specific classifiers using a golden dataset that includes both clear-cut and ambiguous cases, and you must monitor the escalation rate and the overturn rate (how often human reviewers disagree with the escalation) as primary health metrics. If your upstream classifiers change or your risk taxonomy evolves, re-evaluate this prompt's thresholds before the change reaches production.

This prompt template is designed to be the final decision gate in a multi-modal classification pipeline. It receives the output of upstream classifiers—content type, intent, risk flags, PII detection, and confidence scores—and makes a binary decision: automate or escalate. The prompt is structured to produce a machine-readable JSON output that your application can use to route the item to a review queue or continue automated processing. Replace every square-bracket placeholder with your application's specific taxonomies, thresholds, and policy definitions before sending to the model.

textCopy
SYSTEM:
You are an escalation decision engine for a multi-modal AI platform. Your job is to decide whether a classified input requires human review before automated processing continues.

You will receive a structured classification summary containing:
- The detected content types and modalities present in the input.
- The classified intent and domain category.
- Risk flags, PII detection results, and policy violation signals.
- Confidence scores from upstream classifiers.
- Any modality-specific uncertainty signals.

Your task is to apply the escalation policy defined below and return a JSON decision.

## Escalation Policy (apply in order; first match wins)
1. **Mandatory Escalation:** Escalate if any of the following are true:
   - [HIGH_RISK_DOMAIN] flag is present (e.g., regulated content, safety-critical).
   - [PII_DETECTED] flag is true AND the input is not already in a redaction pipeline.
   - [POLICY_VIOLATION] severity is [HIGH] or [CRITICAL].
   - [ABUSE_DETECTED] confidence is above [ABUSE_THRESHOLD].
2. **Confidence-Based Escalation:** Escalate if the primary intent classifier confidence is below [INTENT_CONFIDENCE_THRESHOLD] OR the content type classifier confidence is below [CONTENT_TYPE_CONFIDENCE_THRESHOLD].
3. **Modality Conflict Escalation:** Escalate if the text and attachment imply different intents with a conflict score above [MODALITY_CONFLICT_THRESHOLD].
4. **Ambiguity Escalation:** Escalate if the ambiguity flag is true AND the input is not a simple clarification candidate.
5. **Default:** Do not escalate. Route to automated processing.

## Priority Assignment Rules
If escalation is triggered, assign priority as follows:
- **CRITICAL:** [POLICY_VIOLATION] severity is CRITICAL, or [ABUSE_DETECTED] confidence is above [CRITICAL_ABUSE_THRESHOLD].
- **HIGH:** [HIGH_RISK_DOMAIN] flag is present, or [PII_DETECTED] is true, or [POLICY_VIOLATION] severity is HIGH.
- **MEDIUM:** Confidence below threshold on a single classifier, or modality conflict detected.
- **LOW:** Ambiguity flag only, no other triggers.

## Output Schema
Return ONLY valid JSON. No markdown, no commentary.
{
  "escalate": boolean,
  "escalation_reason": string | null,  // Required if escalate is true. Use one of: "mandatory_policy", "low_confidence", "modality_conflict", "ambiguity"
  "priority": string | null,  // "CRITICAL", "HIGH", "MEDIUM", "LOW" if escalated, null otherwise
  "matched_rule": string | null,  // The first policy rule number that triggered escalation
  "review_queue": string | null,  // The target review queue: [QUEUE_SAFETY], [QUEUE_COMPLIANCE], [QUEUE_GENERAL_REVIEW]
  "summary_for_reviewer": string | null  // One-sentence summary of what the reviewer needs to know
}

USER:
Classification Summary:
[CLASSIFICATION_INPUT]

Adaptation guidance: The power of this prompt lies in the policy rules and thresholds you define. Start by setting [INTENT_CONFIDENCE_THRESHOLD] and [CONTENT_TYPE_CONFIDENCE_THRESHOLD] conservatively (e.g., 0.85) and adjust based on your observed escalation rate and missed-risk rate in production. The [CLASSIFICATION_INPUT] placeholder should be populated by your upstream pipeline—do not pass raw user input here. Instead, pass a structured JSON or text summary of all classification results. The review queue mapping ([QUEUE_SAFETY], [QUEUE_COMPLIANCE], [QUEUE_GENERAL_REVIEW]) should match your actual operational queues. If you only have one review queue, simplify the schema. Before deploying, run this prompt against a golden dataset of known escalation cases and measure both the escalation rate and the false-negative rate (items that should have been escalated but were not).

This prompt is designed to sit at a critical decision point in your pipeline: after a multimodal input has been classified and processed, but before an automated action is taken. The implementation harness must treat the prompt's output as a structured control signal, not a suggestion. The JSON response—containing escalate (boolean), escalation_reasons, priority, and confidence_breakdown—should be parsed by application code that enforces the decision. If escalate is true, the input and all associated processing artifacts must be placed into a human review queue with the specified priority and reasons. If escalate is false, the system proceeds with the automated workflow. There is no middle ground: a missing or malformed JSON response must be treated as an escalation to fail safely.

To wire this into an application, wrap the prompt in a function that accepts the multimodal input, classification results, confidence scores, risk flags, and any policy thresholds as variables. The function should construct the prompt, call the model with response_format set to json_object (or equivalent for your provider), and validate the response against a strict schema before acting on it. Validation checks must include: (1) escalate is a boolean, (2) escalation_reasons is a non-empty array when escalate is true, (3) priority is one of the allowed enum values (low, medium, high, critical), and (4) confidence_breakdown contains all required modality-specific scores. If validation fails, retry once with a repair prompt that includes the validation error. If the retry also fails, escalate to a human operator with a system_error reason. Log every decision—including non-escalations—with the full prompt, response, validation result, and timestamp for auditability. For high-throughput systems, consider adding a lightweight pre-filter: if all modality confidence scores exceed 0.95 and no risk flags are triggered, you may bypass the LLM call entirely and auto-approve, but log this bypass decision with the same fidelity as a model decision.

Model choice matters here. This is a classification decision with high stakes for both safety and operational efficiency. Use a model with strong instruction-following and JSON output reliability—GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro are appropriate. Avoid smaller or older models that may hallucinate escalation reasons or produce inconsistent JSON under complex multimodal conditions. Set temperature=0 to maximize determinism. If your system processes images, audio, or video, ensure the model you choose supports those modalities natively; do not attempt to describe visual or auditory content in text and pass it to a text-only model, as this introduces a lossy translation step that undermines the confidence scoring. For cost-sensitive deployments, consider routing low-risk, high-confidence cases to a smaller model and reserving the full multimodal model for ambiguous or flagged inputs—but only after validating that the smaller model's escalation decisions match the larger model's on a golden test set.

Human review queue integration must be explicit and traceable. When escalate is true, your application code should create a review ticket that includes: the original multimodal input, the classification output that triggered the workflow, the full escalation prompt response, and a direct link to the evidence that caused the escalation (e.g., the specific image region with low confidence, the audio timestamp with unclear speech, or the risk flag that fired). The priority field should map directly to your queue's SLA tiers. Do not rely on the human reviewer to re-run classification; give them the context they need to make a quick, informed decision. Track escalation rates by reason and modality to identify systemic issues—if 30% of audio inputs escalate due to low confidence, your audio processing pipeline may need improvement rather than more human reviewers. Finally, close the loop: when a human reviewer resolves an escalated case, capture their decision and use it to evaluate whether the escalation was justified. This feedback is essential for tuning confidence thresholds and updating the prompt's escalation criteria over time.