Human Review Escalation Prompt for Low-Confidence Answers

This prompt is designed for operators of high-stakes RAG systems who need an automated trigger to create a human review task. When a primary Q&A pipeline returns a low-confidence score, insufficient evidence, or a direct abstention, this prompt takes the original query, the retrieved context, and the system's preliminary assessment and produces a structured escalation payload. The output is meant to be consumed by a ticketing system, a review queue, or an internal operations tool. Use this when the cost of a wrong answer is higher than the cost of a delayed answer, such as in clinical decision support, legal research, financial compliance, or safety-critical technical documentation.

The ideal user is an AI engineer or platform operator who already has a confidence signal from an upstream abstention or scoring prompt. This prompt does not make the abstention decision itself; it assumes the decision has been made and focuses on packaging the evidence for a human reviewer. Required context includes the original user query, the retrieved document chunks, the model's preliminary answer or abstention reason, and any confidence scores or tier labels produced by the upstream gate. Without this context, the escalation payload will be incomplete and the human reviewer will lack the information needed to resolve the case efficiently.

Do not use this prompt as a general-purpose answer generator or as a substitute for a proper confidence scoring pipeline. It is a workflow handoff mechanism, not a Q&A system. It should not be used in low-stakes applications where a wrong answer carries minimal risk, or in systems where the human review queue cannot keep up with the escalation volume. If your escalation rate exceeds 10-15% of total queries, revisit your retrieval quality, confidence thresholds, or abstention rules before scaling the review team. The next section provides the copy-ready prompt template you can adapt for your escalation workflow.

This template is designed to be used as the system prompt for a dedicated escalation classifier. It accepts the original user query, the retrieved context provided to the primary answering system, and that system's own confidence assessment. Its sole job is to produce a structured, machine-readable escalation decision. The prompt is built to enforce a strict output schema, require evidence for the decision, and prevent the model from attempting to answer the user's original question itself.

textCopy
You are an escalation classifier for a high-stakes Retrieval-Augmented Generation (RAG) system. Your only task is to decide whether a user query requires human review based on the primary system's low-confidence assessment. You do not answer the user's query. You do not provide advice. You produce a single JSON object.

Evaluate the following inputs:
<user_query>
[USER_QUERY]
</user_query>

<retrieved_context>
[RETRIEVED_CONTEXT]
</retrieved_context>

<primary_system_confidence>
[PRIMARY_CONFIDENCE_ASSESSMENT]
</primary_system_confidence>

Determine if this query should be escalated for human review. An escalation is required if the primary system's confidence is low, the retrieved context is insufficient, contradictory, or missing critical information, or if the query falls into a high-risk category defined in the constraints.

Your output must be a single JSON object conforming to this schema:
{
  "escalate": boolean, // true if human review is required
  "priority": "high" | "medium" | "low", // the urgency of the review
  "reason_code": string, // a machine-readable code from the approved list: "LOW_CONFIDENCE", "EVIDENCE_GAP", "SOURCE_CONFLICT", "HIGH_RISK_DOMAIN", "OUT_OF_SCOPE"
  "summary_for_reviewer": string, // a concise, factual summary of the issue for the human reviewer. Do not add interpretation.
  "evidence_packet": {
    "user_query": string, // the original user query
    "relevant_context_snippets": [string], // the most relevant parts of the retrieved context
    "primary_confidence_score": number | null, // the score from the primary system, if available
    "primary_uncertainty_flags": [string] // specific flags from the primary system
  }
}

[CONSTRAINTS]
- Do not answer the user's query.
- Do not include any text outside the JSON object.
- The "reason_code" must be one of the approved codes.
- The "summary_for_reviewer" must be a dry, factual statement of the problem, not a recommendation.
- If the primary system's confidence is above the threshold and no other risk flags are present, set "escalate" to false and "priority" to "low".
- Treat queries about [HIGH_RISK_TOPICS] as high priority.

To adapt this template, replace the bracketed placeholders with your application's live data. [USER_QUERY] and [RETRIEVED_CONTEXT] should be populated directly from the current request. [PRIMARY_CONFIDENCE_ASSESSMENT] should be the raw output from your confidence scoring step, not just a number. The [CONSTRAINTS] section is critical for safety; replace [HIGH_RISK_TOPICS] with a comma-separated list of your specific risk areas (e.g., 'drug interactions, dosage instructions, self-harm'). The [OUTPUT_SCHEMA] is embedded directly in the prompt to maximize compliance. After generating the output, always validate the JSON structure and check that the reason_code is from your approved list before routing to a human queue.

This prompt is designed to be the final gate in a high-stakes RAG pipeline. It should be invoked only after the system has already generated an answer and a confidence score, and that score has fallen below a pre-defined threshold. The prompt's job is not to re-evaluate the answer itself, but to package the low-confidence event into a structured, actionable escalation ticket for a human review queue. The primary integration point is a post-generation validation layer that catches low-confidence responses before they reach the end user, replacing the uncertain answer with a 'pending review' message and forwarding the escalation payload to your internal tools.

To implement, wrap the prompt in a function that accepts the original user query, the generated answer, the confidence score, and the retrieved context chunks as input. The function should enforce a strict schema on the model's JSON output, validating that the priority field is one of CRITICAL, HIGH, or MEDIUM, and that the reason_code matches a pre-defined list of failure modes (e.g., EVIDENCE_GAP, SOURCE_CONFLICT, LOW_RELEVANCE). If validation fails, retry the prompt once with the validation error message injected into the [CONSTRAINTS] block. After two consecutive failures, log the raw output and escalate the entire event with a priority: CRITICAL and reason_code: ESCALATION_FAILURE to ensure no low-confidence answer is ever silently dropped. This prompt is stateless and should be called as a standalone request to a fast, instruction-following model like gpt-4o-mini or claude-3-haiku to keep latency and cost low for this routing task.

The evidence_packet field is the most critical output for the human reviewer. It must be a self-contained JSON object that includes the original query, the draft answer, the confidence score, and the top-K retrieved chunks with their metadata. Do not rely on the model to perfectly recall or copy the context; instead, construct this packet in your application code by merging the model's evidence_summary with the actual source chunks from your retrieval system. This prevents the model from hallucinating evidence in the escalation ticket itself. For monitoring, log every escalation event with its priority, reason_code, and a unique review_id. Track the escalation rate as a percentage of total queries, and set an alert if the rate spikes above a baseline, as this could indicate a retrieval pipeline degradation or a change in user query patterns. The next step is to integrate this escalation payload into your existing ticketing system (e.g., Jira, ServiceNow) or a dedicated review queue, ensuring that CRITICAL tickets are routed to the on-call reviewer immediately.