Human Review Handoff Prompt for Uncertain Answers

This prompt is for operators of high-stakes RAG systems who need a reliable mechanism to escalate uncertain answers for human review. It is designed for workflows where a wrong answer is worse than no answer, such as clinical decision support, legal research, financial compliance, or safety-critical technical documentation. The prompt instructs the model to produce a structured review packet containing the original question, retrieved evidence, a draft answer, and specific uncertainty flags. This packet is then routed to a human reviewer through a queue, API, or internal tool. Use this prompt when your system must distinguish between confident, grounded answers and cases that require human judgment.

The ideal user is an AI engineer or product team that has already built a retrieval pipeline and needs a controlled handoff mechanism before answers reach end users. You should have a defined human review queue, a set of uncertainty categories that matter for your domain, and clear SLAs for review turnaround. The prompt works best when paired with a confidence threshold or a set of pre-generation checks—such as evidence sufficiency scoring—that determine whether to use this handoff prompt or a standard answer generation prompt. Without these upstream signals, the model may escalate too aggressively or not enough, undermining the system's reliability.

Do not use this prompt for low-stakes Q&A, entertainment, or general-purpose chatbots where escalation adds unacceptable latency or cost. It is also a poor fit for real-time applications where a human reviewer cannot respond within the user's expected wait time. If your system lacks a human review process entirely, this prompt will create a dead end. In such cases, consider an abstention prompt that gracefully declines to answer instead. Finally, avoid using this prompt as a substitute for proper retrieval quality—if your evidence is consistently poor, fix the retrieval pipeline before adding a human review layer.

This prompt template is designed to be wired into a RAG pipeline's post-generation evaluation step. When your confidence scoring or hallucination detection system flags an answer as unreliable, this prompt takes the original question, the retrieved evidence, and the draft answer, then produces a structured review packet for a human operator. The packet includes the specific uncertainty flags that triggered the escalation, making it faster for a reviewer to triage and resolve the case without redoing the retrieval work.

textCopy
You are a review handoff assistant for a high-stakes question-answering system. Your job is to prepare a structured review packet when the system cannot answer with sufficient confidence. Do not attempt to answer the user's question. Instead, organize the available information so a human reviewer can quickly assess the situation and respond.

## Inputs

**User Question:**
[USER_QUESTION]

**Retrieved Context Passages:**
[RETRIEVED_CONTEXT]

**Draft Answer (flagged as unreliable):**
[DRAFT_ANSWER]

**Uncertainty Flags:**
[UNCERTAINTY_FLAGS]

**Confidence Scores:**
[CONFIDENCE_SCORES]

## Instructions

1. Summarize the user's question in one sentence, preserving all key constraints and entities.
2. List the specific uncertainty flags that triggered this escalation, with a brief explanation of each flag's meaning.
3. Summarize the relevant evidence from the retrieved context, organized by source. Note any conflicts, gaps, or quality issues in the evidence.
4. Present the draft answer that was flagged, clearly marking which claims triggered which uncertainty flags.
5. Identify what additional information or clarification would help resolve the uncertainty.
6. Suggest 1-3 specific actions the reviewer could take (e.g., search a specific database, consult a subject matter expert, ask the user for clarification).

## Output Format

Produce a JSON object with this exact schema:

{
  "review_packet_id": "string, unique identifier for this review case",
  "question_summary": "string, one-sentence restatement of the user's question",
  "escalation_reason": [
    {
      "flag": "string, name of the uncertainty flag",
      "explanation": "string, what this flag means in context"
    }
  ],
  "evidence_summary": [
    {
      "source_id": "string, identifier from the retrieved context",
      "relevant_excerpts": ["string, key passage"],
      "issues": ["string, any problems with this evidence"]
    }
  ],
  "flagged_answer": {
    "full_text": "string, the complete draft answer",
    "flagged_claims": [
      {
        "claim": "string, the specific claim",
        "flag": "string, which uncertainty flag applies",
        "reason": "string, why this claim is unreliable"
      }
    ]
  },
  "information_gaps": ["string, what is missing or unclear"],
  "suggested_reviewer_actions": ["string, concrete next step for the human reviewer"]
}

## Constraints

- Do not answer the user's question or provide any new information not present in the inputs.
- Do not minimize or downplay the uncertainty. Be explicit about what is unknown.
- If the retrieved context is empty or irrelevant, state that clearly in the evidence summary.
- Preserve all source identifiers exactly as provided in the retrieved context.
- If [RISK_LEVEL] is "high", add a "risk_note" field to the output warning the reviewer about potential harm from an incorrect answer.

Adapt this template by replacing each square-bracket placeholder with data from your application. The [UNCERTAINTY_FLAGS] and [CONFIDENCE_SCORES] placeholders should carry the output of your upstream evaluation step—whether that is a model-graded confidence score, a hallucination detector's findings, or a heuristic rule like missing citations. The [RISK_LEVEL] parameter should be set by your routing logic based on the domain, user context, or question type. For regulated domains, always set this to "high" to ensure the risk note is generated.

Before deploying, test this prompt against cases where escalation is correct (the draft answer is genuinely unreliable) and cases where it is incorrect (the draft answer is actually fine but flagged by an overly conservative detector). A common failure mode is the model minimizing the uncertainty flags in its summary, making the case look less urgent than it is. Validate the output JSON against your schema before presenting it to a reviewer, and log every handoff packet for audit and eval improvement.

The Human Review Handoff Prompt is not a standalone artifact; it is a decision-and-formatting step inside a larger RAG pipeline. The prompt should only be invoked when a preceding confidence or grounding check determines that the generated answer is too uncertain to deliver directly to the user. The implementation harness must therefore gate execution: a confidence_score below a defined threshold, a grounding_failure flag from a hallucination detector, or a sufficiency_check returning no_go should trigger this prompt. The harness supplies the prompt with the original question, the retrieved evidence, the draft answer that failed verification, and the specific flags that caused the escalation. This ensures the review packet contains the full diagnostic context rather than a vague 'something went wrong' message.

Wire the prompt into an asynchronous review queue rather than blocking the user request. The application layer should: (1) assemble the [QUESTION], [RETRIEVED_EVIDENCE], [DRAFT_ANSWER], and [UNCERTAINTY_FLAGS] variables from the failed generation step; (2) call the LLM with this prompt and a strict JSON output schema matching the review packet structure; (3) validate the response—check that escalation_reason is non-empty, evidence_gaps is a list, and recommended_action is one of the allowed enum values (full_review, partial_correct, routing_error); (4) on validation failure, retry once with the validation errors appended to the prompt as [CONSTRAINTS]; (5) on success, write the structured packet to a review database or ticket system with a unique review_id, timestamp, and the raw model response for auditability. Log every handoff decision—both the trigger reason and the generated packet—to enable retrospective analysis of escalation rates and false-positive handoffs.

Model choice matters here. Use a model with strong instruction-following and structured output capabilities (GPT-4o, Claude 3.5 Sonnet, or equivalent). Avoid smaller or older models that may drop required fields or hallucinate evidence gaps that don't exist. If the review packet will be consumed by a human reviewer in a high-stakes domain (healthcare, legal, finance), add a second validation pass: before the packet enters the review queue, run a lightweight check that every cited evidence gap actually corresponds to a missing piece of information in the retrieved context, not a model confabulation. For regulated environments, store the full prompt, response, and validation result as an audit record. Do not use this prompt as a substitute for proper confidence scoring—if your pipeline is generating uncertain answers on more than 10-15% of queries, fix the retrieval or grounding layer rather than routing everything to human review.