Missing Evidence Escalation Prompt

This prompt is a circuit breaker for Retrieval-Augmented Generation (RAG) and evidence-grounded systems. Its job is to detect when the provided context is insufficient to answer a user query and force the model to produce a structured escalation record instead of a confident fabrication. Use it in any production AI harness where silence or a refusal is safer than a hallucinated answer. The prompt instructs the model to audit the provided context against the query, identify specific missing information, and output a machine-readable escalation payload. It does not attempt to answer the question.

The ideal user is an AI engineer or platform developer building a product where factual accuracy is non-negotiable—cited Q&A systems, clinical documentation assistants, legal research tools, or financial audit copilots. Required context includes the user query, retrieved passages, and a retry budget tracker. Do not use this prompt when the model is expected to synthesize an answer from incomplete evidence, when the cost of escalation exceeds the cost of a low-confidence answer, or when the application layer already handles abstention through separate classification logic. The prompt is designed for systems that need the model itself to perform the evidence audit and produce a structured escalation signal that downstream code can act on.

Before deploying this prompt, ensure your application harness can parse the escalation payload and route it correctly—to a human review queue, a fallback model, or a retry loop with expanded retrieval. Wire the retry budget tracker into the prompt's [RETRY_COUNT] and [MAX_RETRIES] placeholders so the model knows when to stop retrying and escalate. The most common failure mode is the model attempting to answer despite insufficient evidence; mitigate this by including strong negative examples in [EXAMPLES] that show correct abstention behavior. For regulated domains, always require human review of escalated cases before any action is taken.

This prompt template is designed to be placed in the system message of your retry handler or post-generation validation step. It instructs the model to cross-check its own output against provided context, identify missing evidence, and produce a structured escalation record instead of fabricating an answer. The template uses square-bracket placeholders that your application must populate before sending the request. Each placeholder maps to a specific input your harness controls: the original query, retrieved context, the model's draft response, and operational parameters like retry count and escalation thresholds.

textCopy
You are an evidence auditor in a production AI system. Your job is to detect when retrieved context is insufficient to answer a user query and escalate instead of fabricating an answer.

## INPUTS
- User Query: [USER_QUERY]
- Retrieved Context: [RETRIEVED_CONTEXT]
- Draft Response: [DRAFT_RESPONSE]
- Retry Count: [RETRY_COUNT] of [MAX_RETRIES]
- Escalation Threshold: [ESCALATION_THRESHOLD]

## TASK
1. Extract every factual claim from the Draft Response.
2. For each claim, determine whether it is directly supported by the Retrieved Context.
3. Classify each claim as: SUPPORTED, INFERRED, or UNSUPPORTED.
4. If any claim is UNSUPPORTED, do NOT attempt to answer. Instead, produce an escalation record.
5. If all claims are SUPPORTED or INFERRED, produce a confidence assessment.

## ESCALATION RULES
- Escalate when: any UNSUPPORTED claim exists, context is empty, context is clearly irrelevant, or Retry Count equals [MAX_RETRIES].
- Do NOT escalate for INFERRED claims that are reasonable given the context.

## OUTPUT SCHEMA
Return a JSON object with this exact structure:
{
  "decision": "ESCALATE" | "PASS",
  "claims": [
    {
      "claim_text": "string",
      "classification": "SUPPORTED" | "INFERRED" | "UNSUPPORTED",
      "source_excerpt": "string or null",
      "reasoning": "string"
    }
  ],
  "missing_evidence": ["list of specific facts needed but not found"],
  "escalation_reason": "string explaining why escalation is required, or null if PASS",
  "retry_remaining": [RETRY_COUNT] < [MAX_RETRIES],
  "recommended_action": "RETRY_WITH_REFINED_QUERY" | "HUMAN_REVIEW" | "ABSTAIN"
}

## CONSTRAINTS
- Never invent facts not present in Retrieved Context.
- If Retrieved Context is empty or irrelevant, escalate immediately.
- Do not use external knowledge or training data to fill gaps.
- If Retry Count equals [MAX_RETRIES], recommend HUMAN_REVIEW.
- Preserve the original User Query in escalation records for downstream routing.

To adapt this template, replace each square-bracket placeholder with values from your application state. [USER_QUERY] and [RETRIEVED_CONTEXT] come from your RAG pipeline. [DRAFT_RESPONSE] is the model output you're auditing—this could be from the initial generation or a previous retry attempt. [RETRY_COUNT] and [MAX_RETRIES] come from your retry budget tracker. [ESCALATION_THRESHOLD] defines your risk tolerance: set it to "strict" to escalate on any UNSUPPORTED claim, or "moderate" to allow INFERRED claims through. Wire the output decision field into your routing logic: PASS continues the workflow, ESCALATE triggers your escalation handler. Always log the full claims array and missing_evidence list for observability and eval improvement.

The Missing Evidence Escalation Prompt is not a standalone module—it is a decision boundary inside a larger retrieval-augmented generation harness. Wire it as a post-retrieval, pre-generation gate. After your retriever returns chunks but before the model drafts a user-facing answer, invoke this prompt with the user query and the retrieved context. If the prompt returns an escalation record (i.e., evidence_sufficient: false), suppress the answer generation step entirely and route to the escalation path. If it returns evidence_sufficient: true, pass the validated context forward to your answer-generation prompt. This prevents the model from ever seeing a prompt that asks it to answer from empty or irrelevant context, which is the single largest driver of hallucination in RAG systems.

Implement a retry budget tracker in the harness layer, not in the prompt itself. Before calling this prompt, check retry_count against a configured maximum (start with 3). On each insufficient-evidence escalation, increment the counter and optionally expand the retrieval query—add synonym expansion, relax metadata filters, or switch from vector to hybrid search—then re-invoke the evidence check. If the budget is exhausted, bypass further retrieval attempts and route directly to a structured escalation payload that includes the original query, the last retrieval attempt's context, the missing evidence fields from the final check, and a retry_exhausted: true flag. Log every attempt with timestamps, retrieval parameters, and the prompt's raw output for observability. This trace is essential for tuning retrieval quality and retry thresholds over time.

Validation and routing logic should parse the prompt's structured output (JSON) and branch on evidence_sufficient. If true, extract the usable_context field and inject it into your answer-generation prompt—do not re-retrieve. If false, check retry_count. If under budget, trigger a re-retrieval with the missing_evidence fields as query expansion hints. If over budget, serialize the full escalation record and push it to your human review queue, support ticketing system, or a dead-letter topic for asynchronous triage. Never show the raw escalation payload to end users; instead, surface a graceful degradation message like 'I don't have enough information to answer that accurately right now.' The escalation record is for internal operators, not consumers.

Model choice matters. This prompt performs a structured classification and extraction task—it does not require creative generation. Use a fast, cheaper model (e.g., a small fine-tuned classifier, or a low-latency instruction model) for the evidence-sufficiency check, reserving your larger generation model for the actual answer step. This keeps latency low and costs predictable, especially when retries multiply calls. If you observe the evidence-check model itself hallucinating missing fields or fabricating evidence summaries, add a post-check validator that confirms every missing_evidence entry maps to a gap in the provided context—not a gap in the model's own knowledge. A model that invents missing fields defeats the purpose of the gate.

Human review integration is mandatory for high-stakes domains. When the escalation record lands in a review queue, present the reviewer with the original query, the retrieved context (or lack thereof), the structured missing-evidence fields, and the retry history. Allow the reviewer to either supply the missing context manually, approve a 'cannot answer' response, or override the escalation if the context was actually sufficient. Capture the reviewer's decision to close the feedback loop: use it to improve retrieval, adjust the sufficiency threshold, or fine-tune the evidence-check prompt. Without this loop, the system will repeatedly escalate the same class of queries without improvement.