Refusal with Evidence Request Prompt for Follow-Up Retrieval

This prompt is designed for agent-based Retrieval-Augmented Generation (RAG) systems that possess a critical capability: the ability to perform follow-up retrieval. In a standard RAG setup, an initial search yields insufficient evidence, and the model either hallucinates an answer or delivers a dead-end refusal. This prompt transforms that failure state into a recovery path. Instead of a simple 'I don't know,' the model outputs a structured refusal containing specific, actionable retrieval queries designed to find the missing evidence. The primary job-to-be-done is to bridge the gap between a failed initial search and a successful subsequent one, enabling a multi-step agent to self-correct its knowledge state.

You should use this prompt when your system architecture includes a search tool available in a subsequent agent step and you need the model to articulate exactly what evidence would resolve the gap. The ideal user is an AI engineer or RAG pipeline architect building a system where an orchestrating agent can parse the model's structured refusal, execute the suggested queries against a database or API, and feed the new results back into the context window for a second answer attempt. This pattern is particularly valuable in research assistants, legal document review, and complex customer support systems where the cost of a wrong answer is high, but the cost of a second retrieval loop is acceptable. Do not use this prompt for static Q&A systems with no retrieval capability, single-shot answer generation where no tool is available, or when you need a final answer rather than a retrieval plan. In those cases, a standard evidence insufficiency refusal or a calibrated uncertainty expression would be more appropriate.

Before implementing, verify that your agent harness can parse the structured output from this prompt and has a reliable search function to call. A common failure mode is generating excellent retrieval queries that the downstream search system cannot parse or execute, leading to a second failure. Ensure your search tool's capabilities and query syntax are clearly described in the prompt's [TOOLS] section. The next step is to pair this prompt with an evaluation rubric that checks not just for valid JSON, but for the actionability and precision of the suggested retrieval queries.

This template instructs the model to act as a strict grounding gate. Instead of guessing or producing a low-confidence answer, the model must refuse and articulate exactly what evidence would change its decision. The output is a machine-readable refusal payload containing one or more retrieval queries designed for a downstream search tool. This pattern is essential for agent-based RAG systems where a follow-up retrieval loop can resolve the gap before the user sees a response.

textCopy
You are an evidence-grounded assistant in a retrieval-augmented system. Your primary directive is to answer only when the provided evidence is sufficient. If the evidence is insufficient, you must refuse to answer and instead produce a structured request for the missing evidence.

## INPUT
- User Query: [USER_QUERY]
- Retrieved Evidence Passages: [RETRIEVED_EVIDENCE]
- Evidence Sufficiency Threshold: [SUFFICIENCY_THRESHOLD]

## BEHAVIOR RULES
1. Analyze whether the retrieved evidence fully, partially, or does not support an answer to the user query.
2. If the evidence is sufficient (meets or exceeds the threshold), answer the query using only the provided evidence. Cite specific passages.
3. If the evidence is insufficient, do not answer the query. Do not speculate, infer beyond the text, or use internal knowledge.
4. When refusing, output a JSON object following the schema below. The `evidence_requests` array must contain one or more specific, actionable retrieval queries that would fill the identified gaps.

## OUTPUT SCHEMA (when refusing)
{
  "status": "refused",
  "rationale": "String explaining why the provided evidence is insufficient.",
  "missing_information": ["List of specific facts or data points needed."],
  "evidence_requests": [
    {
      "query": "A self-contained search query for a vector or keyword store.",
      "expected_content": "Description of the document type or passage that would satisfy this request.",
      "priority": "critical | high | medium"
    }
  ]
}

## CONSTRAINTS
- Never invent citations or fabricate source details.
- If the user query is ambiguous, request clarification as an `evidence_request` rather than guessing intent.
- The `evidence_requests` must be executable by a search system without further user input.

To adapt this template, replace the bracketed placeholders with your application's runtime variables. [USER_QUERY] is the raw input. [RETRIEVED_EVIDENCE] should be a pre-formatted string of your top-k passages, including source identifiers. [SUFFICIENCY_THRESHOLD] can be a qualitative label (e.g., 'high') or a numerical score if your pipeline produces one. The output schema is designed for direct consumption by a retrieval tool dispatcher. If your system uses function calling, map the evidence_requests array directly to a search tool's arguments. For high-stakes domains, add a [RISK_LEVEL] parameter that adjusts the threshold dynamically—critical applications should refuse more aggressively. Always log the rationale and missing_information fields for offline evaluation of your retrieval pipeline's coverage gaps.

Before deploying, validate that the model reliably outputs valid JSON under the refusal path. Common failure modes include the model answering despite insufficient evidence (false negative on refusal) or generating vague evidence requests like 'more information about the topic.' Use a structured output API or a retry parser to enforce the schema. Pair this prompt with an evaluator that checks whether the generated evidence_requests would actually retrieve the missing facts if executed against your knowledge base.

This prompt is designed as a decision node within an agentic RAG loop, not a terminal response. Its primary output is a structured refusal that includes actionable evidence requests—formatted as retrieval queries or source descriptions—that a downstream search tool can consume. The implementation harness must treat the model's output as a machine-readable instruction, not just a user-facing message. The core integration pattern is: (1) initial retrieval attempt, (2) evidence sufficiency check, (3) if insufficient, invoke this prompt to generate targeted retrieval requests, (4) execute those requests via your search infrastructure, and (5) re-enter the answer generation flow with the expanded evidence set.

To wire this into an application, wrap the prompt in a function that accepts the original user query, the set of retrieved passages, and a configurable sufficiency threshold. The model's response should be parsed into a strict JSON schema containing a refusal_message (for user display or logging) and an evidence_requests array of objects, each with a query string and an optional source_description field. Implement a validator that rejects responses where evidence_requests is empty or contains queries that are trivial reformulations of the original user input—this prevents infinite loops. Log every refusal event with the original query, retrieved passages, generated evidence requests, and the validator result for debugging retrieval gaps.

For retry logic, set a maximum of two follow-up retrieval rounds before escalating to a terminal refusal or human review. After each round, re-run evidence sufficiency scoring. If the new evidence still falls below threshold, invoke this prompt again but include the history of prior retrieval attempts and their results in the [CONTEXT] placeholder to prevent repeated ineffective queries. In high-stakes domains such as healthcare or legal, always route terminal refusals to a human review queue with the full audit trail. Avoid using this prompt for queries where the knowledge base is known to be sparse—pre-filter with an answerability classifier to reduce unnecessary retrieval loops and latency.