Evidence Gap Filling Prompt for Incomplete Reasoning Chains

This prompt is designed for a specific moment in an agentic RAG loop: the system has attempted a reasoning chain, hit a dead end, and identified what type of fact is missing but not where to find it. The ideal user is an AI engineer or CTO who has already built a mechanism to detect reasoning gaps—such as a validator that flags missing premises or a chain-of-thought monitor that detects an unanswerable step. The prompt's job is to translate that detected gap into a precise, targeted retrieval query that can plausibly unblock the chain. It is not a general-purpose query generator and should not be used for initial question decomposition or for generating queries directly from a user's original question without the intermediate reasoning trace.

Concretely, the system must provide three inputs: a partial reasoning chain (the steps completed so far), a natural-language description of the identified gap (what fact or relationship is missing), and the original user question for context. The prompt then produces a structured output containing a single, self-contained sub-question and a keyword-dense retrieval query. For example, if a chain analyzing a company's Q3 performance stalls because the Q2 baseline revenue is unknown, the gap description might be 'missing Q2 revenue figure for comparison,' and the prompt should output a query like "Acme Corp Q2 2024 revenue reported earnings" rather than a vague "Acme Corp financials." The output schema should also include an expected_evidence_type field (e.g., numerical_figure, date_range, entity_relationship) to help downstream retrieval rankers prioritize the right document chunks.

Do not use this prompt when the system cannot articulate a specific gap. If the failure mode is a vague 'I don't know' without a clear missing fact, this prompt will produce an unfocused query that wastes retrieval bandwidth. Similarly, avoid this prompt when the reasoning chain is empty or the user question is a simple lookup—those cases are better served by standard query rewriting prompts. After implementing, validate the output by checking that the generated query contains key entities from the gap description and that the sub-question is answerable from a single document. For high-stakes domains like legal or financial analysis, log the gap description, generated query, and retrieved results for human audit to ensure the self-correction loop isn't drifting off-course.

This prompt template is the core instruction for an agentic step that detects a stalled reasoning process and requests the missing evidence. It is designed to be injected into a reasoning loop when a primary agent identifies a logical gap it cannot resolve with the currently retrieved context. The template forces the model to articulate exactly what is missing and produce a precise, self-contained retrieval query. Use this when your system needs to self-correct and seek new information rather than hallucinating a bridge or prematurely aborting the chain.

markdownCopy
You are an evidence-gap analyst inside a multi-step reasoning system. Your task is to examine an incomplete reasoning chain, identify the specific missing evidence that prevents reaching a conclusion, and generate a precise retrieval query to find that evidence.

**Reasoning Chain with Gap:**
[REASONING_CHAIN]

**Original User Question:**
[USER_QUESTION]

**Currently Retrieved Context (Insufficient):**
[CURRENT_CONTEXT]

**Instructions:**
1.  **Identify the Gap:** State in one sentence the exact logical or factual link that is missing. What specific piece of information is required to connect the last valid step to the next one?
2.  **Define Expected Evidence:** Describe the type of evidence that would fill this gap (e.g., a date, a definition, a causal relationship, a numerical value, a policy statement).
3.  **Generate Retrieval Query:** Produce a single, self-contained search query designed to retrieve the missing evidence. The query must be specific, use keywords likely to appear in the target document, and avoid referencing the current conversation context.
4.  **Output Format:** Return your response strictly as a JSON object with the following keys: `gap_statement`, `expected_evidence_type`, and `retrieval_query`.

**Output Schema:**
[OUTPUT_SCHEMA]

**Constraints:**
- Do not attempt to answer the original question.
- Do not guess the missing evidence. Your job is only to formulate the query to find it.
- The retrieval query must be a standalone string suitable for a vector or keyword search engine.

To adapt this template, replace the square-bracket placeholders with live data from your application's reasoning engine. [REASONING_CHAIN] should contain the step-by-step logic produced so far, with the last step clearly indicating a dead end. [CURRENT_CONTEXT] is the raw text of the documents already retrieved, which helps the model avoid generating a redundant query. The [OUTPUT_SCHEMA] placeholder should be replaced with your application's strict JSON schema definition for the expected response, including required fields and types. For high-stakes domains, ensure the [OUTPUT_SCHEMA] includes a confidence field and that a human review step is triggered if the gap cannot be clearly articulated. The primary failure mode is generating a query that is too vague or simply restates the original question; validate the output by checking if the retrieval_query contains novel, specific keywords not fully present in the [CURRENT_CONTEXT].

The Evidence Gap Filling prompt is not a standalone generator; it is a control-plane component inside an agentic RAG loop. Its job is to inspect a partial reasoning chain, identify a specific missing fact or logical leap, and produce a targeted sub-question and retrieval query. The application harness must treat this prompt as a decision point: if the gap is fillable, the output query is dispatched to the retrieval system; if the gap is fundamental or the prompt fails validation, the agent should escalate or abstain rather than fabricate an answer. Wire this prompt after a reasoning step produces an incomplete chain and before the next retrieval call.

The harness should enforce a strict output contract. Parse the model response into a typed object with fields: gap_description (string), sub_question (string), retrieval_query (string), expected_evidence_type (string), and is_fillable (boolean). If is_fillable is false or the output fails schema validation, the agent must not proceed with retrieval. Instead, log the gap report and either request human clarification or return a bounded answer with an explicit caveat about the unresolved gap. For high-stakes domains, implement a human review queue that surfaces gap reports where is_fillable is false or expected_evidence_type indicates a policy-sensitive fact. Use a model with strong instruction-following and JSON mode (such as GPT-4o or Claude 3.5 Sonnet with structured outputs enabled) to minimize parsing failures. Set temperature=0 to reduce variability in gap classification.

Build a retry-and-repair layer around this prompt. If the output fails JSON schema validation, retry once with the validation error message appended to the prompt as a repair instruction. If the second attempt also fails, log the raw output, increment a gap_fill_failure metric, and fall back to a safe response path—do not loop indefinitely. Instrument the harness to log every gap-fill attempt: the input reasoning chain, the identified gap, the generated query, the retrieval results that follow, and whether the gap was ultimately resolved. This trace data is essential for debugging multi-hop failures and for evaluating whether the gap-filling prompt is generating queries that actually retrieve useful evidence. Use these logs to build a golden dataset for regression testing the prompt across model updates.

The retrieval system that consumes the output query must be prepared for targeted, evidence-seeking queries rather than broad topic searches. The retrieval_query field should be passed directly to your vector store, keyword index, or hybrid retrieval endpoint. Do not rewrite or expand it unless you have a separate query expansion step with its own validation. After retrieval, the new evidence should be appended to the reasoning chain context, and the agent should re-attempt the reasoning step that produced the gap. This creates a tight feedback loop: reason, detect gap, fill gap, retrieve, reason again. Avoid the common failure mode of treating the gap-filling prompt as a one-shot fix; it works best as part of a bounded iterative loop with a maximum of three gap-fill attempts before the agent concedes and reports what it cannot resolve.