Gap-Driven Follow-Up Question Generation Prompt Template

This prompt is designed for conversational RAG systems that encounter evidence gaps during multi-turn interactions. When retrieved context is insufficient to fully answer a user's question, this prompt generates targeted follow-up questions that address specific missing information. It ranks questions by information gain so the system can request clarification efficiently rather than asking generic or redundant questions. Use this prompt when your copilot or chat assistant needs to distinguish between 'answerable with current context' and 'requires user input to proceed' states.

The ideal integration point is after retrieval and evidence sufficiency checking but before answer generation. Your pipeline should already have attempted retrieval, run a pre-answer sufficiency check, and identified that critical information is missing. This prompt takes the gap analysis output—a structured list of what the context cannot address—and converts it into natural, conversational follow-up questions the user can actually answer. It is not a replacement for retrieval expansion or query rewriting. If the gap can be filled by searching a different index, rewriting the query, or expanding the retrieval scope, do that first. Reserve this prompt for gaps that genuinely require human input: missing parameters, ambiguous intent, domain-specific knowledge the user possesses, or clarification between conflicting interpretations.

Do not use this prompt when the system can reasonably infer the missing information from conversation history, user profile data, or additional retrieval rounds. Over-questioning users degrades trust and increases abandonment. Similarly, avoid this prompt for yes/no or trivial clarification requests that a well-designed UI element could handle more efficiently. The prompt works best when each generated question targets a specific, high-value information gap and the question ranking helps the system decide whether to ask one question, present a short list, or escalate to a human agent. Before deploying, test against a golden set of gap scenarios to verify that generated questions are specific, answerable, and correctly prioritized by information gain.

This prompt template is designed for conversational RAG systems that need to request clarification or additional context from users when retrieved evidence is insufficient. It generates follow-up questions that directly target identified evidence gaps, ranked by their potential information gain. The template expects you to provide the current user question, retrieved context, and a structured inventory of what's missing. Replace each square-bracket placeholder with your application's actual data before sending to the model.

textCopy
You are a follow-up question generator for a conversational RAG system. Your job is to produce targeted questions that address specific evidence gaps in the retrieved context.

## USER QUESTION
[USER_QUESTION]

## RETRIEVED CONTEXT
[RETRIEVED_CONTEXT]

## IDENTIFIED EVIDENCE GAPS
[EVIDENCE_GAPS]

## CONVERSATION HISTORY (IF ANY)
[CONVERSATION_HISTORY]

## INSTRUCTIONS
1. Review each evidence gap and determine whether a follow-up question to the user could resolve it.
2. Generate at most [MAX_QUESTIONS] follow-up questions.
3. Rank questions by information gain: prioritize questions whose answers would unlock the largest portion of the original question.
4. Each question must be:
   - Specific and answerable in a short user response.
   - Directly traceable to a listed evidence gap.
   - Phrased in natural, conversational language.
   - Non-redundant with information already present in the retrieved context.
5. Do not ask questions that the retrieved context already answers.
6. Do not ask questions that require the user to have specialized knowledge they are unlikely to possess.
7. If no useful follow-up questions can be asked, return an empty list and explain why.

## OUTPUT FORMAT
Return a JSON object with this exact structure:
{
  "questions": [
    {
      "question": "string",
      "targets_gap": "string (reference to the specific gap from EVIDENCE_GAPS)",
      "information_gain": "high|medium|low",
      "rationale": "string (why this question is worth asking)"
    }
  ],
  "no_questions_reason": null | "string (only if questions array is empty)"
}

## CONSTRAINTS
- [CONSTRAINTS]

Adaptation guidance: The [EVIDENCE_GAPS] placeholder should contain structured gap descriptions, ideally from an upstream gap detection prompt such as the Retrieval Gap Analysis or Evidence Gap Report Generation templates. The [MAX_QUESTIONS] value should be tuned to your UX: 2-3 questions work well for chat interfaces, while 1 question is safer for voice or mobile. The [CONSTRAINTS] placeholder lets you inject domain-specific rules, such as prohibiting questions about personally identifiable information or restricting questions to a specific taxonomy. If your system lacks conversation history, omit that section entirely rather than passing an empty placeholder. Test the output by verifying that every generated question maps to at least one gap in your input and that no question duplicates information already present in [RETRIEVED_CONTEXT].

This prompt operates inside a conversational RAG loop where the system has already retrieved context and determined that evidence is insufficient to answer the user's question. The harness should invoke this prompt only after an upstream sufficiency check (such as a Pre-Answer Sufficiency Check or Answer Readiness Gate) returns a negative decision. The prompt's output—a ranked list of follow-up questions—must be validated before surfacing to the user, because a poorly formed question can degrade trust faster than a simple 'I don't know.' Wire the prompt as a conditional branch: if the sufficiency score falls below your configured threshold, call this prompt with the original question, the retrieved context, and the gap categories identified by the upstream check.

Validation should enforce three constraints. First, each generated question must be answerable from the system's known document corpus—run a quick retrieval check on each question and discard any that return zero results or only low-relevance passages. Second, questions must not repeat the user's original query verbatim; use a simple string-similarity threshold (e.g., cosine similarity below 0.85) to filter near-duplicates. Third, validate the output schema: each question object must contain a non-empty question string, a gap_addressed category matching one of the upstream gap labels, and an information_gain score between 0.0 and 1.0. If validation fails, retry once with the validation errors appended to the prompt as [CONSTRAINTS] feedback. After two failed attempts, fall back to a static clarification template rather than looping indefinitely.

Human review gates are appropriate when the system is deployed in high-stakes domains (healthcare, legal, finance) or when the information gain scores cluster below 0.3, indicating that even the best follow-up questions are unlikely to resolve the gap. In these cases, route the generated questions to a review queue with the original user query, the retrieved context, and the sufficiency report. The reviewer can approve, edit, or reject the questions before they reach the user. Log every invocation—including the sufficiency score, gap categories, generated questions, validation results, and final action (shown, retried, escalated, or reviewed)—to build a dataset for tuning your abstention and follow-up thresholds over time. Avoid wiring this prompt directly into a customer-facing loop without these guardrails; an over-eager clarification request feels like a broken bot.