Session-Aware Sub-Question Generation Prompt

This prompt is for RAG architects and agent builders who need to answer complex, multi-part follow-up questions that cannot be resolved with a single retrieval round. The core job-to-be-done is decomposing a user's current question into a set of independently retrievable sub-questions while preserving the dependencies and constraints established in prior conversation turns. Without this step, a retrieval system treats each turn in isolation, losing critical context like 'What about the second one?' or 'How does that compare to the previous approach?' and producing irrelevant or incomplete results.

The ideal user is an engineer integrating this prompt into a retrieval pipeline or an agent loop where the system must gather evidence across multiple documents before synthesizing an answer. Required context includes the full conversation history, the current user query, and optionally a schema of the available knowledge base to ground sub-questions in retrievable entities. Do not use this prompt for simple factual lookups, single-turn Q&A, or cases where the follow-up is a trivial clarification. It is also not a substitute for a full dialogue state tracker; it assumes the prior turns contain the necessary referents and does not ask the user for missing information.

Before wiring this into production, define clear failure boundaries. The prompt can over-decompose a simple question into unnecessary sub-questions, wasting retrieval budget and latency. It can also hallucinate dependencies that do not exist in the conversation history, creating retrieval paths that lead to fabricated answers. Always validate that generated sub-questions are grounded in explicit prior-turn content, and implement a circuit breaker that limits the number of sub-questions per turn. Pair this prompt with a regression suite that includes cross-turn dependency tests: anaphora, ellipsis, comparative references, and implicit topic carry-forward.

This section provides the core prompt template for session-aware sub-question generation. The prompt instructs the model to decompose a follow-up question into independently answerable sub-questions while preserving dependencies established in prior conversation turns. Use this template as the foundation for your decomposition step in multi-turn RAG pipelines where compound questions require stepwise evidence gathering across retrieval rounds.

textCopy
You are a query decomposition engine for a multi-turn retrieval system. Your job is to break a follow-up question into a sequence of sub-questions that can be answered independently, respecting dependencies established in the conversation history.

## CONVERSATION HISTORY
[CONVERSATION_HISTORY]

## CURRENT QUESTION
[CURRENT_QUESTION]

## OUTPUT SCHEMA
Return a JSON object with the following structure:
{
  "original_question": "string (the current question exactly as provided)",
  "dependencies_from_history": [
    {
      "turn_index": "integer (which prior turn this dependency comes from)",
      "entity_or_fact": "string (the specific entity, fact, or constraint carried forward)",
      "resolution": "string (how this dependency is resolved in the current context)"
    }
  ],
  "sub_questions": [
    {
      "order": "integer (execution order, starting at 1)",
      "question": "string (a self-contained question that can be answered independently)",
      "depends_on": ["integer (list of sub_question order numbers this question depends on, empty array if none)"],
      "expected_answer_type": "string (entity, list, fact, comparison, summary, yes_no, or numeric)",
      "reasoning": "string (why this sub-question is necessary and how it relates to dependencies)"
    }
  ],
  "decomposition_strategy": "string (brief explanation of the overall decomposition approach)"
}

## CONSTRAINTS
- Every sub-question must be self-contained and answerable without referring back to the conversation.
- Resolve all pronouns, implicit references, and elliptical constructions before writing sub-questions.
- If the current question is already simple and self-contained, return a single sub-question.
- Order sub-questions so that dependency questions appear before the questions that depend on them.
- Do not invent facts not present in the conversation history or current question.
- If a dependency from history is ambiguous, flag it in the dependencies_from_history array with a note in the resolution field.
- Maximum of [MAX_SUB_QUESTIONS] sub-questions. If decomposition would exceed this, prioritize the most critical questions and note the truncation in decomposition_strategy.

## EXAMPLES
[FEW_SHOT_EXAMPLES]

## RISK LEVEL
[RISK_LEVEL]

Adapt this template by adjusting the output schema fields to match your downstream retrieval and answering architecture. If your system uses structured metadata filters, add a metadata_filters field to each sub-question. For agentic systems that execute tool calls between sub-questions, add a required_tools array. Replace [FEW_SHOT_EXAMPLES] with 2-3 annotated examples showing correct dependency resolution across turns. Set [MAX_SUB_QUESTIONS] based on your latency budget—typically 3-5 for interactive systems, up to 10 for batch or async workflows. The [RISK_LEVEL] placeholder should be set to low, medium, or high to trigger appropriate validation behavior in your harness. Before deploying, validate that the model correctly resolves cross-turn dependencies by running the test cases described in the evaluation section of this playbook.

Wiring this prompt into an application requires treating it as a structured pre-retrieval step within a multi-turn RAG pipeline. The prompt expects two primary inputs: the current user query and a serialized representation of the conversation history. Before calling the model, the application must assemble the history by extracting prior user turns and assistant responses from the session store, formatting them as a clear dialogue transcript. The model choice matters here—use a model with strong instruction-following and structured output capabilities (such as GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro) because the decomposition task requires precise dependency tracking. Set the temperature low (0.0–0.2) to maximize deterministic, repeatable sub-question generation across identical inputs.

The output must be parsed as structured JSON containing an array of sub-question objects, each with a question string, a dependency field indicating which prior sub-question index it depends on (or null if independent), and a rationale explaining the decomposition logic. Implement a post-processing validator that checks: (1) the output is valid JSON, (2) all dependency indices reference existing sub-questions, (3) no circular dependencies exist in the dependency graph, and (4) every sub-question is a self-contained, answerable query. If validation fails, retry the prompt once with the validation error appended as feedback. If the retry also fails, log the failure, fall back to treating the original user query as a single retrieval query, and flag the session for review. For high-stakes domains, route sessions with persistent decomposition failures to a human review queue rather than silently degrading.

Logging and observability are critical for debugging cross-turn dependency resolution. Log the full prompt input, the raw model output, the parsed sub-questions, the dependency graph, and any validation failures. Instrument metrics for decomposition depth (number of sub-questions generated), dependency chain length, validation pass rate, and retry frequency. These metrics will surface when the model over-decomposes simple queries, creates unnecessary dependencies, or fails to resolve anaphora from the conversation history. When deploying, start with a canary release on a subset of sessions and compare retrieval precision and answer completeness against a baseline that treats follow-up questions as standalone queries. If the decomposition step adds latency that violates your p95 targets, consider caching decomposition results for identical query-history pairs or using a smaller, faster model for simple single-hop queries detected by a lightweight classifier upstream.