Multi-Hop Question Decomposition Prompt Template

This prompt is for AI architects and RAG pipeline engineers who need to answer a single complex question that cannot be resolved with one retrieval round. The job-to-be-done is transforming a dense, multi-part user query into an ordered, dependency-aware decomposition plan. The ideal user is someone building a production system—such as a legal research assistant, a clinical evidence review tool, or a financial due diligence copilot—where missing a dependency or retrieving evidence in the wrong order produces an incomplete or factually incorrect final answer. You should use this prompt when the user's question contains implicit sub-questions, requires information that must be gathered sequentially, or spans multiple documents that cannot be assumed to coexist in a single retrieval context window.

Do not use this prompt for simple factoid lookups, single-hop questions, or queries that can be answered by a single well-constructed retrieval call. It is also the wrong tool when the system already has a static, pre-defined decomposition graph for known question types—hard-coded logic will be cheaper and more reliable. This prompt is a reasoning step, not a retrieval step; it produces a plan, not the final answer. If your system lacks a downstream executor that can run the sub-questions, retrieve evidence for each, and merge the results, the decomposition plan alone provides no user value. In high-stakes domains such as healthcare or legal, the decomposition plan itself should be reviewed or logged as part of the audit trail before any sub-question execution begins.

Before wiring this into your application, confirm that you have a clear merge strategy for the sub-answers and a defined stopping condition for recursive or sequential decomposition. The output of this prompt is structured JSON—validate it against a schema before using it to drive retrieval. If the decomposition plan contains circular dependencies, missing evidence types, or sub-questions that are not independently answerable, reject the plan and retry or escalate. Start by testing this prompt against a golden set of complex questions where you already know the correct decomposition, and measure whether the output plan covers all necessary information needs in the right order.

This prompt template provides the core instruction set for transforming a single complex question into a structured decomposition plan. It is designed to be dropped into a system prompt or a user message in a RAG pipeline where the model must reason about what it needs to know before it can answer. The template uses square-bracket placeholders that you must replace with your specific inputs, output schema, constraints, and examples before use. The goal is to produce a plan that a downstream orchestrator can execute, not to answer the question directly.

textCopy
You are a question decomposition specialist. Your task is to break a complex user question into a set of ordered, dependency-aware sub-questions that can be answered independently from a knowledge base. Do not answer the original question. Do not answer any sub-question. Produce only the decomposition plan.

## INPUT
User Question: [USER_QUESTION]

## OUTPUT SCHEMA
Return a JSON object with the following structure:
{
  "original_question": "string",
  "decomposition_strategy": "string (e.g., sequential, parallel, hybrid)",
  "sub_questions": [
    {
      "id": "string (e.g., SQ1)",
      "question": "string (self-contained, answerable sub-question)",
      "depends_on": ["string (list of sub-question IDs this one requires before execution)"],
      "expected_evidence_type": "string (e.g., definition, numerical_data, event_description, policy_statement)",
      "purpose": "string (why this sub-question is necessary for the final answer)"
    }
  ],
  "merge_strategy": "string (description of how sub-answers should be combined into a final answer)",
  "estimated_retrieval_queries": ["string (suggested search queries for each independent retrieval step)"]
}

## CONSTRAINTS
[CONSTRAINTS]

## EXAMPLES
[EXAMPLES]

## CONTEXT (OPTIONAL)
[CONTEXT]

To adapt this template, start by replacing [USER_QUESTION] with the actual question your system receives. The [CONSTRAINTS] placeholder should be populated with domain-specific rules, such as 'Sub-questions must not introduce assumptions not present in the original question' or 'Limit decomposition to a maximum depth of 2 levels.' The [EXAMPLES] block is critical for few-shot performance; include at least two worked examples showing a complex question and its correct decomposition into the JSON schema. The optional [CONTEXT] field can carry document metadata, user profile information, or prior conversation turns that inform the decomposition strategy. After generating the plan, validate the output against your schema before passing sub-questions to your retrieval engine. For high-stakes domains, route the decomposition plan to a human reviewer or an LLM-as-judge evaluation step that checks for logical ordering, missing dependencies, and sub-question atomicity before execution.

The Multi-Hop Question Decomposition prompt is rarely a standalone step. In production, it sits between the user's original question and a retrieval engine. The harness must accept a complex question, call the decomposition prompt, parse the structured decomposition plan, and then route each sub-question to the appropriate retrieval system. The output schema—typically a JSON object containing an ordered list of sub-questions with dependency labels and expected evidence types—must be strictly validated before any downstream retrieval begins. A malformed decomposition plan that passes through unchecked will cause silent failures in later pipeline stages, so the harness should reject any response that does not conform to the expected schema and trigger a retry or fallback.

Wire the prompt into an async pipeline or agentic loop. After receiving the decomposition plan, iterate through sub-questions in dependency order: independent sub-questions can fan out to parallel retrieval calls, while dependent sub-questions must wait for their prerequisite answers. For each sub-question, log the retrieval query, the retrieved context, and the sub-answer before feeding them into the merge step. Implement a circuit breaker: if any sub-question retrieval returns zero results or the sub-answer confidence score falls below a threshold, halt the chain and either re-plan with a revised query or escalate to a human reviewer. Model choice matters here—use a model with strong instruction-following and structured output capabilities (such as GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro) and set response_format to JSON mode with the decomposition schema enforced. For high-stakes domains like legal or clinical RAG, add a human-in-the-loop approval gate after decomposition before any retrieval executes.

Observability is non-negotiable. Log the original question, the full decomposition plan, the model and prompt version used, the latency of the decomposition call, and any validation errors or retries. This trace data is essential for debugging when the final answer is wrong—you need to know whether the error originated in the decomposition, the retrieval, or the synthesis. Build eval checks into the harness: after decomposition, run a lightweight LLM-as-judge check that scores the plan on completeness (are all aspects of the original question covered?), logical ordering (are dependencies correct?), and atomicity (is each sub-question independently answerable?). If the score falls below a threshold, re-prompt with the original question and the eval feedback. Avoid the temptation to skip validation in favor of speed; a bad decomposition plan poisons every downstream step and is far more expensive to fix after retrieval than before.