Dependency-Aware Query Decomposition for Hybrid Search Prompt

This prompt is designed for infrastructure teams operating multi-index retrieval systems—specifically those managing vector, keyword, and graph backends simultaneously. The core job-to-be-done is translating a complex, multi-hop user question into a structured execution plan that assigns each decomposed sub-query to the most appropriate retrieval backend. You should use this prompt when a single retrieval round is insufficient, and the question requires chaining evidence from different types of indexes (e.g., finding a person via a keyword search, then using their entity ID to traverse a graph for relationships, then performing a semantic search on related documents). The ideal user is an AI engineer or search architect who already has a routing or orchestration layer and needs a reliable decomposition step that respects backend capabilities.

This prompt is not a general-purpose query rewriter. Do not use it for simple factoid questions answerable from a single vector index, or for pure conversational context expansion. It is also not a substitute for a query router—it assumes you have already determined that the question requires decomposition and hybrid retrieval. The prompt requires you to provide a catalog of available backends with their capabilities (e.g., [BACKEND_CATALOG]), so if your system only has one index type, this prompt adds unnecessary complexity. Avoid using this prompt in latency-sensitive, real-time chat without caching or pre-computation, as the decomposition step itself consumes tokens and time before any retrieval begins. It is also inappropriate for questions where the dependency chain is ambiguous and requires user clarification—this prompt plans, it does not interactively disambiguate.

Before implementing, ensure you have a validation harness that checks the output plan for backend assignment correctness, dependency cycle detection, and orphan sub-queries. The prompt outputs a machine-readable plan, but you must verify that the assigned backends actually support the query types specified. Next, proceed to the Prompt Template section to copy the base template, then review the Implementation Harness for wiring it into a production retrieval pipeline with retries, logging, and fallback strategies.

This template is the core instruction set for a dependency-aware query decomposition engine. It takes a complex user question and a description of available retrieval backends, then produces a structured plan. The plan identifies sub-queries, their dependencies, and the most suitable search index for each. Use this as the system prompt for a capable frontier model when building an orchestration layer for a multi-index RAG system.

textCopy
You are a query planner for a multi-index retrieval system. Your job is to analyze a user's question and decompose it into a set of sub-queries. For each sub-query, you must assign it to the optimal retrieval backend and define its dependencies on other sub-queries.

## Available Backends
[BACKEND_DESCRIPTIONS]

## Task
1.  **Analyze the Question:** Identify the core information needs, implicit assumptions, and the type of reasoning required (e.g., factual lookup, entity traversal, comparison, temporal sequencing).
2.  **Decompose into Sub-Queries:** Break the question into the smallest set of independent or dependent sub-queries needed to gather all evidence.
3.  **Assign Backends:** For each sub-query, select the single best backend from the list above based on the query's characteristics (e.g., precise entity match, semantic similarity, relationship traversal, time-bound filter).
4.  **Define Dependencies:** Explicitly state which sub-query IDs must be executed before another. Mark sub-queries that can run in parallel.
5.  **Output Plan:** Generate a JSON object strictly conforming to the [OUTPUT_SCHEMA].

## Constraints
[CONSTRAINTS]

## Examples of Good Decomposition
[EXAMPLES]

## User Question
[USER_QUESTION]

To adapt this template, you must provide concrete values for the square-bracket placeholders. The [BACKEND_DESCRIPTIONS] should be a detailed list of your available indexes, including their strengths, weaknesses, and ideal query types (e.g., 'keyword_index: Best for exact entity names and IDs. Weak for paraphrases.'). The [CONSTRAINTS] field is critical for controlling cost and latency; use it to specify rules like 'Generate no more than 5 sub-queries' or 'Prefer parallel execution when possible.' The [OUTPUT_SCHEMA] placeholder should be replaced with a strict JSON schema definition, which you will use to validate the model's output before execution. The [EXAMPLES] placeholder is your most powerful lever for improving performance; provide 2-3 few-shot examples of a complex question and its correct decomposition plan to teach the model your specific routing logic and dependency syntax. After copying this template, integrate it into your application harness where the [USER_QUESTION] is injected at runtime.

This prompt is not a standalone chatbot. It is a planning stage inside a multi-index retrieval pipeline. The application calls the model with a user question, receives a structured decomposition plan, and then executes each sub-query against the assigned backend. The harness is responsible for validating the plan before execution, routing sub-queries to the correct index, collecting results, and feeding them into a downstream synthesis step. Treat the model output as an untrusted plan that must pass structural and logical checks before any retrieval I/O occurs.

Validation and routing logic must run in application code, not in the prompt. After parsing the JSON output, validate that every backend field is one of your allowed values (vector, keyword, graph). Check that sub-queries marked as dependent on a prior step reference a depends_on ID that actually exists in the plan. Reject plans with circular dependencies or orphaned steps. For each sub-query, confirm that the assigned backend matches the query characteristics: keyword queries should contain literal terms and filters, vector queries should be natural-language passages, and graph queries should specify entity types and relationship traversals. If validation fails, log the rejection reason, increment a plan_rejection metric, and retry with a stricter prompt variant or fall back to a simpler single-hop retrieval.

Execution order must respect the dependency graph. Topological sort the sub-queries so that independent steps run in parallel and dependent steps wait for their upstream results. For each completed step, extract the structured intermediate answer (entities, IDs, date ranges) and inject it into the dependent sub-query's template before execution. The prompt's [INTERMEDIATE_CONTEXT] placeholder is filled by the harness, not by the model at plan time. Log every sub-query, its assigned backend, latency, result count, and any empty-result-set events. These logs feed eval dashboards that track decomposition accuracy, backend assignment precision, and end-to-end answer quality. If a sub-query returns zero results, trigger a repair path: re-prompt with the failed step and ask for an alternative query or backend reassignment, but cap repair attempts at two to avoid latency spirals.

Model choice and latency budgeting matter here. This prompt produces structured JSON with dependency logic, so use a model with strong instruction-following and JSON output reliability (GPT-4o, Claude 3.5 Sonnet, or equivalent). Set response_format to json_object or use structured output mode if available. Budget 2-4 seconds for plan generation; if latency exceeds this, consider caching common decomposition patterns or using a smaller model for simple single-hop questions and only invoking this prompt when a classifier detects multi-hop intent. Wire the entire pipeline through an observability layer that traces each sub-query from plan generation through retrieval to final synthesis, so you can attribute answer failures to decomposition errors, backend misassignment, or retrieval gaps.