Index-Aware Query Planning Prompt for Heterogeneous Backends

This prompt is designed for infrastructure teams operating retrieval pipelines that span multiple backend types—dense vector stores, sparse keyword indexes (like BM25), and graph databases—where each backend has a distinct schema, query language, and performance profile. The core job-to-be-done is translating a single, unstructured user question into a structured query plan that maps the user's intent to backend-specific query formulations. Without this step, teams often resort to sending the same raw query string to every index, which leads to poor recall on sparse backends, nonsensical graph traversals, and wasted compute. The ideal user is a platform engineer, search architect, or RAG pipeline owner who already has a routing or fan-out dispatch layer and needs a reliable, machine-readable plan to drive it.

You should use this prompt when your retrieval topology includes at least two backends with materially different capabilities—for example, a dense index that expects natural language passages and a graph index that requires explicit entity IDs and relationship types. It is also appropriate when your indexes have different metadata schemas, requiring date filters on one backend but not another, or when one backend supports faceted search while another does not. The prompt expects you to provide a description of each available backend, including its schema, supported query operations, and any constraints (e.g., 'graph index only supports single-hop traversals'). The output is a JSON plan that can be parsed by your dispatch service to construct and execute backend-specific queries in parallel. Do not use this prompt for single-backend deployments, for setups where all indexes share an identical schema, or for simple synonym expansion tasks that don't require backend awareness.

Before adopting this prompt, verify that your retrieval dispatch layer can consume a structured plan with per-backend query objects. If your current architecture simply broadcasts a raw string to every index, you'll need to add a plan-parsing step. Start by running the prompt against a representative sample of user queries and manually inspecting whether the generated plans respect each backend's documented capabilities. Common failure modes include the model inventing query operators that don't exist for a given backend or omitting a backend that should have been included. If your retrieval pipeline is high-stakes—for example, in regulated domains where missed documents have compliance implications—add a validation layer that checks each generated query against the backend's actual query schema before execution, and log plan-to-execution traces for auditability.

This prompt template is the core instruction set for an LLM acting as a query planner over heterogeneous retrieval backends. It forces the model to reason about each available index's strengths, schema constraints, and query language before producing a structured plan. The output is not a single query but a mapping of backend identifiers to their respective query formulations, along with a justification for each choice. Use this template when your retrieval architecture includes at least two distinct index types (e.g., a dense vector store, a BM25 keyword index, and a graph database) that require different query shapes.

codeCopy
You are a query planner for a multi-backend retrieval system. Your job is to analyze a user's question and produce a query plan that maps the question to backend-specific query formulations. You must respect each backend's capabilities, schema, and query language. Do not generate a single query for all backends; instead, produce a plan that assigns the right query shape to each backend.

## AVAILABLE BACKENDS
[BACKEND_DEFINITIONS]

## USER QUESTION
[USER_QUESTION]

## CONVERSATION CONTEXT (if any)
[CONVERSATION_HISTORY]

## CONSTRAINTS
- Each backend query must be valid for that backend's query language and schema.
- Do not assign a query to a backend that cannot answer the question type.
- If a backend is irrelevant, mark it as "SKIP" with a brief reason.
- Prefer parallel execution where dependencies allow; mark sequential dependencies explicitly.
- Include any metadata filters, date ranges, or entity constraints extracted from the question.
- If the question is ambiguous, generate clarifying variants rather than guessing.

## OUTPUT SCHEMA
Return a JSON object with this exact structure:
{
  "plan_id": "string, unique identifier for this plan",
  "question_analysis": {
    "intent": "string, classified intent of the question",
    "required_capabilities": ["list of required retrieval capabilities"],
    "extracted_entities": ["list of entities mentioned"],
    "extracted_filters": {"field": "value"},
    "ambiguity_notes": "string or null"
  },
  "backend_plans": [
    {
      "backend_id": "string, matches BACKEND_DEFINITIONS",
      "action": "QUERY | SKIP",
      "skip_reason": "string if SKIP, otherwise null",
      "query_formulation": {
        "query_type": "string, e.g., dense_vector, bm25_keyword, graph_traversal, hybrid",
        "query_body": "string, the actual query text or structured query object",
        "filters": {},
        "top_k": number,
        "embedding_hint": "string or null, guidance for embedding model selection"
      },
      "justification": "string, why this formulation fits this backend and question",
      "depends_on": ["backend_id or null, if this query needs results from another backend first"]
    }
  ],
  "execution_order": ["ordered list of backend_ids for sequential execution"],
  "merge_strategy": "string, brief note on how results should be combined"
}

## EXAMPLES
[FEW_SHOT_EXAMPLES]

## RISK LEVEL
[RISK_LEVEL]

To adapt this template, start by populating [BACKEND_DEFINITIONS] with a structured description of each available index. For each backend, specify its type (dense, sparse, graph, hybrid), its query language or API constraints, its schema fields, and the kinds of questions it handles well. The [FEW_SHOT_EXAMPLES] placeholder should contain at least two worked examples showing a user question and the correct output plan. For high-stakes domains where incorrect retrieval could cause harm, set [RISK_LEVEL] to HIGH and add a human-review step before query execution. The [CONVERSATION_HISTORY] placeholder is optional but critical for multi-turn retrieval scenarios where the current question depends on prior context. Before deploying, validate that the model's output conforms to the JSON schema using a structural validator, and run at least five test questions through the plan to check that backend assignments are correct and query formulations are executable against your actual indexes.

This prompt is designed to sit at the entry point of a multi-backend retrieval pipeline. It receives a raw user question and produces a structured query plan that maps the question to backend-specific query formulations. The output is not a final answer but a machine-readable plan consumed by downstream orchestrators. You should treat this prompt as a planning step, not a retrieval step. Its job is to decide what to ask each index, not to execute those queries. In production, the prompt runs once per user question before any retrieval calls are made. The resulting plan is then dispatched to a fan-out execution layer that sends each backend-specific query to its respective index in parallel or in dependency order.

Integration pattern: Wrap the prompt in a lightweight service or serverless function that accepts a user_query string and an optional session_context object. The function should inject the available backend capabilities and schemas from a configuration store or environment variable, not from user input. A typical request payload includes [USER_QUESTION], [BACKEND_CAPABILITIES] (a JSON array describing each index type, its schema, supported operators, and performance characteristics), and [CONSTRAINTS] (latency budgets, result count targets, and any mandatory filters). The model returns a JSON plan with a backends array, each entry containing a backend_type, a query_formulation, and optional dependency_on fields for multi-hop ordering. Validation layer: Before the plan reaches the execution layer, run structural validation: check that every backend_type in the plan exists in the declared capabilities, that no query formulation references fields or operators the backend doesn't support, and that dependency chains are acyclic. If validation fails, log the failure, capture the raw model output, and either retry with a more explicit constraint message or fall back to a default single-backend query. For high-stakes applications, add a human review gate for plans that introduce new backend combinations not seen in the evaluation dataset.

Model choice and latency: This is a reasoning-heavy prompt that benefits from models with strong instruction-following and structured output capabilities. GPT-4o, Claude 3.5 Sonnet, and similarly capable models perform well. Expect 1-3 seconds of latency for plan generation. Cache the backend capabilities schema in the system prompt prefix to reduce token reprocessing. Observability: Log every generated plan alongside the user query, the backend capabilities snapshot, validation results, and the eventual retrieval quality metrics (recall@k, precision, or task-specific scores). This traceability lets you debug when a poor retrieval result traces back to a bad query plan rather than a bad index. Failure modes to monitor: Plans that route all queries to a single backend despite available alternatives (under-utilization), plans that generate syntactically valid but semantically empty queries for a backend, and plans with circular dependencies. Set up alerts on validation failure rate and on plan-to-execution latency exceeding your retrieval SLA. Next step: After the plan executes and results return, feed the plan and results into a result harmonization prompt to merge, deduplicate, and rank the multi-backend output into a unified result set.