Parallel Query Generation Prompt for Multi-Stage Retrieval Pipelines

This prompt is for RAG architects and infrastructure engineers who need to fan out a single user question into multiple independent query variants, each optimized for a different retrieval backend. The job-to-be-done is producing a batch of queries that can execute in parallel across dense vector, sparse keyword, and graph indexes, with metadata tags that let your routing layer dispatch each variant to the correct backend without additional parsing. You should use this prompt when your retrieval pipeline includes at least two heterogeneous indexes, when query latency matters enough to avoid sequential rewriting chains, and when you need each variant to carry its own backend routing instructions. Do not use this prompt for simple single-index RAG setups, for conversational follow-up resolution where session context is the primary challenge, or for query decomposition where sub-questions have sequential dependencies rather than parallel independence.

The prompt expects a user question, a list of available backend types with their capabilities, and an output schema that enforces one query variant per backend with explicit routing metadata. Each generated variant must be self-contained and independently executable—variant A for dense retrieval should not depend on results from variant B for sparse retrieval. The prompt template includes placeholders for backend capability descriptions, output format constraints, and evaluation criteria that check for variant independence and coverage completeness. Before integrating this into production, you must validate that every variant maps to a real backend in your infrastructure, that no two variants are semantically identical, and that the routing tags match your actual dispatch logic. Missing or misrouted variants are the most common failure mode.

The primary risk is generating variants that look different but retrieve overlapping result sets, which wastes compute and dilutes the value of parallel execution. Mitigate this by adding eval checks that measure semantic distance between variants and flag pairs below a similarity threshold. A secondary risk is producing variants for backends that don't exist or are unavailable, which causes silent failures in your routing layer. Always validate backend tags against a live registry before execution. If your pipeline includes regulated content or high-stakes decisions, add a human review step that samples variant sets and confirms they cover the user's intent without introducing hallucinated constraints. Start by wiring this prompt into a staging pipeline with logging on variant diversity, backend coverage, and result set overlap before promoting to production.

This prompt template is the core instruction set for a fan-out retrieval architecture. It takes a single user question and produces a batch of independent query variants, each optimized for a different retrieval backend (dense vector, sparse keyword, graph). The template uses square-bracket placeholders for all dynamic inputs, making it safe to wire directly into an application harness. The output is a structured JSON object containing an array of query variants, each tagged with a target backend, a confidence weight, and a rationale for the formulation. This allows a downstream orchestrator to dispatch each variant to the correct index in parallel and merge the results.

textCopy
You are a retrieval query planner for a multi-stage RAG pipeline. Your task is to analyze the user's question and generate a batch of independent query variants, each optimized for a specific retrieval backend. The backends available are: [BACKEND_LIST].

## User Question
[USER_QUESTION]

## Conversation Context (if any)
[CONVERSATION_HISTORY]

## User Profile and Permissions
[USER_PROFILE]

## Output Schema
Return a valid JSON object with the following structure:
{
  "original_query": "string",
  "query_variants": [
    {
      "variant_id": "string",
      "backend": "dense_vector" | "sparse_keyword" | "graph",
      "query_text": "string",
      "confidence_weight": 0.0-1.0,
      "rationale": "string"
    }
  ]
}

## Constraints
[CONSTRAINTS]

## Examples
[EXAMPLES]

To adapt this template, replace each placeholder with concrete values from your application context. [BACKEND_LIST] should be a comma-separated list of available backends (e.g., 'dense_vector, sparse_keyword, graph'). [USER_QUESTION] is the raw user input. [CONVERSATION_HISTORY] can be an empty string or a formatted transcript of prior turns to resolve anaphora. [USER_PROFILE] should include role, permissions, and any access scope restrictions that affect what documents the user is allowed to see. [CONSTRAINTS] is where you inject operational rules, such as 'Do not generate more than 5 variants' or 'Exclude any query that references PII.' [EXAMPLES] should contain 1-3 few-shot demonstrations of the expected input-output pattern, including edge cases like ambiguous queries or questions that require multi-hop reasoning. After generating the variants, always validate the JSON structure against your schema before dispatching to backends. For high-stakes domains, add a human review step before the variants are executed against production indexes.

This prompt is designed to be called as a fan-out service within a multi-stage retrieval pipeline. The application layer should receive a single user query, inject it into the prompt template along with the available backend capabilities and output schema, and then parse the structured response into a set of independent query objects. Each generated query variant must include a target_backend tag (dense, sparse, graph, hybrid) and a query_string payload so the downstream dispatcher can route it to the correct index without additional parsing. The prompt is stateless by design: do not include conversation history or session context unless the user query explicitly depends on prior turns, and even then, prefer resolving anaphora in the application layer before calling this prompt.

Validation and retry logic should be implemented in the harness, not in the prompt. After receiving the model response, validate that: (1) the output is valid JSON matching the expected schema, (2) every variant has a non-empty query_string and a recognized target_backend value, (3) no two variants are near-duplicates (use a simple token-overlap threshold or embedding cosine similarity check), and (4) the total number of variants falls within the configured min/max bounds. If validation fails, retry with the same prompt but append the validation errors to the [CONSTRAINTS] field as explicit correction instructions. After two retries, fall back to a simpler single-variant generation or escalate to a human reviewer if the retrieval pipeline is customer-facing. Log every generation attempt, validation result, and retry decision to your observability stack with the query_id and prompt_version attached.

Model selection matters for this prompt. The task requires structured output generation with high schema adherence, so prefer models with strong JSON mode or function-calling support (GPT-4o, Claude 3.5 Sonnet, or equivalent). Avoid models under ~7B parameters unless you have validated their schema-following reliability on your specific backend taxonomy. For high-throughput pipelines, consider caching the prompt prefix (the system instruction and backend capability descriptions) to reduce latency and cost. The user query and any dynamic constraints should be the only variable suffix. Do not use this prompt for real-time user-facing chat without a caching layer; the generation latency for multiple variants can exceed acceptable response times. Instead, run this as an async pre-retrieval step or use a faster model for the initial variant generation and a stronger model for validation.

Integration points downstream of this prompt should expect a list of query objects and execute them in parallel against the specified backends. Each backend client should receive only the query_string and any backend-specific parameters extracted from the variant metadata. After retrieval, a fusion step must deduplicate and merge results across backends. The variant_id field in each generated query object should be preserved through the retrieval and fusion stages so you can trace which variant produced which results. This traceability is critical for debugging retrieval failures: if a fused result set is poor, you can identify whether the problem was in query generation, backend retrieval, or result merging. Wire the variant_id into your observability traces alongside the query_id and backend label.

When to avoid this prompt: if your retrieval pipeline uses only a single backend type, use a dedicated query rewriting prompt optimized for that backend instead. If the user query is a simple keyword lookup with no semantic ambiguity, skip generation entirely and pass the raw query directly to the sparse index. If latency is the primary constraint and you cannot run parallel backend queries, use a query routing prompt to select the single best backend rather than generating variants for all of them. Finally, if your retrieval system lacks the operational maturity to monitor variant quality, backend latency, and fusion effectiveness, start with a simpler single-query pipeline and add parallel generation only when you have the observability to tune it.