Vector vs Keyword vs Hybrid Query Decision Prompt

This prompt is for search infrastructure teams and RAG orchestration engineers who manage multiple retrieval backends and need a reliable, automated way to decide which retrieval strategy to apply to a given user query. The prompt analyzes query characteristics such as specificity, entity density, natural language complexity, and temporal requirements to classify whether dense vector search, sparse keyword search, or a hybrid combination is optimal. Use this prompt inside a retrieval dispatch layer before query rewriting or index selection occurs. It is not a replacement for A/B testing your retrieval pipelines, nor does it handle the actual execution of the search. It is a decision component that reduces latency and cost by avoiding unnecessary multi-strategy retrieval for queries that a single backend can handle well.

The ideal user is an engineer who already has at least two retrieval backends running in production—typically a dense vector index and a sparse keyword index—and needs a lightweight classifier to route queries. The prompt requires the user's query text as input and optionally accepts context such as conversation history, user role, or latency budget constraints. It works best when the distinction between backends is clear: vector search for semantic similarity and conceptual queries, keyword search for exact term matching and entity lookup, and hybrid for queries that blend both needs. Do not use this prompt if you only have one retrieval backend, if your backend automatically handles fusion, or if you need the model to generate the actual search queries rather than just the strategy decision.

Before deploying this prompt, ensure you have defined clear evaluation criteria for what constitutes a correct routing decision in your system. Common failure modes include over-routing to hybrid when a simpler strategy would suffice, misclassifying entity-heavy queries as purely semantic, and failing to account for temporal signals that make keyword retrieval more appropriate. Wire the prompt's output into a decision harness that logs every classification alongside the eventual retrieval latency and relevance metrics. This allows you to measure whether the prompt's decisions actually improve system performance. If your retrieval pipelines are still under active development, start by logging decisions in shadow mode before letting the prompt control live traffic.

This template provides a production-ready prompt for a query routing decision. It is designed to be placed inside a retrieval dispatch layer where a user's natural language query must be analyzed before hitting any search index. The prompt forces the model to reason about query structure, entity density, and precision/recall trade-offs before committing to a single strategy. Copy the template below and replace every square-bracket placeholder with your system's specific backends, constraints, and output schema.

textCopy
SYSTEM:
You are a retrieval strategy router. Your job is to analyze a user's search query and select the optimal retrieval backend: dense vector search, sparse keyword search, or a hybrid combination.

You have access to the following backends:
- [VECTOR_INDEX_NAME]: Dense embeddings optimized for semantic similarity. Best for conceptual queries, paraphrases, and natural language questions.
- [KEYWORD_INDEX_NAME]: Sparse inverted index using [BM25/TF-IDF]. Best for exact term matching, entity names, codes, and rare tokens.
- [HYBRID_INDEX_NAME]: Combines vector and keyword results with [RECIPROCAL_RANK_FUSION/LEARNED_WEIGHTING]. Best for queries with both semantic and lexical components.

## Decision Criteria
1. **Exact match priority**: If the query contains proper nouns, product codes, error codes, or rare technical terms, prefer keyword or hybrid.
2. **Conceptual queries**: If the query is a natural language question, description, or paraphrase without specific entity anchors, prefer vector.
3. **Mixed queries**: If the query has both conceptual language and specific entities, prefer hybrid.
4. **Short queries**: Queries under [N] words with no clear entities default to hybrid to balance precision and recall.
5. **Latency budget**: If the user context indicates a strict latency requirement under [LATENCY_BUDGET_MS]ms, prefer vector or keyword alone unless hybrid is essential for relevance.

## Output Format
Return a JSON object with the following schema:
{
  "selected_strategy": "vector" | "keyword" | "hybrid",
  "confidence": 0.0-1.0,
  "reasoning": "Brief explanation of the decision.",
  "fallback_strategy": "vector" | "keyword" | "hybrid",
  "query_type": "exact_lookup" | "conceptual" | "mixed" | "ambiguous"
}

## Constraints
- Do not select hybrid if the query is clearly an exact term match with no semantic component.
- Do not select vector alone if the query contains a product SKU, error code, or legal citation.
- If confidence is below [CONFIDENCE_THRESHOLD], set fallback_strategy to hybrid.
- Never invent or assume entities not present in the query.

USER QUERY:
[USER_QUERY]

USER CONTEXT (optional):
[USER_CONTEXT]

Adaptation guidance: Replace [VECTOR_INDEX_NAME], [KEYWORD_INDEX_NAME], and [HYBRID_INDEX_NAME] with your actual index identifiers. Set [N] to your short-query word-count threshold (commonly 3). Set [LATENCY_BUDGET_MS] to your p95 latency target. Set [CONFIDENCE_THRESHOLD] to your minimum confidence before forcing a hybrid fallback (0.7 is a reasonable starting point). If your system does not support hybrid retrieval, remove that option and adjust the decision criteria to a binary vector vs. keyword choice. The [USER_CONTEXT] field is optional but should be populated with session history, user role, or latency requirements when available to improve routing accuracy.

What to do next: After copying this template, wire it into your retrieval dispatch layer as described in the Implementation Harness section. Before deploying, run the evaluation checks in the Testing and Eval Criteria section to calibrate your confidence threshold and verify that entity-dense queries are not incorrectly routed to vector-only retrieval. If your system uses multiple specialized indexes beyond these three backends, consider extending the output schema with an index_target field rather than overloading the strategy selection.

This prompt is not a standalone service. It is a decision component that sits inside a retrieval dispatch layer. Its job is to inspect the user query and return a routing decision—vector, keyword, or hybrid—along with a confidence score and reasoning. The dispatch layer reads that decision and calls the appropriate retrieval backend. Do not treat this prompt as the retrieval step itself; it only decides which retrieval strategy to invoke next. The output must be consumed by application code that maps the decision to actual index calls, latency budgets, and fallback chains.

Wire the prompt into a pre-retrieval hook that fires after query normalization but before any index query. The application should: (1) receive the user query, (2) strip PII and apply any access-control filters, (3) call this decision prompt with the cleaned query and available backend metadata, (4) parse the JSON output and validate the strategy field against an allowlist of ["vector", "keyword", "hybrid"], (5) route to the corresponding retrieval function. If the confidence score falls below a configured threshold (start at 0.7 and tune from production traces), fall back to hybrid retrieval as the safe default. Log every decision—query fingerprint, chosen strategy, confidence, latency, and whether the user accepted or reformulated the results—so you can measure strategy accuracy over time. For high-throughput systems, cache decisions for identical normalized queries with a short TTL (e.g., 5 minutes) to avoid redundant LLM calls.

Add a circuit breaker: if the decision prompt fails, times out, or returns unparseable output after one retry, default to hybrid retrieval and log the failure for investigation. Do not block the user on a decision prompt error. For latency-sensitive applications, consider running this prompt in parallel with a fast keyword fallback so results are ready regardless of the decision path. The next step after implementing this harness is to build an eval set of 200+ queries with ground-truth strategy labels, run the prompt against them, and measure precision and recall per strategy class. Pay special attention to queries that sit on the boundary between vector and hybrid—these are where misclassification costs the most relevance.