Entity-Dense Query Routing Prompt for Graph vs Vector Index

This prompt is designed for orchestration engineers who operate a dual-index retrieval architecture: a knowledge graph index for relationship traversal and a vector index for semantic similarity search. Its job is to inspect a user query and decide whether the query is entity-dense, requiring relationship traversal, or semantic, requiring vector similarity search. Use it as a dispatch layer before retrieval execution to reduce latency and improve precision by sending each query to the index best suited to answer it. The core assumption is that your graph index supports relationship-first queries (e.g., 'find all reports authored by people who worked on Project X') and that your vector index handles open-ended natural language questions (e.g., 'what are the best practices for cloud cost optimization?').

Do not use this prompt when you have a single index, when your graph is purely for metadata filtering on vector search results, or when every query should hit both indexes and merge results. It is also inappropriate when your entity catalog is undefined or incomplete—the router cannot make accurate decisions without a known set of entities and relationship types. This prompt assumes you have a defined entity catalog and that your graph index supports relationship-first queries. If your system lacks these prerequisites, you will get unreliable routing decisions that degrade retrieval quality rather than improve it. The router is a precision tool for a specific architectural pattern, not a general-purpose query classifier.

Before implementing this prompt, verify that your retrieval architecture genuinely benefits from index-level routing. If your vector index already handles entity-heavy queries adequately, or if your graph index is immature, the added complexity of a dispatch layer may not justify the marginal improvement. Start by logging a sample of production queries and manually labeling them as entity-dense or semantic. If fewer than 20% of queries are entity-dense, a simpler 'always vector' strategy with optional graph enrichment may be more maintainable. If you proceed, pair this router with an evaluation pipeline that measures routing accuracy against human-labeled ground truth, and monitor the downstream impact on answer quality, not just routing precision.

The following prompt template is designed to be placed directly into your orchestration layer. It instructs the model to act as a retrieval router, analyzing the incoming user query for entity density and relationship requirements. The core decision is binary: route to a knowledge graph for relationship traversal or to a vector index for semantic similarity. The prompt also requires the model to extract key entities, which serves as both a debugging artifact and a potential input for the downstream retriever.

textCopy
System: You are a retrieval routing specialist. Your task is to analyze a user query and decide whether it should be answered by traversing a knowledge graph or by searching a vector index. A knowledge graph is optimal for queries that are dense with specific entities and require understanding the relationships between them (e.g., 'Who is the CEO of the company that acquired Acme Corp?'). A vector index is optimal for broad, conceptual, or semantic queries where exact entity relationships are less critical (e.g., 'strategies for entering the European market').

User Query: [USER_QUERY]

Analyze the query and produce a JSON object with the following keys:
- `strategy`: A string, either "graph" or "vector".
- `rationale`: A brief, one-sentence explanation for the routing decision.
- `extracted_entities`: A list of the key named entities (people, companies, products, etc.) found in the query. Return an empty list if none are found.

[OUTPUT_SCHEMA]
{
  "strategy": "string",
  "rationale": "string",
  "extracted_entities": ["string"]
}

To adapt this template, replace the [USER_QUERY] placeholder with the raw input from your application. The [OUTPUT_SCHEMA] is provided inline as a strict JSON specification. For production use, you should hardcode this schema into your application's validation layer and remove the [OUTPUT_SCHEMA] token from the prompt itself. The next step after receiving a valid JSON response is to parse the strategy field and use it to dispatch the [USER_QUERY] to the correct retrieval backend. If the strategy is 'graph', you might also use the extracted_entities list as a starting point for a graph traversal query.

This prompt functions as a classification router that sits between the user's query and your retrieval backends. It should be called synchronously before any index query is executed. The model's output—a routing decision (graph_index, vector_index, or hybrid) and an entity list—must be parsed and used to select the downstream retrieval path. Do not treat this as a conversational prompt; it is a structured decision point in a pipeline. The prompt expects a single user query as [QUERY] and an optional [DOMAIN_CONTEXT] describing the available knowledge domains. The output schema should be enforced via structured output mode (JSON mode or function calling) to avoid parsing errors in production.

Validation and retry logic is critical because a misrouted query silently degrades answer quality. After parsing the model's JSON response, validate that the strategy field is one of the allowed enum values (graph_index, vector_index, hybrid). If the field is missing or invalid, retry the prompt once with a stricter system instruction that emphasizes the enum constraint. If the retry also fails, default to hybrid retrieval and log the failure for review. For the entities array, check that each entry has a non-empty name field and a type value from your supported entity types. Empty entity lists are valid for queries with no detectable entities, but a list of entities where half are malformed suggests a model output issue worth sampling and reviewing.

Model choice matters. This routing prompt benefits from models with strong instruction-following and JSON output discipline. GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro are all suitable. Avoid smaller or older models that may conflate the routing decision or hallucinate entity types not present in the query. If latency is a concern, consider hosting a fine-tuned classifier model on this specific task rather than using a general-purpose model with a long prompt. Log every routing decision with the query, the model's raw output, the validated strategy, the extracted entities, and the retrieval path taken. This log becomes your evaluation dataset for measuring routing accuracy over time. Sample routing decisions weekly and have a human reviewer check whether the chosen strategy would have retrieved better results than the alternatives. Use these samples to tune your routing thresholds and update few-shot examples in the prompt template.

Integration pattern: In your retrieval orchestrator, call this prompt first. Use the strategy output to branch: graph_index routes to your graph database with a Cypher or SPARQL query generated from the entities; vector_index routes to your dense vector store with the original query; hybrid runs both in parallel and merges results with reciprocal rank fusion. The entities array can be passed as metadata filters or used to construct the graph query. If the model returns hybrid but your latency budget only allows one path, use the confidence score from the output (if you add it to the schema) to break ties. Avoid over-routing to hybrid by monitoring the distribution of strategy selections; if more than 40% of queries route to hybrid, your classification thresholds may need adjustment or your indexes may need better scope definition.

Failure modes to instrument: The most common production failure is the model routing a relationship-heavy query to the vector index because it failed to recognize implicit entity relationships. Instrument a metric for 'graph-routable queries sent to vector' by sampling vector-routed queries and checking if they contain multiple linked entities. A second failure mode is entity extraction hallucination, where the model invents entity names or types not present in the query. Add a post-processing check that each extracted entity string appears verbatim in the original query or is a known canonical form from your entity resolution system. If you maintain an entity catalog, validate extracted entities against it and flag mismatches. Finally, watch for routing instability: if the same query routed through the prompt twice produces different strategies, your temperature may be too high or your prompt constraints too loose. Set temperature to 0 for this routing task in production.