Query Routing

A lightweight classifier model is trained to categorize incoming queries into predefined buckets (e.g., factual, conversational, code-generation). Each bucket maps to a specific retrieval pipeline. This is the most common production pattern because it is deterministic and debuggable. The classifier can be as simple as a fine-tuned BERT model or a logistic regression on query embeddings. Key considerations include training data drift as user behavior evolves and the need for an 'unknown' fallback class that defaults to a general-purpose index.

Rather than a hard classification, the system computes the centroid similarity between the query embedding and the centroid of each candidate index. The index with the highest cosine similarity is selected. This approach is fully unsupervised and adapts smoothly to semantic shifts without retraining a classifier. However, it assumes that the centroid of an index accurately represents its contents. Outlier documents can skew the centroid, and the computational cost of comparing against all centroids grows linearly with the number of indices.

A rules engine parses the query for explicit trigger tokens or regular expressions to route to specialized indices. For example, a query containing 'SEC filing' or a ticker symbol like '$AAPL' routes directly to a financial document index. This method offers zero latency overhead and is trivially explainable. The trade-off is maintenance: the keyword dictionary requires manual curation and cannot handle paraphrased queries that lack the exact trigger token. Often used as a fast-path optimization before falling back to a semantic router.

A small, fast language model is prompted to output a structured routing decision based on the query. The prompt includes descriptions of available indices and their strengths. This enables nuanced, compositional routing—a query like 'summarize the legal risks in our Q3 financials' can be routed to both a legal index and a financial index for a hybrid retrieval. The primary cost is inference latency and token overhead. This pattern is gaining traction with the availability of low-latency inference APIs, but requires careful prompt engineering to prevent hallucinated routing targets.

The query is broadcast to all candidate indices in parallel, and results are merged using a fusion algorithm like Reciprocal Rank Fusion (RRF). This avoids the single-point-of-failure risk of misrouting a query to the wrong index. The trade-off is increased computational cost and latency, as every index must be queried. This pattern is ideal when the cost of a missed result is extremely high, such as in medical literature search or legal discovery. Caching and index pruning can mitigate the overhead.

Complex multi-hop queries are decomposed into sub-queries by a planner model. Each sub-query is independently routed to the most appropriate index. For example, 'What was the revenue impact of the competitor product launched after our patent expired?' decomposes into a patent index query, a competitor product index query, and a financial index query. The retrieved contexts are then synthesized. This is the most sophisticated routing pattern, foundational to agentic RAG systems, but introduces significant orchestration complexity and latency.