Query Expansion

Pseudo-Relevance Feedback (PRF), also known as blind relevance feedback, is an automatic technique that assumes the top k documents retrieved by an initial query are relevant. It extracts the most statistically significant terms from these documents to expand the original query. The process is iterative and does not require explicit user input.

• Core Mechanism: Uses algorithms like Rocchio to compute a new query vector as a weighted combination of the original query vector and the centroid vectors of the top-ranked documents.
• Key Benefit: Improves recall by pulling in documents that share thematic content with the initially retrieved set but may not contain the original query terms.
• Primary Risk: Query drift, where the expanded query moves away from the user's original intent if the initial results are poor.

This technique builds expansion term lists by analyzing statistical co-occurrence patterns within a large corpus, such as a document collection or search logs. It identifies terms that frequently appear in similar contexts.

• Common Methods: Pointwise Mutual Information (PMI) and Latent Semantic Indexing (LSI) are used to find terms with high semantic association.
• Application: If 'machine learning' frequently co-occurs with 'neural network' and 'supervised learning' in documents, those terms become candidate expansions.
• Advantage: Creates a domain-specific expansion resource tailored to the vocabulary of the target corpus.
• Limitation: Requires a large, representative corpus and computational overhead for analysis.

Knowledge-based expansion leverages structured external knowledge sources, such as ontologies, knowledge graphs (e.g., Wikidata, DBpedia), or domain-specific taxonomies, to find related concepts.

• Process: Identifies entities in the query (e.g., 'Python') and retrieves related entities via semantic relationships like is-a, part-of, or used-for (e.g., 'programming language', 'NumPy', 'Django').
• Example: The query 'Tesla earnings' could be expanded with 'Elon Musk', 'TSLA stock', and 'Q4 financial report' based on knowledge graph links.
• Benefit: Adds conceptually grounded terms, improving semantic recall and handling ambiguity.
• Challenge: Requires high-quality, up-to-date knowledge bases and robust entity linking.

This modern approach uses a Large Language Model (LLM) to generate potential query expansions in a zero-shot or few-shot manner. The LLM is prompted to rewrite or broaden the query based on its parametric knowledge.

• Methodology: A prompt like 'Generate 5 search query variations for: [original query]' instructs the model to produce synonyms, related concepts, or broader/narrower terms.
• Strength: Highly flexible and can generate nuanced, context-aware expansions without pre-built resources. Effective for zero-shot scenarios on novel topics.
• Consideration: Can introduce hallucinated or irrelevant terms. Performance depends on prompt engineering and model capability. Adds inference latency.

A foundational technique that expands a query using lexical resources and linguistic rules.

• Synonym Dictionaries: Uses resources like WordNet to add synonyms (e.g., 'car' -> 'automobile', 'vehicle').
• Morphological Variants: Applies stemming (e.g., 'running' -> 'run') and lemmatization to match different word forms.
• Compound Splitting: Breaks down compound terms (e.g., 'laptopbag' -> 'laptop bag').
• Use Case: Crucial for handling vocabulary mismatch, especially in lexical (keyword) search systems like those using BM25.
• Limitation: Purely lexical; cannot capture semantic or contextual relatedness beyond synonyms.

This technique analyzes historical query logs and click-through data to discover expansion terms based on real user interactions.

• Method: Identifies patterns such as:
  • Query Reformulations: How users sequentially modify their searches (e.g., 'AI' -> 'artificial intelligence companies').
  • Session Analysis: Terms used within the same search session.
  • Click Associations: Documents frequently clicked for a given query, suggesting term relevance.
• Advantage: Provides user-driven, real-world expansion signals that reflect actual information needs.
• Application: Powers query suggestions and auto-completion systems. Requires significant, privacy-compliant log data.

Query expansion operates by generating synonyms, related concepts, or contextual phrases and appending them to the original query. This creates a richer representation that can match documents using varied terminology. For example, the query "LLM optimization" might be expanded to "large language model optimization techniques quantization pruning distillation." The goal is to increase recall—the fraction of all relevant documents retrieved—by casting a wider semantic net, though it can sometimes reduce precision if overly broad terms are added.

Pseudo-Relevance Feedback (PRF) is a classic, unsupervised expansion method. It performs an initial retrieval, assumes the top-k results are relevant, and extracts frequent or salient terms from those documents to augment the original query. Common algorithms include Rocchio's algorithm for vector space models. In modern RAG, this can be implemented by:

• Retrieving an initial set of chunks.
• Using an LLM or statistical method (like TF-IDF) to identify key expansion terms from those chunks.
• Forming a new, weighted composite query for a second, more comprehensive retrieval pass.

Large language models are powerful tools for generating high-quality expansions. The original query is provided to an LLM with an instruction to generate multiple perspectives, synonymous phrases, or hypothetical documents. Common prompting strategies include:

• Multi-Query Generation: "Generate 3 different versions of this user question."
• Step-Back Prompting: "Generate a more general, conceptual question based on this specific query."
• HyDE (Hypothetical Document Embeddings): Instruct the LLM to generate a hypothetical ideal answer, then use that document's embedding for retrieval. This shifts retrieval from the query space to the answer space, often improving precision.

Effective expansion often combines lexical (keyword-based) and semantic (meaning-based) approaches. A hybrid retrieval system might:

• Use a dense retriever (semantic) with the original query.
• Use a sparse retriever like BM25 (lexical) on an LLM-expanded version of the query.
• Fuse the results from both retrievers. This mitigates the vocabulary mismatch problem where a document uses different words than the query but describes the same concept, while maintaining robustness for keyword-specific matches.

To prevent irrelevant drift, expansion can be constrained using a knowledge graph or enterprise ontology. The system identifies entities in the query, traverses the graph to find related entities and predicates (relationships), and uses those as expansion terms. For example, for a query "side effects of MedicationX," the knowledge graph links MedicationX to its active compound, known adverse reactions, and brand names. This provides a structured, domain-aware expansion that maintains factual grounding and avoids introducing noise from unrelated concepts.

The primary trade-off is between improved recall and the risk of decreased precision or conceptual drift. Key evaluation metrics include:

• Recall@k: Measures if more relevant documents are found in the top k results.
• Mean Reciprocal Rank (MRR): Checks if the first relevant result appears higher.
• End-to-End Answer Accuracy: The ultimate test for RAG—does expansion lead to more correct final answers? A/B testing is critical. Expansion also adds latency (for LLM calls or a second retrieval pass) and computational cost, which must be justified by significant gains in retrieval quality for the specific use case.

The process of altering a user's original query to better align with the underlying information need. Unlike expansion, which adds terms, reformulation often involves rewriting, correcting spelling, disambiguating terms, or paraphrasing the query.

• Example: Rewriting "best phone 2024" to "top-rated smartphones released in 2024".
• Purpose: Improves precision by refining the query's semantic focus, often using large language models for contextual rewriting.

An automatic query expansion technique where the system assumes the top-ranked documents from an initial search are relevant. It then extracts salient terms from these documents to augment the original query.

• Classic Algorithm: The Rocchio algorithm is commonly used to weight and combine terms from the original query and the pseudo-relevant documents.
• Key Assumption: The initial retrieval contains some relevant documents. If this fails, PRF can introduce noise.

The task of categorizing a user's search query into a predefined intent type to guide the retrieval and ranking strategy. Common intents include:

• Informational: Seeking knowledge (e.g., "What is quantum computing?")
• Navigational: Looking for a specific site/page (e.g., "LinkedIn login")
• Transactional: Aiming to complete a purchase or action (e.g., "buy wireless headphones")

Understanding intent allows a system to decide if expansion (broadening) or reformulation (sharpening) is more appropriate.

The task of converting a natural language query into a formal, machine-executable meaning representation. This is a deeper form of understanding than keyword-based expansion.

• Outputs: Can include logical forms, database queries (e.g., SQL, SPARQL), or API calls.
• Use Case: Enables Query-to-SQL systems where "Show me sales in Q4" is parsed into SELECT * FROM sales WHERE quarter = 'Q4'.
• Relation to Expansion: A parsed query can be used to generate precise expansion terms from a knowledge graph.

A neural search paradigm where queries and documents are encoded into dense vector embeddings. Relevance is computed via similarity in a high-dimensional space. Query expansion in this context often happens implicitly within the embedding model.

• Contrast with Lexical Search: Dense vectors capture semantic meaning, allowing matches beyond keyword overlap.
• Query Embedding: The process of transforming a query into its dense vector representation using models like BERT or Sentence Transformers.
• Hybrid Approach: Often combined with traditional keyword-based expansion for robust hybrid retrieval.

The process of disambiguating textual mentions of named entities in a query and linking them to unique entries in a knowledge base (e.g., Wikidata, a domain-specific ontology).

• Example: For the query "Apple stock price," linking "Apple" to the entity Apple_Inc. and not the fruit.
• Role in Expansion: Once linked, properties and related entities from the knowledge base (e.g., CEO, competitors, products) provide a rich source of high-precision expansion terms.