Sparse Retrieval

Dense retrieval is a neural search paradigm where queries and documents are encoded into dense, low-dimensional vector embeddings (e.g., using transformer models like BERT), and relevance is determined by the similarity between these embeddings. Unlike sparse retrieval's exact term matching, dense retrieval captures semantic meaning, allowing it to find conceptually related content even without keyword overlap.

• Key Mechanism: Uses a bi-encoder architecture to independently map text into a shared vector space.
• Primary Use: Powering semantic search in modern RAG systems.
• Trade-off: Higher computational cost for embedding generation but enables efficient approximate nearest neighbor search.

Hybrid search is a retrieval strategy that combines the results of multiple search methods—typically sparse retrieval (e.g., BM25) and dense retrieval (vector search)—to improve overall recall and relevance. It leverages the lexical precision of sparse methods with the semantic understanding of dense methods.

• Combination Method: Results are merged using algorithms like Reciprocal Rank Fusion (RRF).
• Benefit: Mitigates the weaknesses of each approach (e.g., vocabulary mismatch for sparse, sensitivity to phrasing for dense).
• Implementation: Common in production RAG systems using vector databases that support multi-index querying.

BM25 (Best Matching 25) is a probabilistic ranking function and the modern standard for sparse retrieval. It estimates the relevance of documents to a query based on term frequency (TF), inverse document frequency (IDF), and document length normalization.

• Core Formula: score(D, Q) = Σ IDF(q_i) * (f(q_i, D) * (k1 + 1)) / (f(q_i, D) + k1 * (1 - b + b * |D| / avgdl))
• Advantage over TF-IDF: Includes saturation controls (k1) and length normalization (b) to prevent very long documents from dominating results.
• Usage: The default lexical retrieval algorithm in search engines like Elasticsearch and OpenSearch.

Reranking is a two-stage retrieval process where a large set of candidate documents is first retrieved (via sparse or dense search) and then re-scored by a more powerful, computationally expensive model to improve final ranking precision. A cross-encoder is the typical architecture used for this stage.

• Cross-Encoder Mechanism: Jointly processes a query and a document pair through a transformer model (e.g., BERT) to produce a direct relevance score, allowing deep interaction between tokens.
• Trade-off: Highly accurate but too slow for initial retrieval over large corpora.
• Application: Used as a final precision layer in high-stakes RAG pipelines to select the best context from a top-K candidate set.

Query expansion is a technique that augments a user's original search query with additional related terms, synonyms, or phrases to improve retrieval recall. It directly addresses the vocabulary gap problem inherent in sparse retrieval, where relevant documents may use different terminology than the query.

• Methods: Can be rule-based (using a thesaurus), model-based (using a language model to generate expansions), or pseudo-relevance feedback (adding terms from top initial results).
• Impact on Sparse Retrieval: Significantly improves the effectiveness of BM25 by making the query representation more comprehensive.
• Risk: Can introduce noise and reduce precision if expansions are not well-targeted.

Lexical search is the broader category of information retrieval that relies on the exact matching of words or tokens between queries and documents. Sparse retrieval methods like Boolean search, TF-IDF, and BM25 are all forms of lexical search.

• Core Principle: Operates on the bag-of-words model, ignoring word order and syntax but considering frequency and distribution.
• Strengths: Highly interpretable, efficient to compute, and excellent for finding documents with precise keyword matches.
• Evolution: While foundational, modern systems often combine it with semantic (vector) search in a hybrid approach to overcome its limitations with synonymy and conceptual search.