BM25

Sparse retrieval is the traditional information retrieval paradigm where documents and queries are represented as high-dimensional, sparse vectors based on term occurrence. BM25 is its state-of-the-art probabilistic ranking function.

• Core Mechanism: Uses bag-of-words models like TF-IDF or BM25, where dimensions correspond to vocabulary terms.
• Characteristics: Highly interpretable, efficient for exact keyword matching, but suffers from the vocabulary gap where different words with the same meaning are not matched.

Dense retrieval is a neural search paradigm that encodes queries and documents into dense, low-dimensional vector embeddings (e.g., 768 dimensions). Relevance is computed via vector similarity, capturing semantic meaning.

• Contrast with BM25: Moves beyond lexical matching to semantic similarity. The query "automobile" can retrieve a document about "cars."
• Architecture: Typically uses a bi-encoder model (like SBERT) to independently encode text into embeddings.
• Trade-off: Requires significant training data and compute but excels at understanding intent and paraphrasing.

Hybrid search strategically combines the strengths of sparse retrieval (BM25) and dense retrieval (vector search) to improve overall recall and precision. It mitigates the weaknesses of each approach when used alone.

• Implementation: Runs BM25 and a vector search in parallel, then fuses the result rankings using algorithms like Reciprocal Rank Fusion (RRF).
• Benefit: BM25 excels at exact term matching and IDF-based importance, while dense retrieval captures semantic intent. Together, they cover both keyword and conceptual relevance.

Reciprocal Rank Fusion is a simple, highly effective method for combining multiple ranked lists of search results (e.g., from BM25 and a vector search) into a single, unified ranking.

• Formula: score = sum(1 / (k + rank)) for each list, where k is a constant (typically 60).
• Why it works: It gives high weight to documents that appear near the top of multiple lists, promoting consensus. It requires no tuning or score normalization.
• Use Case: The default fusion algorithm in hybrid search architectures to merge lexical and semantic results.

Reranking is a two-stage retrieval process where a broad set of candidates (e.g., top 100 from BM25 or hybrid search) is re-scored by a more powerful, computationally expensive model to produce the final precision-optimized ranking.

• Model: Typically uses a cross-encoder architecture (like a BERT model) that jointly processes the query and each candidate document to produce a direct relevance score.
• Advantage: Cross-encoders achieve very high accuracy due to deep attention between query and document tokens but are too slow for initial retrieval over large corpora.

TF-IDF is a foundational statistical measure used in information retrieval to evaluate the importance of a word to a document within a collection. It is the direct predecessor to BM25.

• Components: Term Frequency (TF) counts word occurrences in a doc. Inverse Document Frequency (IDF) penalizes words common across many docs.
• Limitation vs. BM25: TF-IDF lacks BM25's saturation mechanism for term frequency and its length normalization is less robust. BM25 is generally considered a more advanced, probabilistic evolution of the TF-IDF concept.