BM25 (Best Matching 25)

An inverted index is the foundational data structure for sparse, keyword-based retrieval like BM25. It maps each unique term (or token) to a list of documents (and often positions within those documents) where the term appears.

• Core Mechanism: Enables fast full-text search by allowing the system to look up a term and immediately retrieve all documents containing it.
• Contrast with BM25: BM25 uses the statistics derived from an inverted index (term frequencies, document frequencies) to compute its probabilistic relevance score. The index provides the raw data; BM25 provides the ranking algorithm.

Hybrid search is an information retrieval strategy that combines the results of sparse (e.g., keyword-based BM25) and dense (e.g., vector similarity) retrieval methods.

• Implementation: Typically uses a weighted score fusion (e.g., reciprocal rank fusion or a learned linear combination) to merge ranked lists from both retrievers.
• Rationale: Leverages the precise lexical matching of BM25 for fact-seeking queries while utilizing the semantic understanding of vector search for conceptual or paraphrased queries. This combination often yields higher recall and precision than either method alone.

TF-IDF is a statistical measure used to evaluate the importance of a word to a document in a collection. It is the direct conceptual predecessor to BM25.

• TF (Term Frequency): Measures how often a term appears in a document.
• IDF (Inverse Document Frequency): Measures how rare or common a term is across the entire corpus.
• Key Difference from BM25: TF-IDF produces a static weight for a term-document pair. BM25 improves upon this by introducing non-linear term frequency saturation and document length normalization, making it more robust for ranking in search engines.

Dense Passage Retrieval is a neural retrieval architecture that represents queries and documents as dense vector embeddings in a shared semantic space.

• Core Mechanism: Uses separate BERT-based encoders to map questions and passages into high-dimensional vectors. Relevance is scored via vector similarity (e.g., dot product).
• Contrast with BM25: DPR is a dense retriever that captures semantic meaning, allowing it to match queries with lexically different but semantically similar passages. BM25 is a sparse retriever that relies on exact keyword overlap. DPR requires significant training data, while BM25 is unsupervised.

Query expansion is a technique to improve retrieval recall by augmenting a user's original search query with additional related terms or phrases.

• Methods: Can be rule-based (using a thesaurus), statistical (using pseudo-relevance feedback), or neural (using a language model to generate expansions).
• Synergy with BM25: Since BM25 relies on term overlap, a poorly formulated query may miss relevant documents. Query expansion introduces synonymous or related terms, increasing the chance of lexical matches and improving BM25's effectiveness, especially for short or ambiguous queries.

Maximal Marginal Relevance is a re-ranking algorithm designed to reduce redundancy and increase diversity in a set of retrieved documents.

• Core Function: Selects documents that are both relevant to the query and sufficiently novel compared to documents already selected.
• Use with BM25: After BM25 retrieves a top-k set of relevant documents, MMR can reorder them. This is crucial in Retrieval-Augmented Generation (RAG) to ensure the language model receives a set of passages that cover different aspects of the query, rather than multiple near-identical passages that all score highly on BM25.