BM25

Unlike linear TF-IDF, BM25 applies a non-linear saturation curve to term frequency. The parameter k1 controls how quickly the impact of additional term occurrences diminishes. A document containing a term 5 times is not 5 times as relevant as one containing it once. This prevents keyword stuffing from dominating scores and models the diminishing returns of repeated term usage in natural language.

BM25 normalizes term frequency by document length relative to the average corpus length. The b parameter (typically 0.75) controls the strength of this normalization:

• b=1: Full normalization by length
• b=0: No length normalization This compensates for the higher probability of term occurrence in longer documents, ensuring short, dense documents can compete fairly with lengthy ones.

BM25 inherits the IDF component from TF-IDF, weighting rare terms more heavily than common ones. The standard IDF formula is:

codeCopy
IDF(q_i) = ln((N - n(q_i) + 0.5) / (n(q_i) + 0.5) + 1)

Where N is the total document count and n(q_i) is the number of documents containing the term. This gives high discriminative power to terms that appear in few documents.

BM25 is grounded in the Probability Ranking Principle, which states that documents should be ranked by their probability of relevance given the query. It is derived from the 2-Poisson model of term distribution, where relevant and non-relevant documents are modeled as two distinct Poisson distributions. This theoretical foundation distinguishes it from purely heuristic vector space models.

BM25's behavior is governed by two free parameters:

• k1 (default 1.2): Controls term frequency saturation. Higher values give more weight to repeated terms.
• b (default 0.75): Controls length normalization. Higher values penalize long documents more. These parameters can be optimized via grid search against relevance judgments on a development set, allowing domain-specific tuning without retraining.

BM25 operates on sparse bag-of-words vectors, where each dimension corresponds to a vocabulary term. Unlike dense embeddings, these representations are:

• Interpretable: Exact term matches are visible
• Efficient: Inverted index lookups are extremely fast
• Zero-shot: No training data required This makes BM25 the standard baseline against which all neural retrieval models are benchmarked.

Encodes queries and documents into dense vector embeddings to perform semantic matching via approximate nearest neighbor (ANN) search. Unlike BM25's exact token matching, dense retrieval captures conceptual similarity and paraphrasing. It excels when vocabulary mismatch exists between query and document terms.

Fuses relevance signals from dense vector search and sparse lexical retrieval (BM25) to combine semantic understanding with exact keyword matching precision. This approach mitigates the weaknesses of each method: dense retrieval's struggle with rare entities and BM25's vocabulary mismatch problem.

A cascade architecture where a fast, lightweight retriever (often BM25 or a bi-encoder) selects candidate documents from a large corpus, and a computationally intensive cross-encoder re-ranker refines the ordering of the top-k candidates. BM25 is the most common first-stage retriever due to its speed and zero-shot capability.

An unsupervised algorithm that merges multiple ranked lists into a single consensus ranking by scoring documents based on the reciprocal of their rank positions across input lists. Formula: score(d) = Σ 1/(k + rank_i(d)). It requires no training data or score calibration, making it ideal for combining BM25 and dense retrieval results.

A supervised machine learning paradigm that trains models to optimize the ordering of documents for a given query. Unlike BM25's unsupervised formula, LTR models learn from relevance judgments using pointwise, pairwise, or listwise loss functions. LambdaMART is a classic gradient boosted tree implementation.

A listwise ranking evaluation metric that measures ranking quality by discounting relevance gains logarithmically by position and normalizing against an ideal ranking. It is the standard metric for evaluating BM25 baselines against learned rankers, emphasizing top-weighted relevance.