BM25 Retrieval

Unlike simple TF-IDF, BM25 applies a non-linear saturation function to term frequency. This prevents a document containing a term 100 times from being scored 100 times higher than one containing it 10 times.

• The formula uses tf / (k1 * ((1 - b) + b * (docLength / avgDocLength)) + tf)
• k1 (typically 1.2–2.0) controls the saturation curve steepness
• This models the diminishing returns of repeated term usage in clinical notes

BM25 compensates for the natural tendency of longer documents to contain more term occurrences. The b parameter (0 to 1) controls the degree of length normalization.

• b=1 fully normalizes by length, treating all documents equally
• b=0 applies no normalization, favoring longer documents
• In clinical entity linking, b=0.75 is common to slightly favor comprehensive records without overwhelming short, focused notes

BM25 incorporates an IDF-like weight that penalizes common terms and rewards rare, discriminative ones. The formula uses log((N - n + 0.5) / (n + 0.5)) where N is total documents and n is documents containing the term.

• Rare disease names receive high IDF boosts
• Common words like 'patient' or 'normal' are naturally suppressed
• This is critical for distinguishing between common and rare clinical entities in large corpora

BM25 treats documents as unordered collections of terms, ignoring word order and syntax. This lexical simplicity makes it:

• Extremely fast to compute using inverted indices
• Highly interpretable and debuggable
• Robust to typos and clinical shorthand variations
• An ideal candidate generation stage before neural reranking

For clinical entity linking, BM25 retrieves a broad set of candidate UMLS concepts that a cross-encoder can then precisely rank.

BM25 is derived from the Binary Independence Model, a probabilistic framework that ranks documents by the odds of relevance given term occurrences. The formula estimates:

Score(D, Q) = Σ IDF(qi) * (tf(qi, D) * (k1 + 1)) / (tf(qi, D) + k1 * (1 - b + b * |D|/avgdl))

This probabilistic grounding provides theoretical justification for its strong empirical performance across decades of IR research.

The statistical foundation upon which BM25 is built. Term Frequency-Inverse Document Frequency assigns a weight to each term in a document proportional to how often it appears, offset by how common it is across the entire corpus. In a clinical context, a rare term like 'rhabdomyosarcoma' receives a much higher weight than a ubiquitous term like 'patient'. BM25 refines this by introducing saturation curves to prevent high-frequency terms from dominating and by normalizing for document length, making it a more robust evolution of the basic TF-IDF model.

The initial, high-recall stage of an entity linking pipeline where BM25 is most commonly deployed. The goal is to retrieve a small, manageable set of plausible knowledge base entries for an ambiguous mention like 'cold'. BM25 acts as a fast, non-neural filter over a search index of entity descriptions and synonyms, efficiently narrowing down millions of concepts to a top-k list—perhaps 50-100 candidates—that a more computationally expensive neural cross-encoder can then precisely rank.

A preprocessing technique that significantly boosts BM25's recall. Before indexing, programmatic rules generate morphological and orthographic variations of clinical terms to populate the search index. For example, the term 'myocardial infarction' might be expanded to include 'infarction of myocardium', 'MI', and 'heart attack'. This ensures that BM25's exact-match heuristics can bridge the gap between the shorthand found in clinical notes and the canonical forms stored in ontologies like SNOMED CT.

The core data structure that makes BM25 retrieval computationally feasible at scale. An inverted index maps each unique term to a sorted list of document IDs where it appears, along with positional data. When a query like 'acute chest pain' is processed, the engine fetches the posting lists for 'acute', 'chest', and 'pain' and performs a fast intersection. This allows BM25 to score only documents containing at least one query term, avoiding a linear scan of the entire knowledge base.

The mathematical encoding of text used by BM25, standing in contrast to the dense vectors used in neural semantic search. A sparse vector has a dimensionality equal to the vocabulary size, where most values are zero. Only the terms present in the document have non-zero weights. This representation is highly interpretable—you can see exactly which tokens contributed to a match—and is extremely efficient for exact keyword matching, making it an ideal, transparent baseline before introducing the opacity of dense embeddings.