BM25F

Unlike standard BM25, which treats a document as a flat bag of words, BM25F calculates term frequency and document length independently for each field. A term appearing in a short, high-value field like a title contributes more to the final score than the same term in a lengthy body field. This prevents long body text from diluting the signal from concise, important fields.

BM25F introduces a field weight (boost) parameter for each document field. This allows search engineers to explicitly control the relative importance of different structural elements. For example:

• Title field: boost of 5.0
• Abstract field: boost of 2.0
• Body field: boost of 1.0 These weights are multiplied against the normalized per-field score before summation, providing granular control over ranking behavior.

The algorithm maintains a critical distinction between global term statistics and per-field statistics. Inverse document frequency (IDF) is typically computed globally across the entire document collection to measure a term's overall rarity. However, term frequency and length normalization are computed per-field. A term's frequency in a title is normalized by the average title length, not the average document length, ensuring the saturation curve is appropriate for each field's typical size.

The final BM25F score for a document is a weighted linear combination of the BM25 scores calculated for each individual field. The formula is: Score(d,q) = Σ (weight_field * BM25_field(term_frequency_field, length_field)) This additive model assumes independence between fields, which is computationally efficient and allows for straightforward implementation on top of existing BM25 infrastructure in engines like Apache Lucene.

A robust BM25F implementation must gracefully handle documents where a specific field is missing. If a query term does not appear in a field, that field contributes zero to the final score. The document is not penalized for the absence of a field; it simply does not receive the boost associated with that structural element. This is crucial for heterogeneous document collections where not all records have the same metadata.

A naive approach to multi-field search is to index each field separately and apply a static boost at query time. BM25F is superior because it integrates length normalization into the field weighting. A naive boost of 5.0 on a title field would over-score a single-term match in a very long title. BM25F's per-field normalization ensures that the boost is applied to a score that has already been correctly normalized for that field's length, producing a more principled relevance calculation.