Elasticsearch BM25

BM25 parameters can be configured per field in the index mapping, allowing different relevance tuning for different document structures. For example, a title field might use k1=1.8 for aggressive saturation, while a body field uses k1=1.2. This is configured via the similarity setting in field mappings, enabling precise control over how each field contributes to the final composite score.

The foundational data structure that makes BM25 scoring possible. An inverted index maps every unique term to a postings list containing document IDs, term frequencies, and positional data. During query execution, Elasticsearch reads these postings lists to compute BM25 scores without scanning documents. The index also stores doc values for aggregations and norms for length normalization, both critical inputs to the BM25 formula.

An extension of BM25 that handles structured documents with multiple fields. Instead of treating a document as a flat bag of words, BM25F computes separate term statistics for each field (title, body, tags) and combines them with field-specific weights. Elasticsearch implements this implicitly through field boosts and the tie_breaker parameter, allowing a match in the title field to contribute more to the final score than a match in the body.

The actual formula executed by Apache Lucene under the hood of Elasticsearch. It extends pure BM25 with additional factors:

• Coordination factor: Rewards documents matching more query terms
• Field-length norm: Encoded as a single byte per field per document
• Query normalization: Makes scores comparable across queries The full formula is: score(q,d) = coord(q,d) · queryNorm(q) · Σ(tf(t in d) · idf(t)² · t.getBoost() · norm(t,d))

The two tunable parameters that control BM25 behavior in Elasticsearch. k1 (default 1.2) governs term frequency saturation—higher values give more weight to repeated terms. b (default 0.75) controls document length normalization—0 disables it entirely, 1 applies full normalization. These can be configured per-field in the index settings:

• "similarity": {"type": "BM25", "k1": 1.2, "b": 0.75}
• Lower b values favor longer documents
• Higher k1 values amplify term frequency impact

The text processing chain that transforms raw text into the tokens BM25 scores. An analyzer consists of:

• Tokenizer: Splits text into tokens (standard, whitespace, n-gram)
• Token filters: Lowercase, stop word removal, stemming, synonym expansion Analysis happens identically at index time and query time to ensure lexical matching. A mismatch in analysis (e.g., stemming at index but not query) causes BM25 to miss relevant documents because the inverted index contains terms the query cannot match.

The modern pattern of combining BM25's sparse lexical retrieval with dense vector retrieval (kNN). Elasticsearch executes both queries in parallel and merges results using Reciprocal Rank Fusion (RRF). BM25 excels at exact term matching and rare keywords, while dense vectors capture semantic similarity. The fusion formula: score = Σ(1 / (k + rank_i)) where k=60 by default. This addresses BM25's vocabulary mismatch problem without sacrificing its precision on precise queries.