Glossary

BM25 (Okapi BM25)

A bag-of-words retrieval function that ranks documents by estimating the probability of relevance, using non-linear term frequency saturation and document length normalization.

Get in touch Learn more

Stylish WeWork-like workspace with hot desks and document wall, professional searching through enterprise knowledge base on a mounted ultrawide display, warm industrial pendants overhead.

PROBABILISTIC RELEVANCE FRAMEWORK

What is BM25 (Okapi BM25)?

BM25 is a bag-of-words retrieval function that ranks documents by estimating the probability of their relevance to a given search query, using term frequency saturation, inverse document frequency, and document length normalization.

BM25 (Okapi BM25) is a probabilistic relevance ranking function that scores documents based on the weighted frequency of query terms appearing in each document, while controlling for document length. It improves upon classic TF-IDF by introducing a non-linear saturation function that prevents a single term from dominating the score through excessive repetition, and a tunable document length normalization parameter.

The algorithm operates within the Probabilistic Relevance Framework and relies on a sparse retrieval model, using an inverted index for efficient lexical matching. Its behavior is governed by two free parameters, k1 and b, which allow engineers to tune the impact of term frequency saturation and length normalization, respectively, making it the default relevance scorer in engines like Elasticsearch and Apache Lucene.

Core Mechanisms

Key Features of BM25

The BM25 algorithm's effectiveness stems from a precise interplay of statistical components that model term importance and document length.

Non-Linear Term Frequency Saturation

BM25 applies a saturation function to term frequency (TF). Unlike linear models where a word appearing 10 times is 10x as relevant, BM25 models the diminishing returns of repeated term occurrences. The k1 parameter controls the curve's steepness, preventing a single high-frequency term from dominating the relevance score and rewarding the first few occurrences far more than subsequent ones.

Probabilistic Inverse Document Frequency

The IDF component is derived from the Robertson-Spärck Jones weighting formula, grounded in a probabilistic relevance framework. It estimates a term's informativeness based on document frequency:

Rare terms across the collection receive a high IDF weight
Common terms like 'the' approach zero weight
The formula log((N - df + 0.5) / (df + 0.5)) provides a robust, theoretically justified scaling factor

Document Length Normalization

BM25 normalizes term frequency by document length to level the playing field. Longer documents naturally have higher raw term frequencies, but are not inherently more relevant. The b parameter (typically 0.75) controls the normalization strength:

b=1: Full normalization relative to average document length
b=0: No normalization applied
The ratio |D| / avgdl adjusts the saturation curve per document

Tunable k1 and b Parameters

BM25's two free parameters allow precise calibration to specific collections:

k1 (default 1.2): Controls TF saturation. Higher values give more weight to repeated terms; lower values flatten the curve faster
b (default 0.75): Controls length normalization. Adjust based on document type—verbose articles benefit from higher b, while short product titles may need lower b
These parameters are typically tuned via grid search against relevance judgments

Sparse Vector Representation

BM25 operates in the sparse retrieval paradigm, representing documents as high-dimensional vectors where only terms present in the document have non-zero weights. This enables:

Efficient storage and retrieval via inverted indices
Exact lexical matching with no semantic approximation
Fast query-time scoring using optimized postings list traversal
Transparent, explainable relevance scores per term

Elasticsearch and Lucene Implementation

Since Elasticsearch 5.0, BM25 has been the default similarity algorithm, replacing the classic TF-IDF model. The Lucene Practical Scoring Function extends BM25 with:

Coordination factor: Rewards documents matching more query terms
Field-length norms: Encoded per-field normalization values
Query-time boosting: Allows term-level importance adjustments
Shard-level IDF computation with distributed index statistics

RETRIEVAL PARADIGM COMPARISON

BM25 vs. TF-IDF vs. Dense Retrieval

A feature-level comparison of three core information retrieval paradigms: the classic TF-IDF weighting scheme, the probabilistic BM25 ranking function, and modern neural dense retrieval using embeddings.

Feature	BM25	TF-IDF	Dense Retrieval
Core Mechanism	Probabilistic relevance framework with term saturation and length normalization	Algebraic product of term frequency and inverse document frequency	Neural encoding of text into dense vectors for semantic similarity search
Term Frequency Handling	Non-linear saturation via k1 parameter; additional occurrences yield diminishing returns	Linear scaling; each occurrence adds equal weight with no upper bound	No explicit term frequency; meaning is compressed into a fixed-dimension vector
Document Length Normalization	Configurable via b parameter (0 to 1); pivots on average document length	Cosine normalization or none in basic form; longer documents often score higher	Implicit; fixed-size embeddings abstract away raw token counts
Vocabulary Mismatch Handling
Out-of-Vocabulary Terms	Exact match only; unknown terms ignored	Exact match only; unknown terms ignored	Handled via subword tokenization and contextual embeddings
Interpretability	High; per-term score contributions are transparent and auditable	High; simple multiplication of TF and IDF components	Low; vector similarity scores are opaque and non-decomposable
Index Structure	Sparse inverted index with postings lists	Sparse inverted index with postings lists	Dense vector index using approximate nearest neighbor (ANN) algorithms
Computational Cost at Query Time	Low; efficient postings list traversal	Low; efficient postings list traversal	Moderate to high; requires vector similarity computation across embeddings
Training Data Required	None; unsupervised statistical formula	None; unsupervised statistical formula	Large corpora of labeled or contrastive query-document pairs
Domain Adaptation	Manual tuning of k1 and b parameters	Manual stop word lists and stemming rules	Fine-tuning or domain-specific pre-training of the encoder model
Matching Granularity	Lexical token overlap	Lexical token overlap	Semantic and conceptual similarity
Synonym and Paraphrase Handling

Enabling Efficiency, Speed & Accuracy

Intelligent Analysis, Decision & Execution

We build AI systems for teams that need search across company data, workflow automation across tools, or AI features inside products and internal software.

Talk to Us

Search across company data

Give teams answers from docs, tickets, runbooks, and product data with sources and permissions.

Useful when people spend too long searching or get different answers from different systems.

Enterprise searchRAGPermissions

Automate internal workflows

Use AI to route work, draft outputs, trigger actions, and keep approvals and logs in place.

Useful when repetitive work moves across multiple tools and teams.

AI agentsWorkflow automationGovernance

Add AI to products and internal tools

Build assistants, guided actions, or decision support into the software your team or customers already use.

Useful when AI needs to be part of the product, not a separate tool.

AI integrationDecision supportModel routing

BM25 DEEP DIVE

Frequently Asked Questions

Clear, technical answers to the most common questions about the Okapi BM25 ranking function, its parameters, and its role in modern search architectures.

BM25 (Okapi BM25) is a probabilistic relevance ranking function that estimates the relevance of a document to a given search query. It works by calculating a score based on three main components: term frequency (TF), which is dampened by a non-linear saturation function to prevent high-frequency terms from dominating; inverse document frequency (IDF), which downweights common terms that appear in many documents; and document length normalization, which prevents longer documents from having an unfair scoring advantage. The formula is:

code
score(D, Q) = Σ IDF(qi) * (f(qi, D) * (k1 + 1)) / (f(qi, D) + k1 * (1 - b + b * |D| / avgdl))

Where f(qi, D) is the term's frequency in the document, |D| is the document length, avgdl is the average document length across the collection, and k1 and b are tunable parameters. The function models the intuition that a term's relevance contribution grows sub-linearly with frequency—seeing a word twice is more valuable than once, but seeing it twenty times is not ten times as valuable as seeing it twice.

FOUNDATIONAL CONCEPTS

Related Terms

Understanding BM25 requires familiarity with the core information retrieval concepts that it builds upon and the modern techniques it often complements.

TF-IDF

The foundational weighting scheme that BM25 evolved from. TF-IDF calculates a score by multiplying Term Frequency (how often a word appears in a document) by Inverse Document Frequency (how rare the word is across the collection). Unlike BM25, classic TF-IDF lacks a saturation function for term frequency and sophisticated document length normalization, making it more prone to bias toward longer documents that repeat terms verbatim.

EXPLORE

Sparse Retrieval

The retrieval paradigm in which BM25 operates. Documents and queries are represented as high-dimensional vectors where most dimensions are zero. Non-zero dimensions correspond to explicit term weights. This enables efficient matching via an inverted index without the computational cost of dense vector comparison.

Key Advantage: Exact lexical matching and fast lookup
Key Limitation: Suffers from vocabulary mismatch when relevant documents use different words than the query

Inverted Index

The data structure that makes BM25 scoring computationally feasible at scale. An inverted index maps each unique term to a postings list containing the document IDs where it appears, along with position and frequency metadata. During query evaluation, the engine retrieves these lists to compute BM25 scores without scanning every document.

Hybrid Search Fusion

A modern architecture where BM25's sparse lexical scores are combined with dense vector similarity from neural embedding models. Techniques like Reciprocal Rank Fusion (RRF) merge the two result sets to capture both exact keyword matches and semantic meaning.

BM25 excels at rare, precise terms like product codes
Dense retrieval handles paraphrasing and conceptual matches
Fusion eliminates the weaknesses of either approach alone

BM25F

An extension of BM25 for structured documents with multiple fields of varying importance. A match in a title field should contribute more to relevance than a match in the body. BM25F computes per-field statistics and weights them accordingly.

Field Weighting: Assign importance multipliers to each field
Per-Field Normalization: Each field has its own average length and saturation parameters
Used extensively in Elasticsearch for multi-field queries

Elasticsearch BM25

The default relevance scoring algorithm in Elasticsearch since version 5.0, implementing the Okapi BM25 function across sharded indices. It replaces the older TF-IDF default and is exposed through the BM25Similarity class in the underlying Apache Lucene library.

k1 and b parameters are configurable per field
Combines BM25 with coordination factors and field-length norms
Powers relevance in millions of production search applications

About the author

Prasad Kumkar

CEO & MD, Inference Systems

Prasad Kumkar is the CEO & MD of Inference Systems and writes about AI systems architecture, LLM infrastructure, model serving, evaluation, and production deployment. Over 5+ years, he has worked across computer vision models, L5 autonomous vehicle systems, and LLM research, with a focus on taking complex AI ideas into real-world engineering systems.

His work and writing cover AI systems, large language models, AI agents, multimodal systems, autonomous systems, inference optimization, RAG, evaluation, and production AI engineering.

Limited slotsGet a Free AI Consultation

How We Work

Custom AI workflows for your Business

One-fit-all AI don't work for modern businesses. At Inferensys, we aim to understand your business & custom requirements; which we use to define most efficient agentic workflows, the data, and the tools for your business.