BM25, or Best Match 25, is a probabilistic retrieval function that scores documents by calculating the weight of each query term's occurrence, independent of term proximity. It improves upon classic TF-IDF by introducing a non-linear saturation curve for term frequency, preventing a document that repeats a word excessively from dominating the rankings. The algorithm also applies a document length normalization parameter, calibrated by the b hyperparameter, to avoid bias toward longer documents.
Glossary
BM25
BM25 is a bag-of-words retrieval function that ranks documents based on query term frequency, inverse document frequency, and document length normalization within a probabilistic relevance framework.
PROBABILISTIC RELEVANCE FRAMEWORK
What is BM25?
BM25 is a bag-of-words retrieval function that ranks documents based on the frequency of query terms appearing in each document, using a probabilistic relevance framework that saturates term frequency and penalizes document length.
The core formula computes the Inverse Document Frequency (IDF) of a query term multiplied by a modified term frequency component. The k1 hyperparameter controls the saturation rate, defining how quickly additional term occurrences stop contributing to the score. BM25 remains the default lexical retrieval algorithm in many modern RAG architectures, often serving as the first-stage retriever before a neural re-ranker refines the candidate set for semantic relevance.
PROBABILISTIC RETRIEVAL
Key Features of BM25
BM25 is a bag-of-words retrieval function that ranks documents based on query term frequency, document length, and inverse document frequency. It remains the default relevance scoring mechanism in Elasticsearch and a critical baseline for modern hybrid search systems.
01
Term Frequency Saturation
Unlike TF-IDF, BM25 applies non-linear saturation to term frequency. The formula tf / (tf + k1) ensures that the fifth occurrence of a term contributes far less than the first. This prevents keyword stuffing from artificially inflating relevance scores. The k1 parameter (typically 1.2–2.0) controls the saturation curve:
- Lower
k1: faster saturation, less impact from repetition - Higher
k1: more linear scaling, closer to classic TF-IDF behavior
02
Document Length Normalization
BM25 normalizes term frequency by document length using the b parameter (typically 0.75). This addresses the verbosity hypothesis: longer documents naturally contain more term occurrences without necessarily being more relevant. The normalization factor is:
b = 0: no length normalizationb = 1: full normalization proportional to average document length- Default
b = 0.75provides a balanced penalty for verbose documents
03
Inverse Document Frequency Component
BM25 uses a probabilistic IDF formulation derived from the Robertson-Spärck Jones weight. The formula log((N - n + 0.5) / (n + 0.5)) penalizes common terms while rewarding rare, discriminative ones. Unlike TF-IDF's simpler log(N/n), this variant:
- Handles edge cases where a term appears in all documents
- Produces negative weights for terms appearing in more than half the corpus
- Provides a theoretically grounded probability of relevance framework
04
Elasticsearch Default Scoring
BM25 replaced TF-IDF as the default similarity algorithm in Elasticsearch 5.0 (2016) and Apache Lucene 6.0. In production systems, BM25 powers:
- Full-text search across billions of documents
- Hybrid retrieval pipelines where BM25 results are fused with dense vector search using Reciprocal Rank Fusion (RRF)
- Sparse retrieval baselines for evaluating neural ranking models like ColBERT and SPLADE
05
Bag-of-Words Independence Assumption
BM25 treats documents as unordered collections of terms, ignoring word order, proximity, and syntax. This is both its strength and limitation:
- Strength: Fast to compute, no GPU required, highly interpretable
- Limitation: Cannot capture semantic similarity or paraphrasing Modern systems compensate by combining BM25 with dense vector retrieval using embeddings from models like BERT, creating hybrid search architectures that leverage both lexical precision and semantic understanding.
06
Probabilistic Relevance Framework
BM25 is grounded in the Probability Ranking Principle, which states that optimal retrieval ranks documents by their probability of relevance given the query. The scoring function is derived from the Binary Independence Model using the 2-Poisson model for term eliteness. Key theoretical properties:
- Scores are additive across query terms
- No document-to-document comparisons required at query time
- Provides a statistically principled alternative to the vector space model
RETRIEVAL FUNCTION COMPARISON
BM25 vs. TF-IDF vs. Dense Retrieval
A technical comparison of sparse lexical retrieval functions against modern dense vector search for information retrieval pipelines.
| Feature | BM25 | TF-IDF | Dense Retrieval |
|---|---|---|---|
Core Mechanism | Probabilistic relevance framework with term saturation and document length normalization | Linear product of term frequency and inverse document frequency | Learned dense vector embeddings with approximate nearest neighbor search |
Term Frequency Saturation | |||
Document Length Normalization | |||
Handles Out-of-Vocabulary Terms | |||
Semantic Matching Capability | |||
Training Data Required | |||
Inference Latency (relative) | Low | Low | Medium-High |
Interpretability | High (exact term match scoring) | High (exact term match scoring) | Low (black-box vector similarity) |
Enabling Efficiency, Speed & Accuracy
Intelligent Analysis, Decision & Execution
We build AI systems for teams that need search across company data, workflow automation across tools, or AI features inside products and internal software.
Talk to Us
Search across company data
Give teams answers from docs, tickets, runbooks, and product data with sources and permissions.
Useful when people spend too long searching or get different answers from different systems.
Enterprise searchRAGPermissions
Read more
Automate internal workflows
Use AI to route work, draft outputs, trigger actions, and keep approvals and logs in place.
Useful when repetitive work moves across multiple tools and teams.
AI agentsWorkflow automationGovernance
Read more
Add AI to products and internal tools
Build assistants, guided actions, or decision support into the software your team or customers already use.
Useful when AI needs to be part of the product, not a separate tool.
AI integrationDecision supportModel routing
Read moreBM25 EXPLAINED
Frequently Asked Questions
Clear, technical answers to the most common questions about the BM25 probabilistic retrieval function, its mechanics, and its role in modern search and AI pipelines.
BM25 (Best Match 25) is a bag-of-words retrieval function that ranks documents by estimating the probability of relevance based on query term occurrences. It improves upon its predecessor, TF-IDF, by introducing term frequency saturation and document length normalization. The algorithm calculates a score for each document-query pair by summing the weighted frequencies of matching terms. A term's weight is determined by its Inverse Document Frequency (IDF), which penalizes common words. Critically, BM25 applies a non-linear saturation curve to term frequency—the function tf / (k1 + tf)—so that the fifth occurrence of a word contributes far less than the first. The k1 parameter controls this saturation slope. Simultaneously, the b parameter tunes document length normalization, preventing longer documents from having an unfair scoring advantage simply because they contain more words. This dual calibration makes BM25 highly effective for general-purpose text search.
About the author
Prasad Kumkar
CEO & MD, Inference Systems
Prasad Kumkar is the CEO & MD of Inference Systems and writes about AI systems architecture, LLM infrastructure, model serving, evaluation, and production deployment. Over 5+ years, he has worked across computer vision models, L5 autonomous vehicle systems, and LLM research, with a focus on taking complex AI ideas into real-world engineering systems.
His work and writing cover AI systems, large language models, AI agents, multimodal systems, autonomous systems, inference optimization, RAG, evaluation, and production AI engineering.
LinkedIn
Limited slotsGet a Free AI Consultation
Partnered with leading AI, data, and software stack.
How We Work
Custom AI workflows for your Business
One-fit-all AI don't work for modern businesses. At Inferensys, we aim to understand your business & custom requirements; which we use to define most efficient agentic workflows, the data, and the tools for your business.
01
Review the use case
We understand the task, the users, and where AI can actually help.
Read more02
Pick the right approach
We define what needs search, automation, or product integration.
Read more03
Build the first useful version
We implement the part that proves the value first.
Read more04
Improve from there
We add the checks and visibility needed to keep it useful.
Read moreThe first call is a practical review of your use case and the right next step.
Talk to Us