BM25, or Best Match 25, is a probabilistic bag-of-words retrieval function that ranks documents by calculating a relevance score for each document relative to a search query. It operates by weighting the significance of each query term based on its frequency in the document, its rarity across the entire corpus via inverse document frequency (IDF), and a normalization factor that accounts for document length to prevent longer documents from having an unfair advantage.
Glossary
BM25 Scoring
A probabilistic, bag-of-words retrieval function that ranks documents by estimating their relevance to a given search query based on term frequency, inverse document frequency, and document length normalization.
PROBABILISTIC RETRIEVAL FUNCTION
What is BM25 Scoring?
BM25 is a bag-of-words retrieval function that ranks documents by estimating their relevance to a given search query based on term frequency, inverse document frequency, and document length normalization.
The algorithm refines classic TF-IDF by introducing a non-linear saturation curve for term frequency, ensuring that a term appearing dozens of times does not linearly inflate the score. This makes BM25 the standard baseline for sparse retrieval in modern hybrid search pipelines, where its precise keyword-matching results are fused with the semantic understanding of dense vector retrieval.
PROBABILISTIC RETRIEVAL
Key Features of BM25
BM25 is a bag-of-words retrieval function that ranks documents by estimating their relevance to a query. It builds on TF-IDF but introduces critical refinements for term saturation and document length normalization.
01
Term Frequency Saturation
Unlike linear TF-IDF, BM25 applies a non-linear saturation curve to term frequency. The first occurrence of a term contributes significantly to the score, but each subsequent occurrence adds diminishing returns.
- Prevents keyword stuffing from dominating rankings
- Controlled by the
k1parameter (typically 1.2–2.0) - A term appearing 100 times is not 100x more relevant than one appearing once
02
Inverse Document Frequency (IDF)
BM25 uses a probabilistic IDF formula derived from the Robertson-Spärck Jones weight. Rare terms that appear in few documents receive higher weight than common terms.
- A term found in 10 out of 1M documents gets a massive boost
- Stop words like 'the' receive near-zero weight automatically
- IDF is computed globally across the entire corpus
03
Document Length Normalization
BM25 normalizes scores by document length, preventing longer documents from having an unfair advantage simply because they contain more words.
- Controlled by the
bparameter (typically 0.75) b=1.0fully normalizes by length;b=0disables normalization- Longer documents are penalized relative to their average corpus length
- Critical for fair comparison across heterogeneous document collections
04
No Term Coordination
BM25 treats each query term independently and sums their individual contributions. It does not model proximity, phrase structure, or term dependencies.
- A document containing 'machine' and 'learning' separately scores the same as one containing 'machine learning' as a phrase
- This is both a strength (simplicity, speed) and a limitation (no positional awareness)
- Often paired with phrase boosters or re-rankers in modern hybrid systems
05
Tunable Parameters: k1 and b
BM25 exposes two free parameters that allow practitioners to tune retrieval behavior for specific corpora and use cases.
- k1: Controls term frequency saturation rate. Higher values (1.5–2.0) allow more contribution from repeated terms
- b: Controls document length normalization strength.
b=0.75is the classic default - Grid search over (k1, b) is common during relevance tuning
- No training data required—parameters are set heuristically
06
Efficient Inverted Index Execution
BM25 scoring is implemented using inverted indexes, enabling sub-millisecond retrieval even across billion-document corpora.
- Posting lists store pre-computed document frequencies and lengths
- Query-time scoring requires only a few arithmetic operations per candidate
- Scales linearly with the number of matching documents, not corpus size
- Foundation of production search engines like Elasticsearch and Lucene
BM25 SCORING
Frequently Asked Questions
Clear, technical answers to the most common questions about the BM25 probabilistic retrieval function, its mechanisms, and its role in modern search pipelines.
BM25, or Best Match 25, is a bag-of-words probabilistic retrieval function that ranks documents by estimating their relevance to a given search query. It works by calculating a score for each document based on a combination of three main components: term frequency (TF), which measures how often a query term appears in a document, but with a saturation function to prevent the score from increasing linearly; inverse document frequency (IDF), which gives higher weight to rare terms that are more discriminative; and document length normalization, which prevents longer documents from having an unfair advantage simply because they contain more words. The formula applies a soft, non-linear normalization to term frequency using a parameter k1 (typically between 1.2 and 2.0), and adjusts for document length using a parameter b (typically 0.75). This makes BM25 significantly more effective than the classic TF-IDF weighting scheme for ad-hoc retrieval tasks.
Enabling Efficiency, Speed & Accuracy
Intelligent Analysis, Decision & Execution
We build AI systems for teams that need search across company data, workflow automation across tools, or AI features inside products and internal software.
Talk to Us
Search across company data
Give teams answers from docs, tickets, runbooks, and product data with sources and permissions.
Useful when people spend too long searching or get different answers from different systems.
Enterprise searchRAGPermissions
Read more
Automate internal workflows
Use AI to route work, draft outputs, trigger actions, and keep approvals and logs in place.
Useful when repetitive work moves across multiple tools and teams.
AI agentsWorkflow automationGovernance
Read more
Add AI to products and internal tools
Build assistants, guided actions, or decision support into the software your team or customers already use.
Useful when AI needs to be part of the product, not a separate tool.
AI integrationDecision supportModel routing
Read more
About the author
Prasad Kumkar
CEO & MD, Inference Systems
Prasad Kumkar is the CEO & MD of Inference Systems and writes about AI systems architecture, LLM infrastructure, model serving, evaluation, and production deployment. Over 5+ years, he has worked across computer vision models, L5 autonomous vehicle systems, and LLM research, with a focus on taking complex AI ideas into real-world engineering systems.
His work and writing cover AI systems, large language models, AI agents, multimodal systems, autonomous systems, inference optimization, RAG, evaluation, and production AI engineering.
LinkedIn
Limited slotsGet a Free AI Consultation
Partnered with leading AI, data, and software stack.
How We Work
Custom AI workflows for your Business
One-fit-all AI don't work for modern businesses. At Inferensys, we aim to understand your business & custom requirements; which we use to define most efficient agentic workflows, the data, and the tools for your business.
01
Review the use case
We understand the task, the users, and where AI can actually help.
Read more02
Pick the right approach
We define what needs search, automation, or product integration.
Read more03
Build the first useful version
We implement the part that proves the value first.
Read more04
Improve from there
We add the checks and visibility needed to keep it useful.
Read moreThe first call is a practical review of your use case and the right next step.
Talk to Us