Sparse Retrieval

Sparse retrieval relies on exact lexical overlap between query and document terms. Unlike dense retrieval, it does not encode semantic meaning—a query for 'automobile' will not match a document containing only 'car' unless synonym expansion is applied. This property creates the vocabulary mismatch problem, where relevant documents using different terminology are missed. However, explicit matching provides full interpretability: every score contribution can be traced to a specific term occurrence, making debugging and relevance tuning straightforward for search engineers.

Sparse retrieval is powered by the inverted index data structure, which maps each unique term to a postings list of document identifiers and associated metadata. During query evaluation, only the postings lists for query terms are accessed, enabling sub-linear search times even across billions of documents. This architecture supports efficient Boolean retrieval, phrase matching via positional data, and dynamic index updates. The inverted index is the foundational infrastructure that makes sparse retrieval scalable for production search engines like Apache Lucene and Elasticsearch.

Term weights in sparse vectors are derived from statistical signals:

• Term Frequency (TF): How often a term appears in a document, indicating local importance
• Inverse Document Frequency (IDF): How rare a term is across the collection, suppressing common words like 'the'
• Document Length Normalization: Prevents longer documents from dominating results

The BM25 algorithm refines these signals with a saturation function that models diminishing returns for repeated term occurrences, controlled by the k1 parameter. The b parameter tunes length normalization strength.

Before indexing, text passes through an analyzer chain that transforms raw strings into searchable tokens:

• Tokenizer: Splits text on whitespace and punctuation boundaries
• Stemmer: Reduces words to roots (e.g., 'running' → 'run') to improve recall
• Stop Word Filter: Removes high-frequency terms with low discriminative value
• Synonym Filter: Expands tokens to handle known vocabulary variations
• N-gram Indexing: Decomposes text into overlapping character sequences for robust sub-word matching and typo tolerance

Each preprocessing decision represents a precision-recall tradeoff that search engineers must calibrate.

Modern sparse retrieval extends beyond flat text via BM25F, which handles structured documents with multiple fields of varying importance. A document's title, body, and metadata are scored independently with per-field statistics, then combined into a final relevance score. The Lucene Practical Scoring Function further incorporates:

• Coordination factors rewarding documents matching more query terms
• Field-length norms stored at index time for fast lookup
• Boost factors applied to specific fields or documents

This field-aware architecture enables precise control over ranking behavior in enterprise search applications.

To mitigate the vocabulary mismatch problem, sparse retrieval systems employ query expansion strategies:

• Relevance Feedback: Users mark relevant documents from initial results; terms from those documents augment the query
• Pseudo-Relevance Feedback: Automatically assumes top-K results are relevant and extracts expansion terms without user intervention
• Synonym Expansion: Applies curated or learned synonym mappings at query time
• Spelling Correction: Normalizes misspelled query terms before retrieval

These techniques bridge the gap between exact matching and the semantic understanding provided by dense retrieval, forming a critical component of hybrid search architectures.