Learned Sparse Retrieval

Unlike static methods like BM25 that rely on corpus-level statistics, learned sparse retrieval uses a neural model to predict the importance of each vocabulary token in the specific context of the input text. This allows the model to assign high weight to a term that is rare in the corpus but critical to the query's meaning, or to down-weight common terms that are contextually irrelevant. The output is a sparse vector where non-zero dimensions correspond to tokens in a fixed vocabulary, and the magnitude represents the learned importance.

A defining advantage of learned sparse retrieval is that its output—a sparse vector of term weights—is fully compatible with a traditional inverted index. This means the vast infrastructure and decades of optimization for efficient keyword search can be leveraged directly. The learned term weights are used as term frequencies or boost values during scoring, enabling sub-millisecond retrieval over millions of documents without the need for specialized vector databases or approximate nearest neighbor (ANN) indexes.

The neural model can assign weight to terms that do not literally appear in the input text, a process known as sparse lexical expansion. For example, a query for 'canine' might assign a high weight to the unmentioned term 'dog'. This bridges the vocabulary mismatch problem that plagues pure lexical search, providing a form of semantic matching without leaving the sparse, interpretable token space. The expansion is controlled by the model's learned associations within its fixed vocabulary.

The term weighting model is trained end-to-end on relevance data, not on a proxy task like language modeling. Using contrastive loss with hard negative mining, the model learns to assign weights that directly optimize for distinguishing relevant documents from irrelevant ones. This allows the model to learn nuanced retrieval signals, such as which terms are critical for a specific domain, that hand-crafted formulas like BM25 cannot capture.

Because the representation is sparse and each dimension maps directly to a human-readable vocabulary token, the retrieval process is fully interpretable. A developer can inspect the query vector to see exactly which terms (and their weights) were used to retrieve documents. This is a critical advantage over dense embeddings for debugging, auditing, and building trust in high-stakes enterprise search applications where understanding why a document was retrieved is as important as the retrieval itself.

Learned sparse retrieval is ideally positioned as the candidate generation stage in a multi-stage retrieval pipeline. It provides a high-recall, computationally cheap first pass that can efficiently prune a corpus of billions of documents down to a few hundred candidates. These candidates are then passed to a more expensive and precise cross-encoder reranker for final scoring. This architecture combines the speed of sparse search with the deep semantic understanding of a transformer.

The dominant architecture for Learned Sparse Retrieval, SPLADE uses a BERT-based model with log-saturation and FLOPS regularization to predict term importance for the entire vocabulary. Unlike static term frequency, it performs query/document expansion by activating semantically related but lexically distinct tokens.

• Mechanism: A masked language model head predicts a weight for every token in the vocabulary.
• Sparsity Control: FLOPS loss penalizes the number of non-zero activations, ensuring the index remains efficient.
• Result: A sparse vector where "neural_network" might have a weight of 2.1 and "deep_learning" a weight of 1.8, even if only one term appeared in the text.

A critical advantage of Learned Sparse Retrieval is its native compatibility with the mature inverted index infrastructure that powers traditional search engines like Lucene and Elasticsearch. The predicted term-weight pairs map directly to a postings list structure.

• No ANN Required: Unlike dense vectors, sparse representations use exact term matching, avoiding the accuracy trade-offs of Approximate Nearest Neighbor (ANN) algorithms.
• Lexical Anchoring: Guarantees exact keyword matches for rare entities, product codes, or IDs that dense models often miss.
• Incremental Indexing: New documents can be added to the inverted index without re-indexing the entire corpus, a significant operational advantage over static dense indexes.

Learned Sparse models are trained using contrastive loss functions that require careful selection of negative documents. Hard negative mining is essential to teach the model fine-grained lexical distinctions.

• In-Batch Negatives: Other documents in the same training batch serve as initial negatives, but these are often too easy to discriminate.
• Hard Negatives: Documents retrieved by a baseline system (like BM25) that are top-ranked but not relevant. These force the model to learn why a document about "Java the island" is not relevant for "Java the programming language."
• Distillation Targets: Often trained using the relevance scores from a powerful cross-encoder teacher model to provide soft labels.

Learned Sparse Retrieval models can perform expansion on the document side, the query side, or both, fundamentally altering the retrieval dynamics.

• Document Expansion (doc2query): Enriches a document at indexing time by adding predicted relevant terms. A document containing "CPU" might be expanded with "processor," "microchip," and "silicon." This increases recall but inflates the index size.
• Query Expansion: Transforms a short user query into a richer, more descriptive set of weighted terms at search time. The query "fix bike" might be expanded to "repair bicycle maintenance."
• SPLADE's Innovation: Performs both simultaneously within a single end-to-end model, learning the optimal expansion strategy for each.

An alternative to rank-based fusion like RRF, score calibration transforms raw model outputs into probabilities that can be meaningfully combined using a Weighted Sum Fusion.

• Platt Scaling: Fits a logistic regression model on a held-out validation set to map raw scores to empirical relevance probabilities.
• Histogram Binning: Divides the score range into bins and assigns a calibrated probability based on the fraction of relevant documents in each bin.
• Use Case: Essential when you need to dynamically weight the sparse vs. dense contribution based on query intent—e.g., prioritizing exact match for a product code search while favoring semantics for a conceptual query.