Learned Sparse Retrieval

Unlike static formulas like BM25, learned sparse retrieval uses a deep neural network to predict the contextual importance of each vocabulary term for a document. The model learns to assign higher weights to terms that are semantically central, even if they appear infrequently, and can down-weight common but irrelevant words. This produces a term importance vector where each dimension corresponds to a specific token in the vocabulary, and the value represents its learned significance.

The output is a high-dimensional but sparse vector, where the vast majority of values are exactly zero. This sparsity is a critical design choice, not a limitation. It allows the representations to be indexed using an inverted index, the same battle-tested data structure that powers traditional search engines like Elasticsearch. This means learned sparse retrieval inherits the sub-millisecond latency and scalability of keyword search while operating on semantically richer term weights.

A key advantage is the model's ability to perform implicit document expansion at inference time. The neural network can activate terms in the sparse vector that never literally appear in the source text but are semantically related. For example, a document about 'canines' might receive a high weight for the term 'dog'. This bridges the vocabulary mismatch problem without requiring explicit synonym lists or manual thesauri, all learned directly from training data.

Because the final relevance score is computed as a dot product between two sparse vectors, the contribution of each individual term is fully transparent and auditable. A search architect can inspect exactly which query terms matched which document terms and their respective weights. This white-box nature is a significant advantage over dense embeddings for regulated industries requiring explainability, debugging relevance issues, and tuning search behavior with precision.

Models like SPLADE and uniCOIL are typically trained using large-scale relevance datasets such as MS MARCO. The training objective often involves a ranking loss (e.g., contrastive loss) that teaches the model to assign higher dot-product scores to relevant query-document pairs than to non-relevant ones. A crucial regularizer, such as the FLOPS loss, is applied to enforce sparsity, penalizing the model for activating too many non-zero terms and ensuring the output remains efficient for inverted indexing.

Learned sparse retrieval is a powerful component in a hybrid search architecture. Its scores can be linearly combined with dense vector similarity scores through a fusion algorithm like Reciprocal Rank Fusion (RRF). This pairing is highly effective: the learned sparse component handles precise term matching and rare entities, while the dense component captures broader topical and semantic similarity. The result is a retrieval pipeline that is robust across a wide range of query types.

A family of models that predict term importance for the entire BERT vocabulary, not just the terms present in the document. It uses a log-saturation effect and FLOPS regularization to ensure the resulting sparse vectors are efficient for inverted indexing.

• Mechanism: A masked language model head predicts a weight for every token in the vocabulary.
• Key Innovation: Automatic query and document expansion via the MLM head.
• Regularization: The FLOPS loss penalizes high-weight tokens, balancing sparsity and recall.

A doc-term weighting model that uses a contextualized encoder (like BERT) to score the semantic importance of each token in a document. Unlike static IDF, DeepImpact learns to assign higher weights to terms that are central to the document's meaning.

• Architecture: A transformer encoder processes the document, and a linear layer projects each token's final embedding to a scalar weight.
• Indexing: The output is a standard inverted index where postings are augmented with learned, floating-point weights.
• Efficiency: Enables exact, fast retrieval using existing inverted index infrastructure.

A joint model for learning sparse representations for both queries and documents. It simplifies the process by mapping a token's contextualized embedding to a single scalar weight, creating a direct term-to-weight mapping.

• Training: Uses a dual-encoder approach with a dot-product scoring function between query and document term weights.
• Distinction: Unlike SPLADE, it typically constrains the vocabulary to the terms actually appearing in the input, avoiding full-vocabulary expansion.
• Advantage: Produces highly compact, lexically grounded representations.

A model that independently scores each query term's likelihood of being important for a document. It uses a BERT encoder to generate a probability distribution over vocabulary terms for each document token position.

• Scoring: The relevance score is the sum of the log-probabilities of each query term appearing in the document's predicted distribution.
• Indexing: Pre-computes and stores a term-by-document score matrix, which can be efficiently compressed.
• Interpretability: The per-term scoring provides a clear, auditable signal for why a document was retrieved.

A document expansion framework that uses a query-agnostic model to predict which terms should be added to a document's representation to improve its retrievability. It generates a weighted set of expansion terms.

• Process: Encodes the document, then uses a prediction head to generate a sparse vector of expansion terms with learned weights.
• Integration: The expansion terms are appended to the original document's inverted index entry.
• Goal: Bridges the lexical gap without requiring query-time neural computation.

A model that extends learned sparse retrieval to incorporate user context and personalization. It learns to adjust term weights based on a user's historical interactions and preferences.

• Input: The query, document, and a user representation vector are combined.
• Output: A personalized sparse vector that emphasizes terms relevant to the specific user's domain of interest.
• Use Case: Enterprise search where different departments have distinct relevance criteria for the same query.