Document Expansion

Unlike query expansion, which modifies the user's input at search time, document expansion processes the entire corpus offline. A generative model analyzes each document and appends relevant terms, synonyms, or even full hypothetical sentences directly to the document's text or sparse representation. This ensures the expanded terms are indexed and immediately searchable via standard inverted indexes like BM25, without adding latency to the retrieval pipeline.

The core problem solved is vocabulary mismatch. A user searching for 'cardiac arrest' might miss a document only containing 'myocardial infarction'. Document expansion uses a model to predict that these terms are contextually related and adds 'cardiac arrest' to the document's index entry. This improves recall for sparse retrieval methods by creating a denser, more semantically rich term space without abandoning the efficiency of keyword matching.

A prominent technique uses a sequence-to-sequence model, often a fine-tuned T5 variant, trained to generate potential queries that a given document can answer. For each document, the model predicts multiple queries (e.g., 5-10), which are then appended to the document. This is known as Doc2Query. The expanded document is then indexed normally, making it highly likely to match a user's future, similarly phrased question.

An alternative to generative text expansion is Learned Sparse Retrieval, where a neural model like SPLADE directly predicts term importance weights for the entire vocabulary. This creates a high-dimensional, sparse vector for each document where terms not originally present can have a non-zero weight. This effectively performs document expansion in a latent, weighted space, combining the interpretability of sparse vectors with the semantic power of neural models.

Expansion can be guided by structured knowledge. Using an enterprise knowledge graph or a domain-specific ontology, a document mentioning 'Apple' can be unambiguously expanded with 'AAPL', 'Cupertino', and 'Tim Cook' if the entity is resolved to the technology company. This entity linking step ensures high-precision expansion, adding only verified, disambiguated terms and avoiding the noise that can be introduced by purely generative models.

The primary risk of document expansion is query drift or a loss of precision. Adding too many loosely related terms can cause a document to be retrieved for irrelevant queries. For example, expanding a document about 'python' (the snake) with terms like 'programming' and 'script' would be incorrect. Effective deployment requires careful model tuning, often using pseudo-relevance feedback signals, to balance the significant gains in recall against the potential degradation in precision.

The inverse counterpart to document expansion. Instead of enriching the document at index time, query expansion augments the user's original search terms with synonyms, related concepts, or generated text at query time. Common techniques include:

• Synonym expansion: Adding words with identical meanings
• Pseudo-Relevance Feedback (PRF): Extracting terms from top initial results
• LLM-based expansion: Using generative models to produce additional context

The key tradeoff: query expansion is computationally cheaper at index time but adds latency to every search, while document expansion shifts the cost to indexing.

A classic information retrieval technique that assumes the top-k documents from an initial retrieval pass are relevant. Key terms are extracted from these documents and added to the original query for a second, improved retrieval pass.

• Blind feedback: No user judgment required
• Term selection: Uses tf-idf or more sophisticated weighting
• Iterative: Can be applied multiple times

PRF is a form of automatic query expansion that leverages the corpus itself to bridge the vocabulary gap, making it a natural complement to document expansion strategies.

A modern paradigm where a neural model predicts term importance weights for a document, creating a sparse, high-dimensional representation. Models like SPLADE learn to expand documents with relevant terms during training.

• Combines the efficiency of inverted indexes with neural nuance
• Automatically learns which terms to add or suppress
• Outperforms BM25 while maintaining interpretability

This approach blurs the line between document expansion and retrieval modeling, as the expansion is learned end-to-end as part of the retrieval objective.

The probabilistic retrieval function that document expansion is designed to enhance. BM25 ranks documents using term frequency, inverse document frequency, and document length normalization.

• Sparse matching: Requires exact term overlap
• Vocabulary mismatch problem: Fails when queries and documents use different words for the same concept
• Document expansion's target: Adding terms to documents increases the probability of BM25 term matches

Understanding BM25's limitations is essential to appreciating why document expansion improves sparse retrieval performance.

An alternative paradigm that encodes queries and documents into dense vector embeddings using neural networks, enabling semantic matching via approximate nearest neighbor search. Unlike document expansion, dense retrieval:

• Captures semantics: Matches concepts, not just keywords
• No explicit expansion: The embedding space implicitly handles vocabulary mismatch
• Higher computational cost: Requires GPU-accelerated vector search

Many production systems combine document-expanded sparse retrieval with dense retrieval in a hybrid search architecture for maximum recall.