doc2query

At its core, doc2query employs a sequence-to-sequence (seq2seq) model, typically based on the Transformer architecture. The model is trained on pairs of documents and their associated queries to learn the mapping from document text to potential search queries. During inference, the model takes a document's text as input and autoregressively generates multiple synthetic queries that the document could answer. This is fundamentally different from query expansion because it expands the document side of the index rather than modifying the user's query at search time.

The primary value of doc2query is resolving lexical mismatch between how users search and how authors write. Key mechanisms include:

• Synonym generation: A document about 'automobiles' might generate the query 'cars'
• Paraphrase normalization: Technical jargon in the document gets expanded into layperson query terms
• Question generation: Declarative sentences are transformed into interrogative forms users actually type
• Entity variation: Formal entity names generate colloquial references and abbreviations

This effectively increases recall without requiring manual synonym dictionaries or query-time expansion.

The seq2seq model is trained using supervised learning on large corpora of document-query pairs. Common training data sources include:

• MS MARCO dataset: Real Bing search queries paired with relevant passages
• Natural Questions: Google queries paired with Wikipedia articles
• Proprietary search logs: Internal click-through data from enterprise search systems

The model learns to maximize the conditional probability of generating the actual query given the document text. Once trained, the model is applied to the target corpus at indexing time, generating queries that are appended to each document's indexed text fields.

doc2query was originally designed for sparse retrieval systems like BM25, where the generated queries add new tokens to the inverted index. However, the technique has evolved:

• doc2query-T5: Uses the T5 text-to-text transformer for higher quality query generation
• docT5query: A specific implementation that fine-tunes T5 on MS MARCO, appending predicted queries to document text before BM25 indexing
• Dense integration: Generated queries can also be encoded into dense vectors for hybrid retrieval, enriching the document's semantic representation in the embedding space
• Multi-stage pipelines: Generated queries improve first-stage recall, while cross-encoder re-rankers handle precision

Document expansion with doc2query introduces specific computational considerations:

• Indexing-time cost: Query generation runs once per document during indexing, not at query time, making it an offline cost
• Index size inflation: Appending generated queries can increase the inverted index size by 2-5x, depending on the number of queries generated per document
• Generation quality vs. speed: Larger models like T5 produce better queries but slower generation; distilled models offer practical trade-offs
• Noise management: Not all generated queries are equally useful; techniques like confidence filtering or limiting the number of appended queries help control noise

Despite these costs, the technique is widely adopted because it shifts computation to indexing time, preserving low-latency query serving.

doc2query is the dual operation of traditional query expansion. While query expansion modifies the user's search terms at runtime, doc2query modifies the document representation at indexing time. This distinction matters:

• Query expansion adds latency to every search request
• doc2query front-loads computation to indexing, keeping search fast
• Query expansion must generalize across all possible documents
• doc2query can be document-specific, generating tailored queries per document

In practice, the two techniques are complementary and often combined in production search systems for maximum recall improvement.

The use of large language models (T5, GPT) to generate relevant expansion terms, synonyms, or full alternative queries from a prompt.

• Uses seq2seq architectures similar to doc2query
• Operates on the query side rather than the document side
• Can generate context-aware paraphrases that capture user intent
• Often combined with doc2query for bidirectional vocabulary alignment

An automatic query expansion technique that assumes the top-k documents from an initial retrieval are relevant and extracts expansion terms from them.

• Contrasts with doc2query's offline document expansion approach
• Operates at query time, adding latency to retrieval
• Risk of query drift if top results are not truly relevant
• doc2query avoids this by expanding documents independently of any query

A query expansion method that uses deep language models like BERT to generate expansion terms that are semantically appropriate for the query's specific context.

• Leverages contextualized token representations
• Unlike doc2query's generative approach, this method selects terms from pre-computed embedding spaces
• Handles polysemy by disambiguating word senses in context
• Complements doc2query by enriching the query encoding pipeline

A query expansion technique that adds words with identical or highly similar meanings to the original query terms to increase recall.

• Relies on curated thesauri or lexical databases like WordNet
• Simpler and more deterministic than doc2query's learned approach
• Cannot generate multi-word phrases or capture document-specific terminology
• Often used as a baseline against which neural expansion methods are benchmarked

A query expansion technique that uses a model to generate alternative phrasings of the original query that convey the same meaning.

• Shares doc2query's sequence-to-sequence foundation
• Applied to the input query rather than the indexed document
• Can produce syntactically diverse reformulations
• When paired with doc2query, creates a dual-expansion pipeline that aligns both sides of the retrieval equation

Combines sparse retrieval (BM25) and dense retrieval (embeddings) results using techniques like reciprocal rank fusion.

• doc2query bridges these paradigms by adding lexical anchors to dense indexes
• Expanded documents improve BM25 recall while preserving dense matching
• Fusion strategies determine how expansion terms influence final ranking
• Critical infrastructure for deploying doc2query in production search systems