Query2vec

Query2vec moves beyond simple word vector averaging by learning to represent the holistic meaning of a multi-word query. It accounts for word order and syntactic dependencies, distinguishing 'hot dog' from 'dog hot'.

• Trained on query-query or query-document pairs
• Captures non-linear interactions between query terms
• Produces a single fixed-length dense vector for the entire query

The model is typically trained using weak supervision signals derived from search engine logs. Pairs of queries that lead to clicks on the same document are treated as semantically similar.

• Session co-occurrence: Queries issued in the same session
• Click co-occurrence: Queries leading to clicks on the same URL
• Reformulation pairs: A query and its subsequent refinement by the user

Several neural architectures can implement the Query2vec concept, each with different inductive biases for sequence modeling.

• Convolutional Neural Networks (CNNs): Capture local n-gram patterns within the query
• Recurrent Neural Networks (RNNs/LSTMs): Model the sequential order of terms
• Transformer Encoders: Use self-attention to weigh the contextual importance of each term relative to others in the query

In the embedding space, the cosine similarity between two query vectors directly measures their semantic relatedness. This enables efficient nearest neighbor search to find equivalent reformulations.

• 'cheap flights to paris' is near 'budget airline tickets paris'
• 'how to fix a leaky faucet' is near 'diy faucet repair guide'
• Enables query clustering and trend analysis

Query2vec vectors are indexed using Approximate Nearest Neighbor (ANN) libraries like FAISS or ScaNN for low-latency retrieval. This allows real-time lookup of similar historical queries.

• Billion-scale query indexes become searchable in milliseconds
• Enables query auto-completion and related search suggestions
• Powers query normalization by mapping to a canonical form

A naive approach averages individual Word2vec vectors. Query2vec learns a compositional function that weights terms contextually, providing a more accurate semantic representation.

• Word2vec Avg: 'machine learning' ≈ 'learning machine'
• Query2vec: 'machine learning' ≠ 'learning machine' (captures bigram semantics)
• Handles out-of-vocabulary terms by composing subword representations

A precursor to Query2vec that uses static word vectors like Word2Vec or GloVe to find semantically similar terms. Unlike Query2vec's holistic query-level encoding, this method averages or combines individual word embeddings to represent the query, then retrieves terms with high cosine similarity in the embedding space.

• Relies on distributional semantics from large text corpora
• Cannot resolve polysemy—'bank' has the same vector regardless of context
• Computationally lighter than transformer-based alternatives
• Often used as a baseline for evaluating neural expansion methods

Uses deep transformer models like BERT to generate expansion terms that are sensitive to the query's specific context. Unlike static embeddings, each word token receives a unique vector based on surrounding words, resolving ambiguity.

• Captures polysemy—'apple' in 'apple fruit' vs. 'apple stock' yields different vectors
• Expansion terms are extracted from the model's masked language modeling head
• Significantly higher precision than static embedding methods
• Query2vec extends this concept by encoding the entire query as a single dense vector rather than expanding individual terms

Employs large language models such as T5 or GPT to generate relevant expansion terms, synonyms, or complete alternative queries from a prompt. This represents a shift from retrieval-based to generation-based expansion.

• Models like doc2query predict queries a document might answer and append them to the index
• Can produce zero-shot expansions without training on domain-specific relevance data
• Query2vec differs by learning a dense representation for similarity search rather than generating discrete tokens
• Generative methods excel at paraphrase generation and handling tail queries

An automatic expansion technique that assumes the top-k documents from an initial retrieval are relevant and extracts expansion terms from them. This creates a feedback loop without explicit user input.

• Terms are typically selected using TF-IDF or KL divergence scoring
• The Rocchio algorithm reformulates the query vector by adding relevant document centroids
• Query2vec can replace the initial retrieval step with dense vector search for higher recall
• Risk of query drift if top documents are not truly relevant

Combines sparse retrieval signals like BM25 with dense retrieval signals from models like Query2vec to leverage the strengths of both paradigms. Sparse methods excel at exact term matching while dense methods capture semantic similarity.

• Reciprocal Rank Fusion (RRF) is a common algorithm for merging result lists
• Query2vec embeddings serve as the dense component in hybrid architectures
• Addresses the vocabulary mismatch problem where relevant documents use different terminology
• Production search systems increasingly adopt hybrid approaches for robust retrieval

The training paradigm underlying Query2vec models. Uses Siamese or Bi-Encoder architectures to learn embeddings where semantically similar query-document pairs are pulled together in vector space while dissimilar pairs are pushed apart.

• Training data consists of positive pairs (relevant query-document) and negative pairs (non-relevant)
• In-batch negatives efficiently reuse examples within a training batch as negative samples
• Loss functions like InfoNCE or triplet loss optimize the embedding space
• This framework directly produces the dense representations used in Query2vec retrieval