Multi-Vector Retrieval

Unlike single-vector approaches that compress an entire document into one embedding, multi-vector retrieval generates multiple embeddings per document—one for each chunk, section, or semantic facet. This allows the retrieval system to match a query against the most relevant part of a document rather than a diluted whole-document average. For example, a long technical paper on transformer architectures might have separate vectors for its attention mechanism section, training methodology, and benchmark results, ensuring a query about 'multi-head attention' retrieves the paper based on the precise relevant passage.

Documents can be indexed with multiple complementary vector representations, each serving a distinct retrieval purpose:

• Dense passage vectors: Embeddings of individual chunks for precise content matching
• Summary vectors: A condensed representation capturing the document's core thesis for broad topical matching
• Hypothetical question vectors: Embeddings of synthetic questions the document answers, generated via HyDE-style techniques, enabling query-to-question matching
• Entity-centric vectors: Embeddings focused on key named entities and their relationships within the document This multi-faceted indexing ensures a document is discoverable through diverse query formulations.

Multi-vector retrieval operates at two primary granularities:

• Token-level (ColBERT-style): Every token receives its own contextualized embedding, enabling the finest possible matching granularity. A query about 'Python' will match documents discussing programming languages, not snakes, because surrounding token context shapes each embedding.
• Chunk-level (multi-representation): Documents are split into semantically coherent chunks, each independently embedded. This is more storage-efficient than token-level approaches while still providing significantly better precision than single-vector document embeddings. The choice depends on the precision-storage trade-off required by the application.

Multi-vector retrieval naturally supports query-side decomposition, where a complex query is broken into multiple sub-queries or aspects, each embedded separately. For instance, the query 'Compare BERT and GPT architectures for sentiment analysis' can be decomposed into vectors for 'BERT architecture', 'GPT architecture', and 'sentiment analysis performance'. Each sub-query vector independently retrieves relevant document chunks, and results are fused using algorithms like Reciprocal Rank Fusion (RRF). This approach dramatically improves recall for multi-faceted or comparative queries that single-vector search struggles to satisfy.

Multi-vector retrieval introduces significant infrastructure considerations compared to single-vector approaches:

• Storage footprint: A document with 500 tokens generates 500 vectors in ColBERT-style indexing, requiring substantially more storage than a single document embedding
• Index size: Vector indexes must scale to accommodate millions of token-level or chunk-level vectors rather than thousands of document vectors
• Retrieval latency: The MaxSim scoring operation adds computational overhead during retrieval, though this is mitigated by approximate nearest neighbor (ANN) pre-filtering to reduce the candidate set
• Mitigation strategies: Techniques like dimensionality reduction, product quantization, and Matryoshka embeddings can compress multi-vector representations while preserving retrieval quality

A common enterprise pattern stores two vectors per document: one for a high-level summary and one for each granular chunk. The summary vector captures the document's overall theme for broad topic matching, while chunk vectors enable precise passage retrieval. This is often implemented using LangChain's Parent Document Retriever or LlamaIndex's SummaryIndex. A user query first matches against summary vectors to identify relevant documents, then searches within those documents' chunk vectors to find the exact passage. This two-stage approach prevents the common failure mode where a document's average embedding fails to represent any single concept strongly enough to be retrieved.

Documents containing both text and images require separate vector representations for each modality. A product catalog page, for example, might be indexed with:

• A text embedding of the product description
• An image embedding of the product photo
• A table embedding of the specifications At query time, a user's natural language query is embedded and matched against all three vector spaces simultaneously. The results are fused using Reciprocal Rank Fusion (RRF) to produce a unified ranking. This architecture is foundational for e-commerce search, technical documentation portals, and medical literature retrieval where diagrams and charts carry critical information distinct from surrounding text.

Propositional chunking decomposes a document into atomic, self-contained factual statements, each independently embedded. A single paragraph might produce five or more proposition vectors. This is a form of multi-vector retrieval where the document is represented by the set of its constituent facts. When a query asks a specific question, the system retrieves the exact proposition that answers it, rather than a large block of text containing the answer alongside irrelevant information. This technique is critical for fact-checking systems, medical question answering, and legal citation where precision and granular attribution are non-negotiable. Tools like FactScore and custom spaCy pipelines implement this pattern.

A neural retrieval model that generates token-level embeddings for both queries and documents, then computes relevance via a scalable MaxSim operation. Unlike single-vector approaches, ColBERT stores multiple vectors per document—one per token—enabling fine-grained lexical and semantic matching. This late-interaction paradigm preserves the expressiveness of cross-encoders while maintaining the indexing speed of bi-encoders, making it a foundational implementation of multi-vector retrieval.

A technique where a long-context embedding model first processes an entire document to generate token-level embeddings, which are then segmented into chunks. This preserves cross-chunk contextual awareness that is lost when chunks are embedded independently. The resulting chunk embeddings are richer because each token's representation was influenced by the full document context during encoding, directly supporting multi-vector strategies where multiple enriched vectors represent a single source.

An embedding training method that produces vectors where the first few dimensions capture the most salient information. This enables truncation to smaller dimensions—such as 256, 512, or 768—with minimal accuracy loss. In multi-vector retrieval systems, MRL embeddings allow flexible storage and speed trade-offs: coarse filtering can use truncated vectors while fine-grained re-ranking uses full-dimensional representations, all from a single model.

An algorithm that merges ranked result lists from multiple retrieval sources by assigning a reciprocal score based on each document's rank position. In multi-vector architectures, RRF is critical for combining results from different vector representations of the same document—such as summary embeddings, chunk embeddings, and hypothetical document embeddings—into a single, consensus ranking without requiring normalized relevance scores.

A query expansion technique where a language model generates a synthetic ideal document in response to a query, and the embedding of that hypothetical document is used to search the vector store. This creates a multi-vector query representation: the raw query embedding and the HyDE embedding can be fused. HyDE bridges the gap between short queries and the richer document representations typical in multi-vector indexes.

A fine-grained chunking method that decomposes text into atomic, self-contained propositions, each expressing a single fact. This creates multiple independent vectors per original sentence, directly enabling multi-vector retrieval at the fact level. Each proposition can be independently matched, cited, and verified, maximizing retrieval precision for grounding tasks where a single document may contain dozens of distinct, retrievable claims.