Multi-Vector Embedding

Unlike bi-encoders that collapse an entire document into a single dense vector, multi-vector models encode each token independently. This preserves the distinct semantic contributions of individual words, allowing the model to match specific entities, attributes, or phrases without losing them in a global average.

• Each token in a query and document receives its own embedding
• Avoids the information bottleneck of fixed-size representations
• Enables precise matching of rare terms and named entities

The defining computational mechanism of models like ColBERT is the MaxSim operation. Rather than forcing a single similarity score upfront, MaxSim computes the cosine similarity between every query token and every document token, retaining only the maximum match for each query term. This sum-of-maximums approach allows the model to verify that all aspects of a query are addressed somewhere in the document.

• Computes all-pairs similarity between query and document token embeddings
• Retains the maximum similarity score for each query token
• Sums these maximums to produce a final relevance score

Multi-vector models generate significantly more data per document than single-vector approaches. A 128-token passage produces 128 vectors instead of one. To manage this, ColBERT employs aggressive compression techniques like product quantization (PQ) and dimensionality reduction, reducing the storage footprint by up to 50x while retaining high fidelity. This allows the token-level representations to be stored in memory for fast retrieval.

• Uses PQ to compress token embeddings into compact codes
• Balances storage cost against retrieval accuracy
• Enables end-to-end retrieval from millions of documents in milliseconds

ColBERT (Contextualized Late Interaction over BERT) is the canonical multi-vector model. Its retrieval engine, PLAID (Performance-optimized Late Interaction Driver), implements a multi-stage pipeline to make late interaction scalable. PLAID first uses a fast, coarse candidate generation step, then applies centroid pruning to discard irrelevant documents, and finally computes the full MaxSim only on the remaining candidates.

• Stage 1: Coarse candidate generation via approximate nearest neighbor lookup
• Stage 2: Centroid-based pruning to filter candidates
• Stage 3: Exact MaxSim computation on the final shortlist

While cross-encoders process query-document pairs jointly through full self-attention and achieve the highest accuracy, they are computationally prohibitive for large-scale retrieval. Multi-vector embedding occupies a sweet spot: it pre-computes document representations offline and performs a lightweight late interaction at query time. This provides accuracy closer to a cross-encoder with the speed closer to a bi-encoder.

• Bi-Encoder: Fastest, single vector per item, lowest accuracy
• Multi-Vector: Balanced speed and accuracy, token-level matching
• Cross-Encoder: Slowest, joint processing, highest accuracy

Single-vector models often struggle when a query term appears in a document with a different meaning. Multi-vector embeddings mitigate this by matching tokens in their specific contexts. The word 'bank' in a financial query will have a different token embedding than 'bank' in a river context, and the MaxSim operation will naturally align the query's financial 'bank' with the document's financial 'bank'.

• Token embeddings are contextualized by surrounding words
• Disambiguates polysemous terms automatically
• Improves precision for queries with multiple distinct constraints

Unlike cross-encoders that jointly process query-document pairs, or bi-encoders that collapse everything into a single vector, late interaction defers the costly attention computation until after retrieval. Document token embeddings are pre-computed and indexed. At query time, only the lightweight MaxSim summation is performed. This provides a crucial trade-off: 95%+ of the expressiveness of a cross-encoder with the pre-computability required for sub-second search over millions of documents.

Multi-vector embeddings operate at the sub-word token level, typically using a WordPiece or BPE vocabulary. A 512-token passage generates 512 distinct vectors. This granularity is the key advantage: a query for 'solar battery storage' can match the token 'photovoltaic' against a document's 'photovoltaic' token and 'battery' against 'battery', even if they appear in different sentences. Single-vector models would average these signals into a single point, losing this precise alignment.

Storing 128 vectors per document creates a storage and latency explosion. PLAID (Performance-optimized Late Interaction Driver) addresses this with a multi-stage pipeline:

• Centroid Pruning: Candidate documents are filtered by proximity to coarse cluster centroids.
• Residual Compression: Token vectors are stored as quantized residuals from their centroid, reducing memory by 10-20x.
• Candidate Generation: A fast, approximate stage eliminates 99% of documents before the expensive MaxSim computation. This makes ColBERT-style retrieval practical at billion-document scale.

Single-vector bi-encoders (e.g., text-embedding-3-large) compress an entire passage into one fixed-size vector. This is fast but loses granularity. Multi-vector models retain token-level information, enabling:

• Higher Recall@K: Especially for rare entities and long-tail queries.
• Explainability: The MaxSim operation produces a token-to-token alignment heatmap, showing exactly why a document matched.
• Cost: 10-100x more storage and slower retrieval without optimizations like PLAID. Choose single-vector for latency-critical, high-throughput systems; multi-vector for precision-critical, explainable search.

Token embeddings in ColBERT are typically 128-dimensional, compared to 768 or 1024 for BERT's hidden states. This is a deliberate dimensionality reduction via a linear projection layer, trained end-to-end. Further compression is achieved through:

• Product Quantization (PQ): Decomposing the 128-d vector into sub-vectors and clustering each subspace.
• Binary Quantization: Binarizing each dimension to a single bit, enabling Hamming distance computation. These techniques are essential for fitting multi-vector indexes in memory.