Late Interaction

Unlike standard bi-encoders that compress an entire passage into a single vector, late interaction models store multi-vector representations—typically one embedding per token. This preserves fine-grained linguistic information such as word order, syntax, and local context that is lost during mean pooling. The index stores these token-level embeddings, enabling a more nuanced comparison at query time without requiring the document to be re-encoded.

The core relevance computation in late interaction is the Maximum Similarity (MaxSim) operator. For each query token embedding, the model finds the document token embedding with the highest cosine similarity and sums these maximum values. This allows the model to softly align query terms to the most relevant parts of a document, handling lexical mismatch and synonymy without the quadratic complexity of full cross-attention.

All document-side computation is performed offline and stored in a vector index. During indexing, a passage is encoded into a matrix of token embeddings. At query time, only the query is encoded, and the MaxSim scoring is computed against the pre-computed document token embeddings. This asymmetric computation shifts the heavy lifting to the indexing phase, enabling sub-second retrieval latency even against large corpora.

Late interaction occupies a unique position on the speed-accuracy Pareto frontier:

• Bi-Encoders: Fastest retrieval via single-vector dot product, but lose token-level detail.
• Cross-Encoders: Highest accuracy via full joint attention, but computationally prohibitive for retrieval.
• Late Interaction: Achieves cross-encoder-like expressiveness by storing token vectors, while maintaining bi-encoder-like speed through pre-computation and efficient MaxSim aggregation.

The canonical implementation of late interaction is ColBERT (Contextualized Late Interaction over BERT). It uses a BERT-based encoder shared between queries and documents, but processes them independently. Documents are encoded into a bag of token embeddings, and queries are encoded similarly. The final relevance score is the sum of MaxSim operations. ColBERT introduced a denoising mechanism that filters out punctuation and stopword embeddings from the document representation, reducing index size by approximately 50% without degrading retrieval quality.

Storing full-precision token embeddings for every document can be memory-intensive. ColBERTv2 addresses this by using a residual compression technique. Token embeddings are clustered, and only the centroid IDs and residual vectors are stored. The residuals are further compressed using product quantization. This reduces the index footprint by 6-10x compared to storing full embeddings, making billion-document retrieval feasible on a single machine while preserving the fine-grained matching capability of late interaction.

An approach that represents a text passage as multiple dense vectors—typically one per token—rather than a single pooled vector. This enables late interaction scoring where query and document tokens interact at the final stage. Key benefits include:

• Preserves word-level semantic nuances lost in single-vector pooling
• Enables granular matching between specific query terms and document regions
• Supports efficient pre-computation of document token embeddings for low-latency retrieval

The core scoring function in ColBERT-style late interaction. For each query token embedding, it finds the maximum cosine similarity with any document token embedding, then sums these maximum values across all query tokens. This allows each query term to match the most relevant part of the document independently, capturing localized relevance signals without full cross-attention.

A training process where a computationally expensive cross-encoder (which processes query-document pairs with full attention) teaches a faster bi-encoder or late interaction model to improve retrieval accuracy. The student model learns to mimic the teacher's relevance distributions, achieving near-cross-encoder quality with bi-encoder speed. Critical for training high-performance ColBERT variants.

A retrieval setting where queries and documents are processed by separate, independently parameterized encoders. In late interaction, this asymmetry extends to the scoring phase: document token embeddings are pre-computed and indexed offline, while query token embeddings are computed at query time. This enables sub-second retrieval over millions of documents without recomputing document representations.

An evaluation metric measuring the proportion of relevant documents retrieved within the top-K results. For late interaction models, Recall@K is critical because the token-level scoring often surfaces relevant passages that single-vector bi-encoders miss. Typical benchmarks target Recall@100 or Recall@1000 to assess retrieval coverage before re-ranking stages.