MaxSim

For each query token embedding, MaxSim identifies the single document token embedding with the highest cosine similarity and uses that maximum value. This allows the model to find the best alignment for each query term independently, capturing soft term matches where a query concept maps to its most semantically related document token. Unlike exact keyword matching, this handles synonyms, paraphrases, and morphological variations naturally.

After computing the maximum similarity for each query token, MaxSim sums these individual maximums to produce a single relevance score. This summation assumes each query token contributes independently to relevance. The final score represents the aggregate alignment quality across all query terms. A query with more tokens will naturally produce a higher sum, which is why query length normalization is often applied in practice to ensure fair comparison across queries of different lengths.

MaxSim operates on pre-computed document token embeddings, avoiding the quadratic complexity of full cross-attention. The operation is:

• O(|q| × |d|) per document, where |q| is query length and |d| is document length
• Highly parallelizable on GPUs using matrix multiplication
• Compatible with approximate nearest neighbor (ANN) indexes for fast candidate retrieval This efficiency enables MaxSim to re-rank hundreds or thousands of candidates in milliseconds, making it practical for production search systems.

MaxSim occupies a middle ground in the efficiency-expressiveness spectrum:

• Bi-Encoder (Dot Product): Fastest, but only captures global semantic similarity with no token-level interaction
• MaxSim (Late Interaction): Balances speed and precision by delaying interaction to the scoring stage while keeping embeddings independently computable
• Cross-Encoder (Full Attention): Most expressive with complete query-document attention, but requires joint computation for every pair, making it too slow for first-stage retrieval

In the ColBERT model, MaxSim is the final scoring step after both query and document are independently encoded through BERT. The process:

• Query tokens are encoded with special [Q] tokens prepended to produce contextualized embeddings
• Document tokens are encoded and stored in an index
• MaxSim computes the sum of maximum cosine similarities
• A query augmentation mask filters out punctuation and stopword tokens
• The resulting score is used to rank the top-k candidates retrieved by ANN search

Raw MaxSim scores require careful handling for consistent ranking:

• Cosine similarity naturally bounds individual token similarities to [-1, 1]
• Query length normalization divides the sum by |q| to prevent longer queries from receiving artificially higher scores
• Score calibration using temperature scaling or Platt scaling can map raw sums to well-calibrated relevance probabilities
• Without normalization, MaxSim scores are not directly comparable across queries of different lengths or tokenization granularities

A retrieval architecture that delays query-document interaction to a final, lightweight scoring step. Unlike Cross-Encoders that perform full self-attention over the concatenated query-document pair, late interaction models pre-compute document token embeddings offline. At query time, relevance is computed via MaxSim, summing the maximum cosine similarity between each query token and its best-matching document token. This balances the efficiency of Bi-Encoders with the expressiveness of token-level matching.

The Contextualized Late Interaction over BERT model that operationalizes MaxSim. ColBERT encodes queries into a set of m fixed-size token embeddings and documents into n token embeddings using a shared BERT backbone. It filters document tokens via a [CLS] query augmentation step before computing the MaxSim sum. The architecture supports end-to-end top-k retrieval by coupling MaxSim re-ranking with a vector-similarity index over document token embeddings.

MaxSim occupies a precision-latency sweet spot between these two extremes:

• Bi-Encoder: Encodes query and document into single vectors; dot-product scoring is O(1) but loses fine-grained token alignment.
• Cross-Encoder: Full self-attention over the concatenated pair captures exact lexical match but is O(n²) and cannot pre-compute.
• MaxSim (Late Interaction): Pre-computes document token embeddings, scores via cheap cosine similarity, capturing soft term matches without full cross-attention cost.

Models using MaxSim are trained with contrastive representation learning objectives. The loss function pulls the token embeddings of relevant query-document pairs closer while pushing apart embeddings of negative samples. Hard negative mining is critical: negatives are sampled from high-scoring BM25 or Bi-Encoder results that are not relevant, forcing the model to learn fine-grained discriminative token alignments. In-batch negatives provide computationally efficient negative sampling.

MaxSim typically operates as the second-stage re-ranker in a multi-stage retrieval pipeline:

1. Stage 1: BM25 or Bi-Encoder retrieves a broad candidate set (e.g., top-1000).
2. Stage 2: ColBERT with MaxSim re-ranks candidates to top-k (e.g., 100).
3. Stage 3 (optional): A full Cross-Encoder scores the final top-10 for maximum precision. This cascading architecture optimizes the latency-relevance trade-off for production search systems.

A Cross-Encoder teacher can transfer its full-attention scoring distribution to a MaxSim-based student model. The distillation process uses KL divergence loss between the teacher's softmax-normalized relevance scores and the student's MaxSim output. This allows the efficient late interaction model to approximate the precision of the expensive Cross-Encoder while maintaining the ability to pre-compute document token embeddings for fast retrieval.