ColBERT

Unlike bi-encoders that compress an entire document into a single fixed-size vector, ColBERT encodes every token in a query and document into its own distinct contextual embedding. This preserves granular semantic signals—such as specific entities, attributes, and syntactic relationships—that are inevitably lost during aggressive embedding pooling operations like mean pooling or CLS token extraction. The result is a matrix of vectors for each text rather than a single point in space.

The core scoring mechanism of ColBERT is the MaxSim operation. For each query token embedding, the model identifies the document token embedding with the highest cosine similarity and sums these maximum scores across all query tokens. This 'late interaction' is computed at query time, after documents have already been indexed, allowing the system to leverage pre-computed document representations while still performing token-level relevance matching. The formula is: score = Σ max(cosine_sim(q_i, d_j)).

ColBERT's architecture decouples document encoding from query processing. All documents in a corpus can be encoded into their multi-vector representations offline and stored in a specialized index. The PLAID (Performance-optimized Late Interaction Driver) engine further accelerates retrieval by using a coarse centroid-based pruning step to eliminate irrelevant documents before performing the expensive MaxSim computation. This makes ColBERT viable for production-scale search over millions of documents.

ColBERT prepends a special [Q] token to queries and a [D] token to documents before encoding. These tokens act as task-specific markers that signal to the underlying transformer model whether it is processing a query or a document, allowing the shared encoder to produce representations optimized for their respective roles in the asymmetric matching process. This is a lightweight alternative to maintaining separate query and document encoders.

ColBERT is trained using a pairwise contrastive learning objective. For each query, the model is presented with a relevant (positive) passage and one or more irrelevant (negative) passages. The training loss maximizes the MaxSim score for the positive pair while minimizing it for negative pairs. Crucially, the model often employs hard negative mining—selecting negative passages that are deceptively similar to the query—to learn highly discriminative token-level representations.