Poly-Encoder

Unlike a bi-encoder that compresses the entire query into a single dense vector, a poly-encoder encodes the query into m distinct context vectors (or codes). These codes are learned via an attention mechanism over the query tokens, allowing the model to capture different semantic facets or intents within a single query. This multi-vector representation enables a more nuanced comparison with candidate documents without the computational cost of full cross-attention.

Poly-encoders perform a lightweight form of cross-attention at scoring time. For each candidate document, the model computes an attention score between the document's single vector representation and each of the query's m context codes. This produces a final query vector as a weighted sum of the codes, tailored to the specific document. This mechanism captures more fine-grained relevance signals than a static dot product while remaining orders of magnitude faster than a full cross-encoder.

The architecture is defined by its linear scaling properties:

• Bi-Encoder: O(1) interaction per document (single dot product).
• Poly-Encoder: O(m) interaction per document, where m is the number of context codes (typically 16-64).
• Cross-Encoder: O(n²) interaction per document (full self-attention over concatenated sequences). This makes poly-encoders ideal for re-ranking top-k candidates from a bi-encoder retrieval stage.

Poly-encoders are trained using a contrastive learning objective. The model learns to maximize the similarity score between a query and its relevant document while minimizing scores for negative samples. Hard negative mining is critical here; the model is exposed to documents that are superficially similar but ultimately irrelevant, forcing the context codes to learn discriminative features. The training data typically consists of (query, positive_document, negative_documents) triplets.

While both are late interaction models, they differ fundamentally in granularity:

• ColBERT stores all token-level embeddings for documents and performs a MaxSim operation between every query token and every document token. This requires significant storage.
• Poly-Encoder stores a single vector per document and compares it against a small set of query codes. This drastically reduces the index footprint and retrieval latency, making it more practical for large-scale production systems where storage is a constraint.

The standard deployment architecture is a two-stage retrieval cascade:

1. Candidate Generation: A bi-encoder performs fast approximate nearest neighbor (ANN) search over millions of documents to retrieve the top-100 or top-1000 candidates.
2. Re-ranking: A poly-encoder scores only these candidates using its attentive pooling mechanism to produce the final ranked list. This pattern captures the semantic recall of dense retrieval with the precision of a more expressive scorer.

The fastest retrieval architecture. Encodes queries and documents independently into single dense vectors. Similarity is computed via a simple dot product or cosine similarity, enabling pre-computation of document embeddings and sub-linear search via Approximate Nearest Neighbor (ANN) indexes. This independence, however, means there is no token-level interaction, limiting its ability to model fine-grained relevance.

The most precise architecture. Processes the query and document as a single concatenated sequence through full self-attention. This allows for deep, token-level interaction, generating highly accurate relevance scores. The computational cost is prohibitive for large-scale retrieval, as it requires a full forward pass for every query-document pair, making it suitable only for a final re-ranking stage on a small candidate set.

A balanced paradigm that delays interaction. Queries and documents are encoded independently into token-level embeddings. Relevance is scored via a MaxSim operation: each query token embedding finds its maximum cosine similarity with any document token embedding, and these maximums are summed. This preserves token-level expressiveness while allowing document embeddings to be pre-computed and indexed for efficient retrieval.

A computational middle ground. The query is encoded into m global context vectors (where m is small, e.g., 64). These vectors attend to the document's token embeddings to form a single, aggregated representation. The final score is a dot product between this aggregated document vector and a final query vector. This captures more interaction than a bi-encoder but is far cheaper than a cross-encoder's full attention.

The standard production architecture where Poly-Encoders excel. A fast retriever (e.g., a Bi-Encoder with an ANN index) selects a candidate set of ~100-1000 documents from a corpus of millions. A more powerful re-ranker (like a Poly-Encoder or Cross-Encoder) then scores this small set to produce the final ordering. This cascade balances the recall of dense retrieval with the precision of deep interaction.

The fundamental operation distinguishing these architectures. Self-attention (used in Cross-Encoders) allows every token in a concatenated query-document pair to attend to every other token. Cross-attention (used in Poly-Encoders) allows the query's context vectors to attend to the document's tokens, but not vice-versa. This asymmetric flow is the key to the Poly-Encoder's computational savings.