Bi-Encoder

The defining architectural trait: queries and documents are processed by separate, non-interacting encoders (often identical siamese networks). This allows document embeddings to be pre-computed and indexed offline, while only the query vector is generated at runtime. Unlike a cross-encoder, there is no cross-attention between the query and document tokens during encoding.

All interaction between query and document is deferred to a lightweight scoring function applied to the final pooled embeddings. Common operations include:

• Cosine Similarity: Measures the angle between vectors.
• Dot Product: A fast, unnormalized similarity measure.
• Euclidean Distance (L2): Measures absolute distance in vector space. This late interaction enables the use of highly optimized Approximate Nearest Neighbor (ANN) indexes.

Bi-encoders achieve sub-linear retrieval latency relative to corpus size. The heavy lifting (document encoding) is an offline, one-time cost. Online inference requires only a single forward pass for the query and an ANN lookup. This makes them the standard choice for the candidate generation (first stage) of a retrieval pipeline, scaling to billions of documents.

The primary trade-off: compressing an entire document into a single, fixed-size dense vector creates an information bottleneck. Fine-grained token-level relationships are lost. This limits the ability to precisely match detailed queries or capture exact keyword overlap, which is why bi-encoders are typically paired with a cross-encoder re-ranker for final precision.

Bi-encoders are commonly trained using contrastive learning with in-batch negatives. The model learns to pull embeddings of relevant query-document pairs together while pushing apart non-relevant pairs. Key training techniques include:

• Hard Negative Mining: Selecting challenging negative samples to improve discriminative power.
• Knowledge Distillation: Training a fast bi-encoder student to mimic the relevance scores of a slower cross-encoder teacher.

A foundational implementation that fine-tunes BERT using siamese and triplet network structures to produce semantically meaningful sentence embeddings. SBERT introduced a pooling operation (mean-pooling) to the standard BERT output, enabling direct cosine similarity comparison and making bi-encoder inference practical for semantic textual similarity tasks.

The re-ranking counterpart to the bi-encoder. Instead of encoding query and document separately, a cross-encoder processes the concatenated [CLS] query [SEP] document [SEP] pair through full cross-attention, allowing every query token to attend to every document token. This deep interaction yields far more precise relevance scores but is computationally prohibitive for full-corpus retrieval, making it ideal for re-scoring the top-k candidates from a bi-encoder.

The cascade architecture where a bi-encoder operates as the retriever and a cross-encoder acts as the re-ranker. Stage one uses fast approximate nearest neighbor search over dense vectors to fetch hundreds of candidates from millions. Stage two applies the computationally intensive cross-encoder to re-score only those top candidates, combining the speed of dense retrieval with the precision of deep interaction.

The primary scoring function used by bi-encoders to compare query and document embeddings. It measures the cosine of the angle between two vectors, producing a value between -1 and 1. Because both embeddings are typically L2-normalized to unit length, cosine similarity reduces to a simple dot product, enabling blazing-fast computation on GPU hardware via matrix multiplication.

A late interaction model that bridges bi-encoders and cross-encoders. ColBERT encodes query and document tokens independently like a bi-encoder, but stores per-token embeddings rather than a single vector. At scoring time, it computes a MaxSim operation: each query token finds its most similar document token, and these maximum similarities are summed. This preserves token-level detail without the full quadratic cost of cross-attention.

An unsupervised algorithm for merging multiple ranked lists into a single consensus ranking. RRF scores each document by summing 1 / (k + rank) across all input lists, where k is a constant (typically 60) that mitigates the impact of high ranks from outlier systems. It requires no training data or score calibration, making it a robust method for combining bi-encoder results with BM25 lexical rankings.

A training technique critical to bi-encoder performance. Standard in-batch negatives are often too easy, leading to poor discriminative power. Hard negative mining selects negative passages that have high BM25 or embedding similarity to the query but are not relevant. Training with these challenging distractors forces the bi-encoder to learn finer-grained semantic distinctions, significantly boosting top-k retrieval accuracy.