Bi-Encoder

The core mechanism involves two distinct transformer towers that process queries and documents independently. There is no cross-attention between the input pair.

• Query Encoder: Processes the user's search string into a single dense vector.
• Document Encoder: Processes the target passage into a single dense vector.
• No Joint Computation: Unlike cross-encoders, the model never sees the query and document together during encoding.

The decoupled nature allows document embeddings to be generated offline and stored in a vector database.

• Offline Indexing: All document vectors can be pre-calculated before any user query arrives.
• Sub-Linear Search: At query time, only the query vector needs to be generated, which is then compared against the pre-indexed document vectors using Approximate Nearest Neighbor (ANN) algorithms.
• Massive Scalability: This enables searching billions of documents in milliseconds.

To convert the variable-length token outputs of a transformer into a single fixed-size sentence embedding, a pooling operation is required.

• Mean Pooling: Averages all token embeddings, often providing a robust general-purpose representation.
• CLS Token: Uses the embedding of the special classification token as the sentence representation.
• Max Pooling: Takes the maximum value over the time dimension for each feature.

Bi-encoders are typically trained using contrastive learning objectives to structure the embedding space.

• Positive Pairs: A query and its relevant document are pulled together in the vector space.
• Negative Pairs: Irrelevant documents (negatives) are pushed away.
• Hard Negatives: Mining deceptively similar but incorrect documents is critical for teaching the model fine-grained distinctions.

Bi-encoders sacrifice some interaction depth for speed, making them ideal as a first-stage retriever.

• Latency: Extremely fast due to pre-computation and ANN search.
• Expressiveness: Less precise than a cross-encoder because the query and document cannot attend to each other.
• Common Pattern: A bi-encoder retrieves the top-100 candidates, which are then re-ranked by a slower, more accurate cross-encoder.

The two towers can be configured with different weight-sharing strategies depending on the symmetry of the inputs.

• Tied Weights: The query and document encoders share the exact same parameters. Used when the input types are semantically identical (e.g., sentence-to-sentence similarity).
• Untied Weights: The towers are trained independently. This is crucial for asymmetric search where queries are short keywords and documents are long paragraphs.

The canonical use case. A query encoder converts the user's search string into a vector, while a document encoder has pre-indexed billions of passages. FAISS or HNSW indexes find the top-K results in sub-10ms latency. This architecture powers Google Search, Bing, and enterprise knowledge bases where lexical overlap fails. The key advantage: document embeddings are computed once and stored, making ingestion of new content a batch operation independent of query volume.

Bi-encoders power two-tower recommendation models used by YouTube, Netflix, and Spotify. The user tower encodes watch history, demographics, and context into a single embedding. The item tower encodes video metadata, audio features, or product descriptions. Candidate generation runs as an ANN search over millions of items in real-time. This architecture elegantly solves the cold-start problem: new items can be encoded immediately without retraining the user tower.

Bi-encoders excel at asymmetric search where queries are short and documents are long. Unlike cross-encoders that require quadratic attention over the concatenated pair, bi-encoders compress documents into a single fixed-size vector. This enables use cases like:

• Question answering over technical documentation
• Customer support ticket routing against knowledge base articles
• Legal e-discovery matching brief queries to case law corpora

The trade-off is lower precision than cross-encoders, which is why production systems often cascade: bi-encoder for candidate retrieval, cross-encoder for re-ranking.

Bi-encoders extend naturally to multi-modal architectures like CLIP. A text encoder and an image encoder are trained contrastively to map both modalities into a shared embedding space. This enables:

• Text-to-image search in digital asset management
• Image-to-text caption retrieval
• Zero-shot image classification by comparing image embeddings to text embeddings of class labels

The decoupled design means images can be indexed once and searched via natural language without reprocessing the visual corpus.

Because bi-encoders produce fixed-size embeddings, they enable efficient semantic clustering of large document collections. Use cases include:

• Near-deduplication of training data to prevent benchmark contamination
• Topic modeling by clustering embeddings and extracting representative terms
• Content moderation by flagging items semantically similar to known policy violations

Cosine similarity thresholds define cluster boundaries, and dimensionality reduction via PCA or UMAP projects embeddings for visual exploration of semantic landscapes.

Unlike the bi-encoder's independent processing, a cross-encoder feeds the query and document together through self-attention. This allows the model to capture fine-grained token-level interactions, producing highly accurate relevance scores. The trade-off is computational cost: cross-encoders cannot pre-compute document embeddings, making them impractical for large-scale retrieval. They are typically used as a re-ranking stage over bi-encoder candidates.

ColBERT represents a middle ground between bi-encoders and cross-encoders. It encodes queries and documents into multiple token-level vectors, storing them independently like a bi-encoder. At query time, it computes relevance via a MaxSim operation, comparing each query token to the most similar document token. This late interaction preserves more nuance than a single-vector bi-encoder while remaining far more scalable than a cross-encoder.

A training technique that produces embeddings where the first k dimensions remain a valid, semantically coherent representation for any k. This allows a single bi-encoder model to support multiple embedding dimensions without retraining. Developers can trade off accuracy for storage cost dynamically: use 768 dimensions for high-precision tasks and 64 dimensions for rapid coarse filtering, all from the same model.

A critical training strategy for bi-encoders. Standard contrastive learning uses random negatives, which are trivially easy to distinguish. Hard negative mining selects documents that are deceptively similar to the query but not relevant—such as documents sharing keywords but differing in intent. Training against these challenging examples forces the bi-encoder to learn more discriminative decision boundaries, significantly improving retrieval precision.

The classical alternative to dense bi-encoder retrieval. BM25 represents text using high-dimensional sparse vectors based on exact term frequency and inverse document frequency. While it excels at precise keyword matching and is highly interpretable, it suffers from the vocabulary mismatch problem: it cannot match semantically related terms like 'car' and 'automobile'. Modern systems often combine BM25 with bi-encoders in hybrid search.

Bi-encoders produce dense vectors, but finding the exact nearest neighbors in a billion-scale index is computationally prohibitive. ANN algorithms like HNSW and IVF-PQ trade a small amount of recall for massive speed gains. They construct navigable graph structures or quantized partitions to achieve sub-linear search complexity, making real-time bi-encoder retrieval feasible at scale.