Bi-Encoder Scoring

The defining characteristic of a bi-encoder is its two separate, non-interacting neural networks. The query encoder processes the user's search string, while the document encoder processes the corpus text. Crucially, these towers do not share information during encoding. This independence means the document vector is a pure function of the document itself, with no cross-attention to the query. This is the fundamental trade-off: latency for fidelity.

Because the document encoding is independent of the query, every document in the corpus can be converted into a dense vector offline, before any search is performed. This is the engine of bi-encoder efficiency. The computationally expensive forward pass through a transformer model happens once per document during indexing. At query time, only the short query string needs to be encoded, reducing the real-time compute burden to a fraction of what a cross-encoder requires.

Retrieval is performed using ANN algorithms like HNSW or FAISS, not brute-force comparison. The query vector is used to probe a pre-built index of document vectors to find the k-nearest neighbors in high-dimensional space. This operation is typically measured in milliseconds, even for corpora containing billions of documents. The cosine similarity or dot product between the query and document vectors serves as the relevance score.

Bi-encoders are typically trained using a contrastive loss function with in-batch negatives. The model learns to pull the vector representations of a relevant query-document pair closer together while pushing irrelevant pairs apart. This training paradigm forces the model to distill all the semantic content of a long document into a single, fixed-size vector that must be broadly useful for a wide variety of potential future queries, a process known as late interaction.

The query and document towers can be configured in two ways. In a tied architecture, both towers share the exact same weights, often using a model like Sentence-BERT. This works well when queries and documents have similar linguistic structures. In an untied architecture, the towers are initialized from the same base model but fine-tuned separately, allowing the document encoder to specialize in processing long-form, structured text while the query encoder focuses on short, often poorly formed questions.

A transformer encoder outputs a vector for every input token, but ANN search requires a single, fixed-length vector per sequence. The pooling strategy determines how token-level embeddings are aggregated. Common methods include taking the [CLS] token embedding, calculating the mean of all token vectors, or using a more sophisticated attention-weighted pooling layer. The choice of pooling has a significant impact on the quality of the resulting semantic representation.

The primary alternative to bi-encoder scoring. A cross-encoder processes the query and document simultaneously through a transformer model, allowing full token-level attention between them. This yields significantly higher relevance accuracy but is computationally prohibitive for large-scale retrieval. It is almost exclusively used as a re-ranking step on a small candidate set (e.g., top 100) pre-filtered by a bi-encoder.

Bi-encoders are the core mechanism enabling dense retrieval. By encoding queries and documents into a shared dense vector space, they allow semantic matching via Approximate Nearest Neighbor (ANN) search. This contrasts with sparse retrieval (BM25), which relies on exact lexical overlap. Dense retrieval excels at finding semantically relevant documents even when they share no keywords with the query.

A variant where the query encoder and document encoder have different architectures or capacities. A common pattern uses a lightweight, fast encoder for queries (to minimize user-facing latency) and a larger, more powerful encoder for documents (to maximize indexing quality). This asymmetry optimizes the speed/accuracy trade-off without changing the fundamental bi-encoder independence principle.

A hybrid paradigm exemplified by ColBERT that bridges bi-encoders and cross-encoders. Like bi-encoders, it pre-computes document token embeddings independently. However, at query time, it performs a lightweight, fine-grained interaction between query and document token embeddings via a MaxSim operation. This captures more nuance than pure bi-encoders while remaining far more scalable than cross-encoders.

The dominant training paradigm for bi-encoders. The model is trained to pull the embeddings of a query and its relevant (positive) document closer together while pushing apart the embeddings of irrelevant (negative) documents. Techniques like in-batch negatives and hard negative mining are critical for learning a high-quality embedding space that generalizes well to unseen queries.

Bi-encoder scoring is only viable at scale when paired with Approximate Nearest Neighbor indexes like HNSW or IVF-PQ. These data structures trade a small amount of recall for massive speed gains, enabling sub-millisecond similarity search over billions of document vectors. The bi-encoder's pre-computed document embeddings are loaded into the ANN index, while the query embedding is computed on the fly.