Bi-Encoder

The defining characteristic of a Bi-Encoder is its two separate encoders that process the mention and the entity independently. Unlike a Cross-Encoder, there is no cross-attention between the two inputs. The mention encoder (often a BERT-based model) maps the textual context to a dense vector, while the entity encoder maps the entity's title and description to a vector in the same shared embedding space. This independence is what enables pre-computation of all entity embeddings.

Because the entity encoder operates independently, all candidate entity embeddings can be computed offline and indexed once. At inference time, only the mention needs to be encoded. This is the key to Bi-Encoder scalability:

• Entity embeddings are stored in a vector index like FAISS
• Indexes can hold millions or billions of entities
• Retrieval is reduced to a fast approximate nearest neighbor (ANN) search
• Index updates require only re-encoding new or changed entities

Relevance between a mention and a candidate entity is computed as the dot product (or cosine similarity) between their respective embeddings. This is a lightweight, linear operation that scales efficiently:

• score(m, e) = embedding(m) · embedding(e)
• Enables fast top-k retrieval from massive indexes
• Supports batching for processing multiple mentions simultaneously
• The score represents semantic similarity in the shared latent space

Bi-Encoders are trained using contrastive learning, typically with an in-batch negatives strategy. The model learns to pull the embedding of a mention close to its correct entity while pushing it away from all other entities in the batch:

• Positive pairs: (mention, correct entity)
• Negative pairs: (mention, all other entities in the batch)
• Loss function: usually cross-entropy over the dot product scores
• Hard negative mining improves discrimination of similar entities

In production entity linking systems like BLINK, the Bi-Encoder serves as the first-stage retriever in a two-stage pipeline:

• Stage 1 (Bi-Encoder): Fast recall — retrieves top-k candidates (e.g., k=100) from millions
• Stage 2 (Cross-Encoder): Precision re-ranking — scores each candidate with full cross-attention
• This architecture balances the speed of Bi-Encoders with the accuracy of Cross-Encoders
• The Bi-Encoder's recall quality is critical; missed candidates cannot be recovered

Both the mention encoder and entity encoder must map their inputs into a common vector space where geometric proximity corresponds to semantic relatedness. Key design considerations:

• Both encoders typically share the same base architecture (e.g., BERT-base)
• The final representation is usually the [CLS] token embedding or mean pooling of all tokens
• Dimensionality is fixed (commonly 768 or 1024)
• The space is normalized so cosine similarity is meaningful

The foundational training paradigm for Bi-Encoders. The model is trained to pull together vector representations of matching mention-entity pairs and push apart non-matching pairs in the embedding space. This is typically achieved using loss functions like InfoNCE or triplet loss, which operate on in-batch negatives. The result is a dense vector space where semantic similarity is directly proportional to dot product or cosine distance, enabling fast nearest-neighbor retrieval.

The Bi-Encoder's essential counterpart in a two-stage retrieval pipeline. While the Bi-Encoder scores all candidates independently for speed, the Cross-Encoder processes the concatenated mention and candidate text through full transformer self-attention. This joint encoding captures fine-grained token-level interactions, producing a high-precision relevance score. The standard pattern: Bi-Encoder retrieves the top-k candidates (e.g., k=100), and the Cross-Encoder reranks them for the final prediction.

The algorithmic backbone enabling Bi-Encoder scalability. Exact nearest-neighbor search over millions of entity embeddings is computationally prohibitive. ANN libraries like FAISS or ScaNN use techniques such as product quantization and hierarchical navigable small world graphs to retrieve approximate top-k results in sub-millisecond time. This allows a Bi-Encoder to query a knowledge base with millions of entities with minimal latency.

The dense vector representation of a knowledge base entity that the Bi-Encoder produces. Each entity is encoded independently from its title and description text into a fixed-size vector (e.g., 768 dimensions for BERT-base). These embeddings are pre-computed and indexed once, then stored in an ANN index. At inference time, only the mention is encoded, and its vector is compared against the static entity index via dot product.