Bi-Encoder Architecture

The defining characteristic of a bi-encoder is its two separate neural networks operating in parallel. One tower encodes the text mention (e.g., 'heart attack'), while the other independently encodes the knowledge base entity (e.g., 'Myocardial Infarction'). This decoupling allows the computationally expensive encoding of all entities in a corpus to be performed offline, once, as a pre-computation step. At inference time, only the query mention needs to be encoded, enabling real-time retrieval against millions of pre-indexed vectors.

Bi-encoders rely on ANN algorithms like HNSW or FAISS to trade a marginal amount of accuracy for orders-of-magnitude gains in search speed. The model outputs dense vectors where semantic similarity is measured via cosine similarity or dot product. Instead of comparing a query vector against every entity vector in the database (an O(N) operation), ANN indexes cluster vectors in a graph structure, enabling retrieval in O(log N) time. This is critical for clinical entity linking against ontologies like UMLS, which contain millions of concepts.

Bi-encoders are trained using contrastive loss functions that pull positive pairs together and push negative pairs apart in vector space. The key to a high-performing model is hard negative mining—selecting negative examples that are lexically similar but semantically distinct (e.g., 'hypertension' vs. 'hypotension'). Without hard negatives, the model learns trivial distinctions. Advanced training regimens use in-batch negatives and global hard negatives retrieved from the full index to teach the model fine-grained disambiguation, directly improving its ability to distinguish between confusable UMLS concepts.

A practical advantage of the bi-encoder is the ability to use asymmetric architectures for the two towers. The query encoder can be a lightweight, low-latency model optimized for CPU inference, while the document encoder can be a massive, high-precision transformer. This allows the heavy lifting of encoding a vast corpus of medical entities to be done offline with a powerful model, while the real-time query path remains fast and cost-effective. This separation of concerns is impossible in a cross-encoder architecture.

The output of a bi-encoder is a fixed-size dense vector, typically 768 or 1024 dimensions. This vector acts as a semantic fingerprint, where the spatial proximity of two vectors directly corresponds to conceptual similarity. Unlike sparse lexical methods like BM25, which rely on exact token overlap, bi-encoders map synonyms and paraphrases to the same region of vector space. For clinical text, this means 'elevated blood pressure' and 'hypertension' will have high cosine similarity, enabling robust retrieval even when terminology varies drastically between physician notes and formal ontology labels.

The operational workflow of a bi-encoder system is split into an offline indexing phase and an online query phase. During indexing, every entity in the target knowledge base (e.g., all SNOMED CT descriptions) is passed through the frozen document encoder to generate a static vector index. This index is then loaded into memory for the online phase. When a new clinical mention arrives, only the query encoder runs, and the resulting vector is used to search the pre-built index. This architecture decouples corpus size from query latency, making it the standard for production clinical NLP systems.

A high-precision neural architecture that processes a mention-candidate pair jointly through a transformer to produce a relevance score. Unlike bi-encoders, cross-encoders apply full self-attention across both sequences simultaneously, enabling rich token-level interactions.

• Role: Serves as the second-stage ranker in a two-stage retrieval pipeline
• Trade-off: 10-100x slower than bi-encoders but significantly more accurate
• Architecture: Typically fine-tuned BERT-style models with a classification head
• Example: A bi-encoder retrieves 100 candidates; the cross-encoder reranks them to select the correct SNOMED CT code

A dual-encoder framework trained to retrieve relevant knowledge base passages by maximizing the inner product of query and context embeddings. DPR uses in-batch negatives and hard negatives to learn discriminative representations.

• Training Objective: Contrastive loss that pulls positive pairs together and pushes negatives apart
• Indexing: Entity descriptions are pre-encoded and stored in a vector index for fast ANN search
• Clinical Adaptation: Entity descriptions from UMLS or SNOMED CT serve as the passage corpus
• Performance: Achieves high recall@k with k=100, making it ideal for candidate generation

An indexing algorithm that trades a small amount of accuracy for massive speed gains when searching for the closest dense vectors in high-dimensional embedding space. ANN is the retrieval backbone that makes bi-encoder inference scalable.

• Common Libraries: FAISS (Facebook), ScaNN (Google), Annoy (Spotify)
• Index Types: IVF (Inverted File), HNSW (Hierarchical Navigable Small World), PQ (Product Quantization)
• Latency Profile: Sub-millisecond search across millions of vectors
• Clinical Scale: Enables real-time linking against the full UMLS Metathesaurus of 4M+ concepts

A self-supervised training paradigm that learns representations by pulling positive mention-entity pairs together and pushing negative pairs apart in vector space. This is the foundational training strategy for modern bi-encoder entity linking models.

• Positive Pairs: A clinical mention and its correct UMLS CUI description
• Hard Negatives: Confusable entities like 'DM' (Diabetes Mellitus vs. Dermatomyositis) that share lexical overlap
• Loss Functions: InfoNCE, Triplet Loss, or Multiple Negatives Ranking Loss
• Biomedical Example: SapBERT uses contrastive learning to align synonymous concepts from different ontologies into a unified embedding space

The initial high-recall retrieval stage that uses fast, approximate methods to fetch a small set of plausible knowledge base entries for a given text mention. This stage determines the upper bound of the linking pipeline's accuracy.

• Bi-Encoder Role: Encodes the mention and performs ANN search over pre-indexed entity embeddings
• Hybrid Approaches: Combine dense retrieval with sparse lexical baselines like BM25 for robustness
• Constraints: Semantic type filtering restricts candidates to relevant UMLS categories (e.g., only 'Disease or Syndrome')
• Output: Typically 50-200 candidates passed to the ranking stage