Contrastive Learning

The engine of contrastive learning relies on defining positive pairs—two views of the same underlying concept. In clinical entity linking, a positive pair consists of a mention (e.g., 'heart attack') and its canonical entity (e.g., UMLS C0027051: Myocardial Infarction). These pairs can be generated through:

• Synonym substitution from ontologies like SNOMED CT
• Dropout augmentation where the same input passes through a model twice
• Contextual paraphrasing of the surrounding clinical text The model learns to map these distinct surface forms to identical vector coordinates.

The discriminative power of contrastive learning comes from negative pairs—entities that the mention should not link to. Effective negative sampling is critical:

• In-batch negatives: Other entities within the same training mini-batch serve as negatives, enabling efficient computation
• Hard negative mining: Selecting highly confusable entities (e.g., 'myocardial infarction' vs. 'myocardial ischemia') that share semantic overlap with the mention
• Random negatives: Uniformly sampled entities from the knowledge base to provide broad repulsive force Without hard negatives, the model fails to learn fine-grained disambiguation.

The standard training objective is InfoNCE (Noise Contrastive Estimation), a categorical cross-entropy loss over similarity scores. For a given mention anchor m and a positive entity e+:

• Compute cosine similarity between m and all candidates in the batch
• Apply a temperature parameter (τ) to control the concentration of the distribution
• Lower τ creates a sharper distribution, penalizing hard negatives more aggressively
• The loss minimizes -log(exp(sim(m, e+)/τ) / Σ exp(sim(m, ei)/τ)) This formulation is a lower bound on mutual information between the mention and entity representations.

Contrastive learning in entity linking typically uses a dual-tower (bi-encoder) architecture:

• Mention Encoder: A transformer (e.g., SapBERT, PubMedBERT) encodes the clinical mention and its surrounding context into a dense vector
• Entity Encoder: A separate or shared transformer encodes the entity's canonical name, synonyms, and description from the knowledge base
• Both towers project into a shared L2-normalized embedding space where inner product equals cosine similarity
• At inference, entity embeddings are pre-computed and indexed using FAISS for fast approximate nearest neighbor search

Beyond self-supervised instance discrimination, supervised contrastive learning leverages label information to structure the embedding space more effectively:

• Multiple mentions that link to the same UMLS concept are treated as positives for each other, not just their entity description
• This pulls together diverse surface forms like 'MI', 'heart attack', and 'myocardial infarct' into a tight cluster
• The loss function generalizes InfoNCE to handle multiple positives per anchor
• This approach is particularly effective for high-variance clinical language where the same condition has dozens of synonymous expressions

Momentum Contrast addresses the challenge of maintaining a large and consistent dictionary of negative samples without massive batch sizes:

• A momentum encoder (slowly updating copy of the main encoder) maintains a dynamic queue of recent entity embeddings
• The queue acts as a large negative sample bank, decoupling batch size from negative set size
• The momentum update rule θ_k ← m·θ_k + (1-m)·θ_q with a high momentum coefficient (e.g., 0.999) ensures consistency
• This is critical for biomedical entity linking where the knowledge base may contain millions of concepts

A dual-tower neural network that independently encodes a text mention and a knowledge base entity into dense vectors for efficient, scalable semantic similarity search. In contrastive learning, the bi-encoder forms the backbone that generates the representations to be pulled together or pushed apart.

• Mention Encoder: A transformer (e.g., PubMedBERT) that contextualizes the ambiguous clinical span.
• Entity Encoder: A separate tower that encodes the canonical entity's title and description.
• Efficiency: Enables pre-computation of all entity embeddings for fast Approximate Nearest Neighbor Search at inference time.

A contrastive learning strategy that selects highly confusable but incorrect candidate entities during training to improve a model's disambiguation capability. Standard random negatives are too easy; hard negatives force the model to learn fine-grained distinctions.

• In-Batch Negatives: Treating other entities in the same mini-batch as negative examples.
• BM25 Hard Negatives: Using lexical retrieval to find top-scoring but incorrect candidates.
• Cross-Encoder Filtering: Employing a reranker to identify the most plausible false positives for the training batch.

The canonical contrastive loss function (Noise Contrastive Estimation) that maximizes the mutual information between positive mention-entity pairs while minimizing it for negatives. It is the mathematical engine driving representation learning in models like SapBERT.

• Temperature Parameter: A hyperparameter that controls the concentration of the distribution; lower values create sharper distinctions between positives and hard negatives.
• Symmetric Formulation: The loss is often computed bidirectionally, pulling the mention to the entity and the entity to the mention.

A pre-trained biomedical language model optimized for entity linking by aligning synonymous concepts from the UMLS Metathesaurus into a shared dense vector space. It is trained using a self-supervised contrastive objective where synonymous concepts are positives and random concepts are negatives.

• Synonym Marginalization: Aggregates multiple synonymous terms for a single CUI to form a robust centroid representation.
• Cross-Lingual Alignment: Extends contrastive learning to map concepts across languages, enabling zero-shot multilingual entity linking.

A memory-efficient contrastive training technique that repurposes other positive examples within the same mini-batch as negative samples. This avoids the computational cost of explicitly mining negatives from the full knowledge base.

• Scalability: The effective number of negatives scales linearly with batch size, making large batch sizes critical for performance.
• Gradient Accumulation: Used to simulate large batches on hardware with limited GPU memory.
• Bias Correction: Techniques to correct for the sampling bias introduced when true positives accidentally appear as in-batch negatives.

A neural architecture that processes a mention-candidate pair jointly through a transformer to produce a high-fidelity relevance score. While too slow for full retrieval, it serves as the precise scoring function during candidate ranking, often trained with a contrastive or cross-entropy objective.

• Joint Encoding: Allows full cross-attention between the mention tokens and entity description tokens.
• Two-Stage Pipeline: A bi-encoder retrieves top-k candidates, and the cross-encoder reranks them.
• Distillation: The cross-encoder's knowledge can be distilled back into a bi-encoder for faster inference.