Candidate Generation

The primary objective is to ensure the true entity is present in the candidate set, even at the cost of including false positives. A recall of >95% is typically targeted. The downstream candidate ranking stage handles precision.

• Trades specificity for comprehensiveness
• Often uses approximate methods to avoid missing rare concepts
• Failure here is catastrophic; a missed entity cannot be recovered later

Candidate generation must execute in milliseconds against knowledge bases containing millions of entities. This is achieved through indexing structures that avoid exhaustive comparison.

• Inverted indices for lexical matching
• Approximate Nearest Neighbor (ANN) graphs for dense vector search
• Pre-computed alias tables mapping surface forms directly to concept IDs

Modern pipelines combine BM25 sparse retrieval with dense bi-encoder similarity to capture both exact terminology and semantic meaning.

• Lexical: Matches 'myocardial infarction' to entries containing that exact phrase
• Dense: Matches 'heart attack' to 'myocardial infarction' via embedding proximity
• Hybrid scoring merges results from both pathways for robust coverage

Knowledge bases are pre-processed to generate extensive surface form dictionaries that map common clinical shorthand and variants to canonical IDs.

• 'HTN' → Hypertension
• 'MI' → Myocardial Infarction
• Abbreviation expansion and lexical variant generation are critical pre-retrieval steps
• Reduces reliance on computationally expensive semantic matching at query time

A hard constraint applied during retrieval to restrict candidates to a specific UMLS Semantic Type. If a mention is identified as a medication, only entities typed as 'Pharmacologic Substance' or 'Clinical Drug' are considered.

• Dramatically reduces the candidate pool size
• Prevents cross-domain false positives (e.g., linking a drug name to a disease concept)
• Requires accurate upstream mention type classification

The retrieval stage must be designed to facilitate NIL prediction—identifying when a mention has no valid target in the knowledge base. This is often handled by setting a minimum similarity threshold.

• If no candidate exceeds the threshold, the mention is flagged as unlinkable
• Prevents forced grounding of novel or misspelled terms
• Critical for maintaining clinical data integrity in production systems

The final stage of entity linking where a more computationally intensive model scores and orders the generated candidates to select the single best match. Unlike candidate generation's focus on high recall with fast, approximate methods, ranking applies cross-encoders or deep transformer models to jointly encode the mention and each candidate, producing a high-fidelity relevance score. This stage handles the nuanced disambiguation that coarse retrieval cannot resolve.

A dual-tower neural network that independently encodes a text mention and a knowledge base entity into dense vectors for efficient, scalable semantic similarity search. This architecture is the backbone of modern candidate generation:

• Mention Encoder: Processes clinical text spans into fixed-length embeddings
• Entity Encoder: Encodes canonical knowledge base entries offline
• Similarity: Computed via dot product or cosine distance between the two vectors
• Indexing: Entity embeddings are pre-computed and stored in a vector index for sub-linear retrieval

A robust, bag-of-words-based ranking function used as a strong lexical baseline for generating candidate entities. BM25 excels at exact and near-exact string matching, making it highly effective for clinical terms with standardized nomenclature. It serves as a complementary signal to dense retrieval:

• Term Frequency: Rewards repeated term overlap between mention and entity description
• Inverse Document Frequency: Down-weights common tokens like 'syndrome' or 'acute'
• Document Length Normalization: Prevents bias toward longer entity descriptions

An indexing algorithm that trades a small amount of accuracy for a massive gain in speed when searching for the closest dense vectors in a high-dimensional entity embedding space. ANN is essential for scaling candidate generation to large ontologies like UMLS, which contains millions of concepts. Common implementations include FAISS (Facebook AI Similarity Search) and ScaNN (Scalable Nearest Neighbors), which use techniques like product quantization and hierarchical clustering to achieve sub-millisecond retrieval.

A constraint applied during candidate retrieval that restricts potential matches to entities belonging to a specific UMLS Semantic Type. For example, if a mention is known to be a medication, candidates are filtered to only 'Pharmacologic Substance' or 'Clinical Drug' types. This dramatically reduces the candidate pool and prevents cross-domain false positives:

• Type Detection: Often performed by a preceding NER model
• Filtering Granularity: Can use broad categories (Disorders, Chemicals & Drugs) or fine-grained types
• Fallback: If no candidates match the type constraint, the filter is relaxed to ensure recall

A self-supervised training paradigm that learns representations by pulling positive mention-entity pairs together and pushing negative pairs apart in vector space. This is the dominant training strategy for the dense retrievers used in candidate generation:

• Positive Pairs: A clinical mention and its correct UMLS concept
• Negative Pairs: The same mention paired with randomly sampled or hard-mined incorrect entities
• Loss Function: Typically InfoNCE or triplet loss, optimizing for a representation space where relevant entities are nearest neighbors
• Hard Negative Mining: Selects highly confusable but incorrect candidates to sharpen the embedding space