Concept Normalization

Resolves diverse surface forms to a single concept. Clinical text is highly variable; 'heart attack', 'myocardial infarction', and 'MI' all map to the same Concept Unique Identifier (CUI). This process handles:

• Acronyms and Abbreviations: 'HTN' to Hypertensive disease
• Morphological Variants: 'painful' vs. 'pain'
• Synonymy: 'renal' vs. 'kidney'
• Trade vs. Generic Names: 'Tylenol' to Acetaminophen

Uses context to select the correct CUI for polysemous terms. The term 'cold' could map to Common Cold (disease) or Cold Temperature (physical attribute). Normalization algorithms analyze surrounding tokens (e.g., 'caught a cold' vs. 'applied cold compress') and leverage contextual embeddings to achieve high-accuracy disambiguation.

Links extracted entities to specific source vocabularies within the UMLS Metathesaurus. A single CUI aggregates identifiers from:

• SNOMED CT (Clinical findings)
• RxNorm (Medications)
• LOINC (Lab tests)
• ICD-10-CM (Billing codes) This allows an NLP pipeline to output a SNOMED code for a diagnosis and an RxNorm code for a drug simultaneously.

Employs fuzzy matching when exact dictionary lookups fail due to typos or OCR errors. Techniques include:

• Levenshtein Distance: Corrects minor character insertions/deletions
• Jaccard Index: Measures overlap of character n-grams
• Phonetic Algorithms: Matches homophones (e.g., 'furosemide' vs. 'frusemide') These are often used as a high-recall fallback layer behind a primary neural model.

Modern systems use bi-encoder neural networks to generate dense vector embeddings for both the clinical mention and all ontology concepts. Normalization is performed by a nearest-neighbor search in this vector space, drastically reducing the candidate list before a final cross-encoder re-ranks the top-k concepts for maximum precision.

Applies hard constraints based on UMLS Semantic Types to prevent category errors. If the NER model tags a span as a 'Drug', the normalization step filters out any candidate CUIs that do not belong to the Pharmacologic Substance or Clinical Drug semantic groups, ensuring a 'medication' is never normalized to a 'disease' concept.

The downstream sibling task that resolves a textual mention to its single, unambiguous identity in a knowledge base. While concept normalization maps to a CUI, disambiguation distinguishes between identical names for different concepts—for example, determining whether 'cold' refers to common cold (C0009443) or cold temperature (C0009264) based on context.

The end-to-end architecture that combines NER with concept normalization. The pipeline typically executes in sequence:

• Mention Detection: Identify entity spans in text
• Candidate Generation: Retrieve possible CUI matches from the ontology
• Candidate Ranking: Score and select the best match using contextual embeddings This modular design allows each stage to be optimized independently.

A critical preprocessing step that resolves surface-form variability before ontology mapping. Techniques include:

• Expansion: Converting 'HTN' to 'hypertension'
• Stemming/Lemmatization: Reducing 'fractures' to 'fracture'
• Case normalization: Standardizing capitalization Without this step, exact-match lookups against the UMLS fail for common clinical abbreviations and morphological variants.

A constraint mechanism that restricts normalization candidates to specific UMLS Semantic Types. For example, when normalizing a medication mention, the system filters to only Pharmacologic Substance (T121) or Clinical Drug (T200) types. This dramatically reduces false-positive mappings by eliminating semantically impossible candidates—preventing a disease name from being mapped to a drug CUI.

Algorithms that handle lexical variation and typos when exact dictionary lookups fail. Common approaches include:

• Levenshtein distance: Edit-distance thresholding
• TF-IDF vectorization: N-gram overlap scoring
• Phonetic algorithms: Soundex for homophone errors These techniques are essential for real-world clinical text, where misspellings like 'metformine' must still normalize to metformin (C0025598).