Contextual Embedding

Unlike static embeddings like Word2Vec where the word 'cold' has a single, unchanging vector, a contextual model generates a unique embedding for 'cold' in 'the patient feels cold' versus 'the patient has a cold.' This dynamic generation is the core mechanism enabling disambiguation. The model's internal attention layers weigh the surrounding words to produce a representation that captures the specific activated sense.

The self-attention mechanism in transformer architectures computes a weighted sum of all other tokens in the input sequence to build the embedding for a target token. For an ambiguous abbreviation like 'MI', the model learns to assign high attention weights to relevant context words:

• Cardiology context: 'chest pain,' 'troponin,' 'EKG' → high attention → 'Myocardial Infarction' sense.
• Cardiology context: 'regurgitation,' 'leaflet,' 'valve' → high attention → 'Mitral Insufficiency' sense. This attention distribution is what makes the embedding contextual.

Contextual models like BERT and ClinicalBERT are bidirectional, meaning the embedding for a word is informed by both the tokens to its left and its right simultaneously. This is critical for clinical text where a defining term may appear after the abbreviation:

• 'The patient's MI was treated with thrombolytics.'
• 'The patient's MI was treated with a mitral clip.' A unidirectional model would fail to disambiguate until after processing the abbreviation, but a bidirectional model builds the correct sense into the embedding from the start.

Once a contextual embedding is generated for an ambiguous mention, it must be mapped to a canonical concept. This is typically done by computing the cosine similarity between the mention's embedding and pre-computed embeddings for each candidate sense from a knowledge base like UMLS.

• The mention embedding for 'MI' in a cardiology note will have a high cosine similarity (e.g., 0.95) with the embedding for the UMLS concept C0027051 (Myocardial Infarction).
• The same mention in a dermatology note will score higher against C0025202 (Melanoma In Situ). A cosine similarity threshold is then used to accept or reject the top mapping.

Contextual models use subword tokenization algorithms like WordPiece or Byte-Pair Encoding (BPE). This allows them to construct embeddings for rare or misspelled clinical abbreviations by composing representations of known subword fragments. For example, an unseen abbreviation like 'TEE' (Transesophageal Echocardiogram) can be tokenized into ['T', '##EE'], and its embedding is built from the context surrounding these fragments. This is essential for handling the long tail of idiosyncratic clinical shorthand.

Contextual embeddings are not a single output but a hierarchy of representations across the model's layers. Lower layers tend to capture local syntax and morphology, while upper layers capture high-level semantic meaning. For disambiguation, the final hidden layers are typically used as they encode the task-specific sense. Research shows that probing the upper layers of ClinicalBERT reveals distinct vector clusters for different senses of the same abbreviation, confirming that the disambiguation signal is encoded in the embedding geometry.

The computational task of identifying which meaning of a polysemous or homonymous word is activated by its use in a particular context. Contextual embeddings are the modern solution to this classic NLP problem.

• Core Challenge: Resolving ambiguity for terms like 'cold' (temperature vs. illness) or 'MI' (myocardial infarction vs. mitral insufficiency).
• Pre-Transformer Era: Relied on knowledge-based methods (WordNet) or supervised classifiers with static features.
• Transformer Era: Models like BERT generate dynamic embeddings that inherently capture the correct sense based on surrounding words.
• Clinical Impact: Directly prevents documentation errors by ensuring the correct ICD-10-CM code is assigned to a resolved abbreviation.

The core mechanism within transformer architectures that enables contextual embeddings to resolve ambiguity. Attention weights quantify how much each surrounding word influences the representation of an ambiguous term.

• Self-Attention: Allows 'MI' to attend to 'troponin' and 'chest pain' (cardiology context) or 'rash' and 'topical' (dermatology context).
• Multi-Head Attention: Different heads can specialize in syntactic dependencies, semantic relatedness, and domain-specific signals simultaneously.
• Visualization: Attention heatmaps provide interpretability, showing clinicians exactly which context words drove the disambiguation decision.
• Clinical BERT: Models like BioBERT and ClinicalBERT have attention heads fine-tuned on MIMIC-III, making them sensitive to clinical co-occurrence patterns.

The initial retrieval step in disambiguation that fetches all possible meanings of an abbreviation from a structured sense inventory before contextual scoring.

• UMLS Metathesaurus: The primary source for clinical senses, containing millions of concepts with unique CUIs.
• Inventory Construction: Pre-compiled from resources like SNOMED CT, RxNorm, and LOINC to cover all possible expansions.
• Filtering: Initial pruning using semantic type filtering (e.g., keeping only 'Disease or Syndrome' senses if the section header is 'Diagnoses').
• Embedding Pre-computation: Each candidate sense is encoded into a static embedding using the same model for efficient cosine similarity comparison.

A mathematical metric measuring the semantic relatedness between the contextualized embedding of an ambiguous abbreviation and the pre-computed embeddings of its candidate senses.

• Calculation: Measures the cosine of the angle between two vectors in high-dimensional space, ranging from -1 (opposite) to 1 (identical).
• Threshold Tuning: A high threshold (e.g., 0.85) ensures high precision but may reject valid matches; a lower threshold increases recall at the cost of potential false positives.
• Clinical Application: When resolving 'CHF', the contextual embedding from 'The patient has CHF' will show high cosine similarity to the 'Congestive Heart Failure' embedding and low similarity to 'Children's Health Fund'.
• Confidence Scoring: The raw similarity score can be surfaced in a human-in-the-loop review interface to prioritize low-confidence predictions for clinical review.

The use of information beyond the immediate sentence to resolve abbreviations that remain locally ambiguous. Contextual embeddings can be pooled or aggregated across an entire clinical note.

• Problem List Integration: A patient's coded problem list provides strong prior evidence for disambiguation (e.g., a history of 'I25.10' strongly favors the cardiac sense of 'MI').
• Section Header Awareness: A model can learn that 'MI' in the 'Past Medical History' section is more likely to be a diagnosis than a procedure.
• Long-Range Dependencies: Transformer architectures with sufficient context windows (e.g., 4096 tokens) can attend to relevant information paragraphs away.
• Temporal Reasoning: Combining document-level context with temporal expression normalization ensures that a resolved abbreviation is correctly placed in the patient's clinical timeline.

The critical task of determining whether a resolved abbreviation falls within the scope of a negation cue, ensuring that 'no MI' is correctly interpreted as the absence of myocardial infarction.

• ConText Algorithm: Extends the classic NegEx algorithm to determine negation, temporality (historical vs. recent), and experiencer (patient vs. family member).
• Scope Boundaries: Defined by syntactic patterns and punctuation; negation typically scopes from the cue ('no', 'denies', 'without') to the next clause boundary.
• Embedding Interaction: The contextual embedding of 'MI' in 'denies any MI' will be influenced by the negation cue, shifting it away from affirmed sense embeddings.
• Downstream Impact: Prevents false-positive extraction that could lead to incorrect quality metrics, billing codes, or clinical decision support triggers.