Confusion Pair Analysis

Builds a sense-level confusion matrix where rows represent the true sense and columns represent the predicted sense. Each cell quantifies how often sense A was incorrectly predicted as sense B. For example, a matrix might reveal that 'MI' as Myocardial Infarction is confused with 'MI' as Mitral Insufficiency in 34% of cardiology cases, while confusion with 'MI' as Mental Illness occurs in only 2% of cases. This granular view moves beyond aggregate accuracy to pinpoint specific failure modes.

Analyzes the linguistic context windows surrounding each confusion pair instance to identify shared features that mislead the model. Key signals examined include:

• Overlapping n-grams: Phrases like 'chest pain' appearing with both cardiac senses
• Section header similarity: Both senses occurring in 'Assessment' sections
• Co-occurring entities: Shared mentions of 'troponin' or 'echocardiogram' that fail to disambiguate This analysis reveals whether the model is relying on spurious correlations rather than discriminative features.

Identifies class distribution skew within confusion pairs. A model may default to the majority sense when training data is imbalanced. For instance, if 'CHF' as Congestive Heart Failure appears 9,000 times but 'CHF' as Chronic Hepatic Failure appears only 100 times, the model learns a prior that overwhelms contextual signals. Confusion pair analysis quantifies this bias by calculating the confusion rate asymmetry—how much more often the minority sense is misclassified as the majority sense than vice versa.

Projects contextualized embeddings of ambiguous mentions into a lower-dimensional space using techniques like t-SNE or UMAP. For each confusion pair, the analysis visualizes whether the two senses form separable clusters or overlapping regions. Overlap indicates the model's encoder fails to produce discriminative representations. This diagnostic guides decisions about whether to improve the encoder architecture, add contrastive learning objectives, or augment training data with harder negative examples.

Uses confusion pair analysis output to generate synthetic training examples specifically for the confused senses. Strategies include:

• Contextual substitution: Replacing one sense with the other in existing sentences to create contrastive pairs
• Expert-crafted minimal pairs: Sentences identical except for the disambiguating context word
• Oversampling minority senses within the confusion pair to rebalance the training distribution This targeted approach is more efficient than general data augmentation, directly addressing the model's specific weaknesses.

Adjusts confidence thresholds on a per-pair basis rather than using a global threshold. If the model consistently confuses 'RA' as Rheumatoid Arthritis with 'RA' as Right Atrium, a higher confidence threshold can be set for this specific pair, routing low-confidence predictions to human-in-the-loop review. This fine-grained calibration optimizes the trade-off between automation rate and error rate, ensuring that known confusion pairs receive appropriate scrutiny without slowing down high-confidence predictions.

The foundational computational task of identifying which meaning of a polysemous word is activated by its context. In clinical NLP, WSD is the parent problem that confusion pair analysis helps debug. Key aspects:

• Resolves ambiguity for homonyms and polysemes
• Relies on a pre-defined sense inventory like UMLS
• Evaluated using precision, recall, and F1 per sense
• Confusion pair analysis identifies the specific sense pairs where WSD models fail most frequently

A dynamic vector representation where the same word receives different embeddings depending on surrounding text. This is the core mechanism that enables disambiguation models to distinguish between senses. How it works:

• Transformer models like ClinicalBERT generate context-aware vectors
• The embedding for 'MI' in 'MI with ST elevation' differs from 'MI with regurgitation'
• Confusion pair analysis examines when these embeddings are insufficiently separated
• Cosine similarity between the contextual embedding and candidate sense embeddings drives classification

The initial retrieval step that fetches all possible meanings of an abbreviation from a structured sense inventory before scoring. The quality of this candidate list directly impacts confusion pair formation. Process:

• Queries UMLS Metathesaurus for all concepts matching an abbreviation string
• Filters by semantic type to remove implausible candidates (e.g., excluding 'Geographic Area' for a clinical abbreviation)
• A narrow candidate list reduces confusion pairs; an overly broad list introduces spurious ones
• Confusion pair analysis often reveals that the correct sense was missing from the candidate set entirely

A disambiguation technique that constrains candidate meanings based on high-level UMLS categories. This is often the first line of defense against confusion pairs. Example:

• 'MI' could map to 'Disease or Syndrome' (Myocardial Infarction) or 'Anatomical Abnormality' (Mitral Insufficiency)
• If the surrounding context strongly signals a disease discussion, the anatomical sense is deprioritized
• Confusion pair analysis identifies when semantic type signals are too weak or ambiguous
• Works in tandem with fine-grained entity typing for hierarchical filtering

The mechanism in transformer architectures that allows a model to weigh the importance of different context words when resolving an ambiguous abbreviation. Confusion pair analysis often traces errors to attention weight distributions. Key points:

• Self-attention heads learn to focus on clinically relevant context words
• For 'MI', one head might attend to 'troponin' (cardiac) while another attends to 'murmur' (valvular)
• Confusion pairs arise when attention is drawn to misleading or insufficient context
• Visualizing attention weights for confused pairs reveals which tokens the model is incorrectly prioritizing

A metric measuring the semantic relatedness between a contextualized abbreviation embedding and candidate sense embeddings. This is the decision boundary where confusion pairs manifest. Operational details:

• A high cosine similarity (>0.85) typically indicates a correct mapping
• Confusion pairs cluster in the 'uncertainty zone' where similarity scores for two candidates are nearly identical
• Adjusting the threshold trades off precision and recall for specific pairs
• Confusion pair analysis often drives per-pair threshold tuning rather than a single global threshold