Weak Supervision

The atomic unit of weak supervision. Labeling functions (LFs) are user-defined heuristics that vote on labels for unlabeled data. In clinical NLP, an LF might be a regex pattern for 'chest pain' or a dictionary lookup against RxNorm. Each LF can output a class label or abstain. The key insight is that individual LFs can be noisy and incomplete; the generative model resolves their conflicts. Common LF types include keyword searches, pattern matching, distant supervision from knowledge bases, and third-party model predictions.

A probabilistic model that combines the outputs of multiple labeling functions to produce a single, probabilistic training label. The generative model learns the accuracy and correlation structure of each LF without any ground truth. It estimates P(Y | LF outputs) by modeling LF agreements and disagreements. This step transforms sparse, conflicting heuristic votes into confidence-weighted training labels, effectively denoising the weak supervision sources before downstream model training.

A weak supervision technique that leverages external knowledge bases to automatically label text. In clinical contexts, a distant supervision approach might link mentions in radiology reports to entries in SNOMED CT or the UMLS Metathesaurus. The core assumption is: if a sentence contains two entities known to be related in a knowledge base, the sentence expresses that relationship. This method is powerful for bootstrapping relation extraction but introduces label noise from incorrect entity linking.

The sparse matrix representation of all labeling function outputs. Rows represent unlabeled data points, columns represent labeling functions, and cells contain either a class label or an abstention. The label matrix is the input to the generative model. Key challenges include managing extreme sparsity (most LFs abstain on most examples) and detecting conflicts where LFs disagree. Matrix analysis reveals LF coverage, overlap, and conflict statistics essential for debugging weak supervision pipelines.

The overarching paradigm that treats weak supervision as a programming problem. Instead of manually labeling data, users write labeling functions as code. The Snorkel framework pioneered this approach. Data programming shifts the bottleneck from annotation time to engineering time. In clinical NLP, this enables domain experts to encode medical knowledge as rules rather than hand-labeling thousands of patient records. The system handles the probabilistic integration automatically.

Critical diagnostic metrics for weak supervision quality. Coverage measures the fraction of the dataset for which a labeling function votes (does not abstain). Overlap measures how often pairs of LFs vote on the same example. High overlap is necessary for the generative model to learn LF accuracies. Conflict measures disagreement between LFs. Effective weak supervision requires balancing these metrics: too little overlap prevents learning, while excessive conflict indicates poorly designed heuristics.

The fundamental building block of weak supervision. A labeling function is a heuristic, rule, or pattern that programmatically assigns a noisy label to unlabeled data. In clinical NLP, these can be:

• Keyword rules: Assigning MEDICATION if a drug name from RxNorm is detected
• Regular expressions: Identifying dosage patterns like \d+ mg
• Distant supervision: Linking clinical mentions to an external knowledge base like the UMLS Metathesaurus
• Cross-modal heuristics: Using structured EHR fields to label corresponding free-text mentions

Each function votes on a label, and these votes are aggregated by a generative model to produce probabilistic training labels.

A weak supervision paradigm that leverages an external knowledge base to automatically annotate a corpus. For medical NER, a disease gazetteer derived from SNOMED CT or ICD-10-CM is matched against clinical text. Any string match becomes a positive training example.

Key challenges in clinical contexts:

• Ambiguity: The abbreviation 'CA' could mean cancer, calcium, or California
• Coverage gaps: Knowledge bases rarely capture every surface form or misspelling
• Context blindness: A dictionary match doesn't account for negation ('no evidence of MI')

Distant supervision is often combined with NegEx-style rules to filter false positives.

The statistical engine that resolves conflicts between labeling functions. Rather than majority voting, a generative model explicitly models each labeling function's propensity to be correct on different classes and their pairwise correlations.

Formally, it learns:

• P(LF_i | Y): The probability that labeling function i outputs a given label, conditioned on the true class Y
• Dependencies: Which labeling functions tend to make correlated errors (e.g., two regex-based functions failing on the same misspelling)

The output is a single probabilistic training label for each data point, which can be used to train a downstream BiLSTM-CRF or transformer model with a noise-aware loss function.

A synergistic strategy that combines weak supervision with targeted human annotation. After training an initial model on noisy labels, active learning identifies the most uncertain or high-impact examples for expert review.

The iterative loop:

• The model queries a clinical annotator to label the 100 most ambiguous spans
• These high-quality labels are added to the training set or used to refine labeling functions
• The generative model re-estimates accuracies with the new ground truth seed

This approach maximizes inter-annotator agreement (IAA) on the examples that matter most, dramatically reducing the cost of building a gold-standard clinical NER corpus while maintaining high F1 scores.

A technique for transforming existing labeled data to create synthetic training examples, often used in conjunction with weak supervision. For clinical text, this involves applying label-preserving transformations:

• Synonym substitution: Replacing 'hypertension' with 'HTN' using a clinical abbreviation dictionary
• Entity swapping: Replacing one medication mention with another from the same RxNorm class
• Contextual perturbation: Inserting or removing adverbial modifiers ('mild', 'severe')

These augmentations increase the diversity of the weakly supervised training set, making the final model robust to the lexical variability inherent in clinical narratives written by different providers.