Active Learning

The most common query strategy where the model requests labels for examples it is least confident about.

• Least Confidence: Selects instances where the model's highest predicted probability is lowest.
• Margin Sampling: Chooses examples where the difference between the top two predicted class probabilities is smallest, indicating decision boundary ambiguity.
• Entropy Sampling: Uses Shannon entropy over all class probabilities to measure overall prediction uncertainty.

In clinical NER, this targets ambiguous mentions like 'cold' (temperature vs. disease) that sit near the model's decision boundary.

Selects batches of examples that are representatively diverse across the feature space, preventing redundant labeling of similar instances.

• Model-based outliers: Identifies examples far from the learned decision boundary that may represent rare clinical phenotypes.
• Cluster-based selection: Partitions unlabeled data into clusters and samples from each to ensure broad coverage of the clinical corpus.

This prevents the common failure mode where uncertainty sampling selects 50 nearly identical ambiguous radiology reports, wasting annotation budget.

Maintains an ensemble of diverse models and selects examples where the committee disagrees most.

• Vote entropy: Measures disagreement across model predictions as a selection criterion.
• KL divergence to the mean: Quantifies how much each model's prediction distribution diverges from the ensemble average.

In medical NER, a committee might include a BioBERT model, a CRF-based tagger, and a dictionary-based system. Disagreement on whether 'mg' is part of a dosage entity signals an informative training example.

Selects examples that would cause the largest gradient update to the current model parameters if their true label were known.

• Expected Gradient Length (EGL): Computes the expected magnitude of the gradient vector for each candidate, selecting those with the highest potential impact.
• Fisher Information Matrix: Uses second-order information to identify examples that would most reduce parameter uncertainty.

This strategy is computationally intensive but highly effective for fine-tuning clinical language models where each annotation hour costs $50-150 for domain experts.

Combines multiple selection criteria to balance exploration and exploitation during the annotation process.

• Uncertainty + Diversity: First filters for uncertain examples, then applies diversity sampling to avoid redundancy.
• Density-weighted uncertainty: Weights uncertainty scores by local density in feature space to avoid selecting outliers that are not representative.
• Curriculum active learning: Starts with easy, high-confidence examples to establish a baseline, then progressively introduces harder cases.

Production clinical NER systems often use hybrid strategies to maintain annotation efficiency across long-running projects with evolving data distributions.

Addresses the initial model bootstrapping problem when no labeled data exists yet.

• Clustering-based seeding: Clusters the unlabeled corpus and selects the most central example from each cluster for initial labeling.
• Prototypical examples: Identifies instances that are most representative of the overall data distribution using unsupervised methods.
• Domain expert curation: Leverages clinician knowledge to manually select a small set of archetypal examples covering known entity types and edge cases.

A well-chosen seed set of 50-100 examples can dramatically accelerate the active learning curve compared to random initialization.