Active Learning

The most common active learning strategy where the model identifies instances it is least confident about for human review.

• Least Confidence: Queries the instance with the lowest predicted probability for its most likely class.
• Margin Sampling: Queries the instance where the difference between the top two predicted probabilities is smallest, indicating class ambiguity.
• Entropy Sampling: Queries the instance with the highest entropy across all possible class predictions, maximizing information gain.

In pharmacovigilance, this efficiently targets ambiguous adverse event mentions where the model cannot distinguish between a drug causing a symptom versus the underlying disease.

A strategy that maintains an ensemble of distinct models trained on the same labeled data. The algorithm queries instances where the committee of models disagrees most.

• Measures disagreement using vote entropy or Kullback-Leibler divergence.
• Effective for identifying edge cases in Individual Case Safety Reports where different model architectures interpret clinical narratives differently.
• Reduces annotator bias by presenting genuinely ambiguous cases rather than random samples.

This approach is particularly valuable when building extraction models for rare adverse drug reactions where consensus among models is inherently low.

A batch-mode strategy that selects a set of unlabeled instances that are both informative and representative of the underlying data distribution.

• Prevents redundant queries by ensuring selected samples are not clustered in the same region of feature space.
• Uses cosine similarity or determinantal point processes to enforce diversity.
• Critical for pharmacovigilance corpora where clinical notes contain repetitive templated language.

Without diversity constraints, uncertainty sampling may query 50 nearly identical seriousness criteria descriptions, wasting annotator effort.

Techniques for initiating active learning when zero labeled data exists and the model cannot yet compute uncertainty.

• Random Sampling: Initial seed set selected purely at random to establish baseline labels.
• Clustering-Based: Unlabeled data is clustered, and a representative sample from each cluster is queried to ensure coverage.
• Domain-Rule Initialization: Weak labelers or regular expressions pre-label obvious cases, allowing the model to bootstrap from high-confidence examples.

In pharmacovigilance, leveraging MedDRA dictionaries and existing Standardised MedDRA Queries provides a strong cold-start signal before human annotation begins.

The defined conditions under which the active learning loop terminates, balancing annotation cost against model performance.

• Performance Plateau: Training stops when F1-score improvements fall below a threshold (e.g., < 0.5%) over consecutive iterations.
• Budget Exhaustion: A hard limit on the number of labeled instances or annotation hours.
• Inter-Annotator Agreement Threshold: Stops when the model's predictions achieve sufficient agreement with human annotators, measured by Cohen's Kappa.

For FAERS extraction models, stopping criteria are often tied to achieving parity with existing rule-based systems before deployment.

The operational framework connecting the active learning algorithm to the annotation workflow and review interface.

• Queried instances are surfaced in a dedicated Human-in-the-Loop Review Interface designed for clinical reviewers.
• Annotations are captured with metadata: annotator ID, timestamp, and confidence.
• Inter-Annotator Agreement is continuously monitored to detect reviewer fatigue or drift.
• Resolved labels are immediately fed back into the training set, triggering model retraining.

This closed-loop architecture ensures that pharmacovigilance models continuously improve from expert causality assessment decisions.

The most common query strategy where the model requests labels for instances it is least confident about. In pharmacovigilance NLP, this targets ambiguous adverse event mentions where the model's predicted probability for a class is closest to the decision boundary. Variants include:

• Least Confidence: Selects the instance with the lowest maximum predicted probability.
• Margin Sampling: Selects the instance with the smallest difference between the top two predicted probabilities.
• Entropy Sampling: Selects the instance with the highest information entropy across all possible class labels.

A strategy that maintains an ensemble of models trained on the same labeled data but with different initializations or hyperparameters. The algorithm selects instances where the committee disagrees most, measured by vote entropy or Kullback-Leibler divergence. This is particularly effective for identifying rare adverse event mentions where a single model might be overconfident in a spurious pattern.

The software layer through which a domain expert—such as a drug safety officer—reviews and labels queried instances. Effective interfaces display the original clinical context, highlight the ambiguous span, and often show the model's top-k predictions with confidence scores. Latency in this feedback loop is a critical bottleneck; optimized interfaces can reduce annotation time by 40-60% compared to raw text review.

The initial phase where the model has zero labeled data and cannot compute uncertainty. Strategies to bootstrap include:

• Random Sampling: Selecting a statistically representative seed set.
• Clustering-Based Sampling: Using unsupervised embeddings to select diverse, representative instances.
• Domain Rule-Based Seeding: Using deterministic patterns to find high-yield examples, such as known adverse event keywords.

A complementary approach that prevents the selection of redundant, similarly uncertain instances. By incorporating a diversity constraint, the algorithm ensures the batch of queried instances covers a broad region of the feature space. Common techniques include k-means clustering on model embeddings or determinantal point processes (DPP) to maximize the representativeness of each annotation batch.

A statistical measure of consensus among multiple human oracles, typically calculated using Cohen's Kappa or Krippendorff's Alpha. In active learning loops, low IAA on queried instances signals genuine ambiguity in the annotation guidelines, not model failure. Tracking IAA drift over time helps detect concept drift in how adverse events are defined and reported.