Active Learning for SDOH

The most common query strategy where the model selects instances it is least confident about for annotation.

• Least Confidence: Picks notes where the model's highest predicted probability is lowest
• Margin Sampling: Selects instances where the difference between the top two predicted classes is smallest
• Entropy Sampling: Chooses notes with the highest overall prediction uncertainty across all SDOH categories

This approach ensures annotators focus on ambiguous cases like distinguishing between 'homeless' and 'housing instability' rather than obvious positives or negatives.

A selection strategy that ensures the annotated corpus represents the full distribution of clinical language and social risk expressions.

• Prevents overfitting to a narrow set of documentation patterns
• Captures rare but critical SDOH mentions like 'lives in a shelter' or 'unable to afford medication'
• Uses clustering on Clinical BERT embeddings to identify underrepresented regions of the feature space

This is essential for SDOH extraction because social risk factors are documented inconsistently across specialties, note types, and provider styles.

A disagreement-based approach where multiple models trained on the same labeled data vote on unlabeled instances.

• Instances with the highest disagreement are prioritized for annotation
• Each committee member can be a different architecture or a differently initialized version of the same model
• Effective for identifying ambiguous SDOH mentions where context is critical, such as distinguishing a patient's housing status from a family member's

This method reduces annotation waste by surfacing only the most contentious clinical narratives.

The initial phase where no labeled data exists and the system must build a seed training set.

• Begins with keyword-based heuristics and regular expressions to identify likely SDOH mentions
• Uses unsupervised techniques like topic modeling to surface diverse note clusters
• A domain expert labels a small, representative sample to train the first weak model
• This seed model then drives subsequent active learning iterations

Cold start is particularly challenging for SDOH due to the wide variety of non-standardized expressions for social risk.

The annotation workflow that connects the active learning algorithm to a clinical reviewer interface.

• Low-confidence predictions are surfaced in a review queue with model suggestions pre-populated
• Annotators confirm, correct, or reject entity spans and classifications
• Each correction is immediately fed back into the training pipeline for the next model iteration
• Review interfaces often integrate with annotation guidelines specific to SDOH entity types like housing, food, and transportation insecurity

This tight feedback loop ensures rapid model improvement while maintaining clinical accuracy.

The metrics and thresholds that determine when the active learning cycle has achieved sufficient performance or requires re-initiation.

• Performance Plateau: When additional annotations no longer improve F1-score on a held-out test set
• Inter-Annotator Agreement: High agreement between reviewers indicates clear, well-defined extraction boundaries
• SDOH Model Drift Detection: Continuous monitoring for performance degradation as documentation patterns or screening tools change over time
• Triggers re-initiation of active learning when drift exceeds acceptable thresholds

This ensures the extraction model remains robust against evolving clinical language and new SDOH screening instruments.

A core query strategy where the model selects instances for annotation that it is least confident about. For SDOH extraction, this means a model trained to find 'housing insecurity' will prioritize sentences where its prediction probability hovers around 50%, rather than those where it is 99% certain. This efficiently exposes the model to ambiguous phrasing like 'couch surfing' or 'staying with a friend' that exist in the model's decision boundary.

A strategy that maintains an ensemble of models and selects instances where the models disagree the most. In an SDOH context, one model might classify 'no money for food' as a financial strain, while another flags it as food insecurity. This disagreement, measured by vote entropy or Kullback-Leibler divergence, highlights complex, multi-faceted social risk statements that provide the highest training signal for the entire ensemble.

The iterative process where a clinical reviewer audits and corrects the model's low-confidence predictions. Unlike passive labeling, this workflow presents the reviewer with pre-annotated suggestions from the model. The reviewer confirms or rejects entities like 'eviction notice' or 'unemployed', and their corrections are immediately fed back into the training set to refine the model's understanding of nuanced social risk language.

The initial phase of an active learning pipeline where no labeled data exists. To overcome this, a small, representative seed set of clinical notes must be manually annotated. Strategies include random sampling or clustering clinical notes by semantic similarity to ensure the initial training batch covers diverse SDOH domains—housing, food, transportation, and interpersonal safety—before the active learning loop can begin querying informatively.

A predefined metric that signals when the active learning loop has achieved sufficient performance, halting the expensive annotation process. For SDOH extraction, this is typically when the model's F1-score on a held-out test set plateaus or when the marginal improvement per new annotated instance drops below a threshold. This prevents over-annotation and controls the cost of building a production-grade extraction model.

A selection approach that ensures the queried instances are not only uncertain but also representative of the broader data distribution. In SDOH extraction, pure uncertainty sampling might repeatedly select similar 'homelessness' examples. Diversity sampling uses clustering or core-set selection to force the query to cover rare but critical social risks like 'utility shut-off' or 'lack of childcare', preventing model bias toward high-frequency determinants.