SDOH Model Drift

The statistical properties of the target variable—the definition of a social risk—change. This occurs when clinical understanding evolves.

• Example: A model trained to identify 'food insecurity' based on mentions of 'hunger' may miss newer documentation using 'nutritional access deficit' or 'low food literacy'.
• Impact: The relationship between the input text and the SDOH label shifts, causing a decay in recall as new linguistic expressions of old concepts are missed.
• Mitigation: Requires periodic re-annotation of a gold standard corpus to capture evolving clinical vernacular and updated screening tool terminology.

The input data distribution changes when a health system adopts a new standardized screening instrument, fundamentally altering the structure of clinical notes.

• Example: A model trained on free-text narrative notes experiences drift when a hospital system mandates the structured PRAPARE tool or Gravity Project terminology in EHR templates.
• Impact: The model encounters a sudden surge of templated, checklist-style language, leading to a drop in precision as it incorrectly parses structured headings as novel risk mentions.
• Mitigation: Implement a data observability pipeline to detect distributional shifts in note structure and trigger model retraining with templated examples.

The demographic makeup of the patient population changes, altering the prior probability of specific SDOHs without changing the clinical definition.

• Example: A model deployed in a region experiencing a sudden influx of refugees will encounter a higher prevalence of 'housing instability' and 'limited English proficiency' mentions than in its training data.
• Impact: The model's calibrated confidence scores become unreliable, potentially underestimating risk for the new demographic group and exacerbating algorithmic bias.
• Mitigation: Continuous monitoring of model performance stratified by demographic features and recalibration using active learning on edge cases from the new population.

The relevance of a social risk mention is time-sensitive, and a model's ability to correctly classify temporality can drift as documentation patterns change.

• Example: A model trained to extract 'current' housing issues may begin to misclassify historical mentions as active if clinicians start using new phrasing like 'resolved homelessness' instead of 'history of homelessness'.
• Impact: False positives for active social risks flood closed-loop referral systems, overwhelming care coordinators with stale alerts and eroding trust in the automated workflow.
• Mitigation: Fine-tune temporality classification heads specifically on new documentation patterns and implement a human-in-the-loop review queue for high-impact, time-sensitive SDOH extractions.

A non-clinical trigger for drift occurs when the source EHR system undergoes a version upgrade, altering data schemas, field mappings, or text encoding.

• Example: An HL7 FHIR API update changes the field where the 'Social History' narrative is stored, or a character encoding shift introduces parsing errors in free-text notes.
• Impact: The NLP pipeline suffers a catastrophic data quality failure, ingesting malformed or truncated text, which leads to nonsensical SDOH extractions and a complete breakdown of the SDOH NLP pipeline.
• Mitigation: Rigorous data contract testing between the EHR interface and the NLP service, coupled with schema validation checks that halt ingestion upon detecting a breaking change.

The ground truth itself changes as clinical coding guidelines and annotation standards are updated, rendering the original training labels obsolete.

• Example: The official ICD-10-CM Z-Codes for SDOH are expanded, or internal annotation guidelines are revised to include 'risk of' states as positive findings, whereas previously they were excluded.
• Impact: A model with high technical accuracy is now clinically inaccurate because it is solving an outdated task. Its outputs no longer align with current USCDI SDOH Data Elements requirements.
• Mitigation: Version-controlled annotation guidelines and a scheduled cadence for re-annotating the evaluation gold standard to align with the latest regulatory and clinical coding standards.

The core statistical phenomenon underlying SDOH model drift. It refers to a change in the relationship between input variables and the target variable over time.

• Real-world drift: A new housing assistance program changes how homelessness is documented, altering the linguistic patterns the model learned.
• Distinction from data drift: Data drift is a change in the input distribution (e.g., a new screening tool is adopted). Concept drift is a change in the mapping function itself (e.g., the same phrase 'couch surfing' now appears in a different context).
• Monitoring: Requires tracking the joint distribution P(X, Y), not just P(X) or P(Y) independently.

A specific type of dataset shift where the distribution of input features P(X) changes, but the conditional distribution of the label given the features P(Y|X) remains constant.

• SDOH example: A health system expands screening to a new pediatric clinic. The vocabulary and note structure differ from adult primary care, but the definition of 'food insecurity' remains the same.
• Detection: Statistical tests like the Kolmogorov-Smirnov test or Maximum Mean Discrepancy can compare feature distributions between training and production data.
• Mitigation: Importance-weighting training samples or domain adaptation techniques can correct for covariate shift without full retraining.

A change in the prior probability of the target variable P(Y) over time, independent of the input features.

• SDOH example: A sudden economic recession causes a genuine increase in the prevalence of housing instability in the patient population. The model's predicted prevalence should shift to match.
• Monitoring: Track the predicted positive rate against a known baseline. A significant deviation without a corresponding change in inputs may indicate label drift.
• Calibration: Label drift often requires recalibrating the model's decision threshold to maintain a target precision or recall, rather than full retraining.

The continuous, automated process of tracking production model performance metrics to detect degradation, including SDOH extraction accuracy.

• Key metrics: Precision, recall, F1-score, and prediction confidence distributions tracked over time windows.
• Ground truth challenge: Unlike structured data models, NLP models require manual chart review for true labels. Proxy metrics like the rate of 'null' extractions or the KL divergence of output distributions are often used.
• Alerting: Threshold-based alerts trigger when a metric deviates beyond an acceptable boundary, initiating a root cause analysis or retraining pipeline.

The scheduled or triggered frequency at which an SDOH extraction model is updated with new labeled data to combat drift.

• Fixed schedule: Retraining quarterly or annually, regardless of observed drift. Simple but potentially wasteful or insufficient.
• Triggered retraining: Retraining initiated automatically when a monitoring metric breaches a threshold. Requires a robust MLOps pipeline.
• Incremental learning: Updating model weights continuously with new data. High-risk for catastrophic forgetting and requires careful validation in a regulated clinical context.

A carefully curated, static, and statistically representative set of annotated clinical notes used as a stable benchmark to evaluate model performance over time.

• Purpose: Provides a fixed reference point to isolate model degradation from changes in the production data distribution.
• Construction: Must span diverse document types, patient demographics, and SDOH categories. Annotation is performed by multiple clinical experts with high inter-annotator agreement.
• Limitation: A golden dataset itself can become stale if clinical language evolves significantly, necessitating periodic updates to the benchmark.