SDOH Extraction Accuracy

Precision measures the exactness of the extraction system—the percentage of identified SDOH mentions that are actually correct. High precision ensures that a patient's record is not incorrectly flagged for a social risk they do not have.

• Formula: True Positives / (True Positives + False Positives)
• Clinical Impact: A false positive for 'homelessness' could trigger an unnecessary social work consult, wasting resources and eroding clinician trust in the AI system.
• Optimization Strategy: Increase the model's confidence threshold for entity classification, requiring stronger evidence before asserting a positive finding.

Recall measures the completeness of the extraction system—the percentage of all actual SDOH mentions in the text that the model successfully captures. High recall is critical for population health screening, where missing a social risk factor means a patient in need does not receive an intervention.

• Formula: True Positives / (True Positives + False Negatives)
• Clinical Impact: A false negative for 'food insecurity' means a malnourished patient is not connected to a nutritional assistance program.
• Optimization Strategy: Expand the model's synonym dictionaries and contextual embeddings to capture non-standard, colloquial expressions of social need (e.g., 'couch surfing' for housing instability).

The F1-score is the harmonic mean of precision and recall, providing a single, balanced metric for overall extraction accuracy. It is the primary benchmark when you need to optimize for both avoiding false alarms and capturing all true cases simultaneously.

• Formula: 2 * (Precision * Recall) / (Precision + Recall)
• Use Case: Comparing the performance of different model architectures (e.g., a fine-tuned Clinical BERT vs. a rule-based system) on a gold-standard SDOH corpus.
• Nuance: F1-score penalizes extreme imbalances between precision and recall, making it a more robust metric than simple accuracy for the typically imbalanced distribution of SDOH mentions in clinical notes.

Accuracy must be evaluated at two distinct granularities. Entity-level accuracy measures if a specific span of text (e.g., 'lost job 2 months ago') is correctly tagged as 'Unemployment'. Document-level accuracy measures if the system correctly determines that a patient has any social risk of a given type.

• Entity-Level: Critical for populating structured data fields in a FHIR SDOH Observation resource.
• Document-Level: Sufficient for triggering a binary screening alert or a CDS Hook.
• Metric Divergence: A system can have high document-level accuracy (it correctly flags the patient as having housing risk) but low entity-level accuracy (it mislabels the specific type of housing risk).

True accuracy requires correctly interpreting the context of a mention. A system must distinguish between affirmed and negated concepts, and identify who is experiencing the risk.

• Negation Detection: 'Patient denies food insecurity' must not be extracted as a positive finding. This requires robust Negation Detection for SDOH algorithms.
• Experiencer Detection: 'Patient's brother is homeless' must be classified as a family history finding, not a patient-level risk, to prevent erroneous data entry.
• Temporality: Identifying whether a risk is current, historical, or future is essential for accurate SDOH Phenotyping and risk stratification.

A model's accuracy is not static. Generalizability refers to its performance on data from new hospitals, different EHR systems, or diverse patient populations. SDOH Model Drift is the degradation of accuracy over time due to evolving documentation styles or screening tools.

• Cross-Institutional Validation: A model trained on academic medical center notes may fail on community health center notes with different linguistic patterns.
• Drift Monitoring: Continuous tracking of precision and recall in production is essential to detect when a model needs retraining with Active Learning for SDOH strategies.
• Algorithmic Fairness: Accuracy must be stratified by demographic subgroups to ensure the model performs equitably and does not amplify bias, a core tenet of Algorithmic Fairness for SDOH.

The foundational metrics for evaluating extraction correctness.

• Precision: Of all SDOH mentions the model flagged, how many were correct? High precision minimizes false alarms.
• Recall: Of all actual SDOH mentions in the text, how many did the model find? High recall minimizes missed risks.
• F1-Score: The harmonic mean of precision and recall, providing a single balanced accuracy metric when class distribution is uneven.

A critical sub-metric measuring the system's ability to distinguish affirmed from negated social risks.

• False Positive Risk: Failing to detect negation (e.g., 'denies homelessness') leads to incorrect risk flags and wasted care coordinator effort.
• Contextual Cues: Models must correctly interpret linguistic patterns like 'no evidence of,' 'rules out,' and 'denies any' to maintain high accuracy.
• Impact: Negation errors directly degrade precision and erode clinician trust in automated SDOH alerts.

Accuracy in identifying who is experiencing the social risk, not just what the risk is.

• Patient vs. Other: The system must distinguish if the patient is homeless or if a family member or caregiver is.
• False Attribution: Misclassifying a spouse's unemployment as the patient's contaminates the patient record and skews population health analytics.
• Evaluation: Measured as a separate accuracy score on a held-out test set annotated for experiencer roles.

The accuracy of determining the chronological status of an SDOH mention.

• Historical: 'Patient was homeless in 2019' — a past risk, potentially resolved.
• Current: 'Patient is currently living in a shelter' — an active, actionable risk.
• Future: 'At risk of eviction next month' — a predictive, preventative signal.
• Misclassifying a historical mention as current inflates false positive rates and misdirects interventions.

The instruction manual that defines the ground truth against which all accuracy is measured.

• Entity Scope: Precisely defines what constitutes an SDOH mention (e.g., does 'stressed about bills' count as financial insecurity?).
• Edge Cases: Provides rules for ambiguous, vague, or conflicting statements to ensure consistent human labeling.
• Inter-Annotator Agreement (IAA): A high IAA score (e.g., Cohen's Kappa > 0.8) on these guidelines is a prerequisite for a reliable gold standard corpus and meaningful accuracy metrics.

Evaluating accuracy not just globally, but across distinct demographic subgroups to detect bias.

• Disaggregated Evaluation: Measure precision and recall separately for populations stratified by race, ethnicity, and primary language.
• Bias Mitigation: If a model has significantly lower recall for Spanish-speaking patients' notes, it systematically underserves that population.
• Equity Metric: Fairness is a core component of overall accuracy; a model that is accurate on average but biased in practice is a failed model.