Clinical BERT for SDOH

Unlike static word embeddings (e.g., Word2Vec), Clinical BERT generates dynamic, context-aware vector representations. The word 'cold' in 'patient has a cold' versus 'patient is cold and homeless' receives entirely different embeddings. This polysemy resolution is critical for SDOH extraction, where terms like 'unstable' or 'stress' carry distinct meanings in social versus clinical contexts. The model's 12-layer Transformer architecture captures bidirectional context, allowing it to understand that 'no stable housing' is a housing insecurity mention even when phrased differently.

The base model is pre-trained on MIMIC-III and other de-identified clinical note repositories, then further fine-tuned on annotated SDOH corpora. This two-stage adaptation means the model inherently understands clinical abbreviations ('pt c/o'), section headers ('Social History'), and note structures before it ever learns to identify SDOH concepts. The result is a model that does not confuse a 'homeless' mention in a family history section with a patient's own housing status, a common failure mode for general-domain NLP.

Clinical BERT uses a WordPiece tokenizer with a vocabulary of 30,000+ tokens specifically adapted to clinical text. This handles rare and complex medical terminology by breaking words into subword units. For SDOH extraction, this means novel compound terms like 'food-insecure' or misspelled entries like 'unemploymnt' are not treated as out-of-vocabulary tokens. The tokenizer preserves meaningful morphemes, allowing the model to infer that 'housing' and 'housed' share semantic roots even when encountering them in novel contexts.

Annotated SDOH datasets are scarce and expensive to produce. Clinical BERT's pre-trained weights encode vast clinical knowledge, meaning it can achieve high extraction accuracy with relatively few labeled SDOH examples. Fine-tuning on as few as 500 annotated notes can yield viable performance. This sample efficiency is achieved because the model already understands clinical syntax and semantics; it only needs to learn the specific decision boundary for SDOH entity classification, rather than learning language from scratch.

When applied to SDOH extraction, Clinical BERT is typically configured with a token classification head (e.g., a linear layer on top of the final hidden states) for Named Entity Recognition. This architecture assigns a label to each token using the BIO (Begin, Inside, Outside) tagging scheme. For example, in 'lives in a shelter', 'shelter' is tagged as B-Housing, enabling precise span detection. This token-level granularity allows the model to extract multi-word entities like 'domestic violence shelter' as a single, coherent concept.

The multi-head self-attention mechanism allows Clinical BERT to relate a social risk mention to its relevant context across an entire clinical note. A mention of 'unemployed' in the chief complaint can be linked to a 'lost job 3 months ago' statement in the history of present illness. This global receptive field is essential for SDOH tasks like temporality classification and experiencer detection, where the model must determine if a risk is current or historical and whether it applies to the patient or a family member.

The fundamental output of Clinical BERT. Unlike static word vectors, these are dynamic representations where the meaning of a word like 'cold' shifts based on surrounding text.

• Mechanism: Generated by the transformer's bidirectional self-attention, weighing all tokens in a sequence.
• SDOH Relevance: Allows the model to distinguish 'homeless' (housing instability) from 'homeless shelter volunteer' (employment status).
• Dimension: Typically 768-dimensional dense vectors.

The specialized dataset of de-identified clinical notes used to adapt the base BERT model to the medical domain.

• Source: MIMIC-III, institutional EHR databases.
• Objective: Update pre-trained weights so the model understands clinical jargon, abbreviations, and note structure.
• Result: A model that recognizes 'c/o SOB' as 'complains of shortness of breath' rather than a derogatory term.

The preprocessing step that segments raw clinical text into sub-word units before feeding it to Clinical BERT.

• Vocabulary: A fixed set of ~30,000 tokens optimized for biomedical text.
• OOV Handling: Unknown words are split into known sub-units (e.g., 'homelessness' -> 'homeless', '##ness').
• Special Tokens: [CLS] for sequence classification and [SEP] for separating sentences.

A task-specific neural layer added on top of Clinical BERT's output to perform SDOH classification.

• Architecture: A simple linear layer that takes the [CLS] token embedding as input.
• Function: Maps the 768-dimension vector to a probability distribution over SDOH categories (e.g., 'Housing Insecurity', 'Food Insecurity', 'None').
• Training: Only this head and the final BERT layers are typically updated during SDOH-specific fine-tuning.

The core computational engine of the transformer architecture that allows Clinical BERT to model long-range dependencies in clinical text.

• Self-Attention: Computes a weighted sum of all token representations, allowing 'patient' to attend to 'homeless' even if separated by many words.
• Multi-Head: 12 parallel attention heads focus on different linguistic relationships (syntax, semantics, negation).
• SDOH Impact: Critical for linking a social risk mention to the correct experiencer (patient vs. family member).

The machine learning paradigm that makes Clinical BERT possible by leveraging knowledge from a general domain to solve a specialized medical task with limited labeled data.

• Pre-training: BERT learns general language structure from massive text corpora (Wikipedia, Books).
• Domain Adaptation: Clinical BERT is further pre-trained on clinical notes.
• Task Fine-tuning: The adapted model is fine-tuned on a small, labeled SDOH dataset. This avoids training a complex model from scratch.