Annotation Guidelines

Defines the precise semantic boundaries of each entity type. A guideline must specify not just what to tag, but what not to tag.

• Inclusion: 'Tag all mentions of current or past homelessness, including sheltered and unsheltered states.'
• Exclusion: 'Do not tag mentions of future hypothetical risk (e.g., "at risk of eviction") unless explicitly stated as an active concern.'
• Edge Case: 'Tag "couch surfing" as an unstable housing indicator even if the patient does not self-identify as homeless.'

This prevents annotator drift and ensures inter-annotator agreement (IAA) remains high.

A structured hierarchy of SDOH domains, each with a canonical definition, trigger words, and a concrete example from real clinical text.

• Housing Insecurity: Lack of stable, permanent shelter. Trigger words: homeless, evicted, shelter, transitional housing.
• Food Insecurity: Limited or uncertain access to adequate food. Trigger words: ran out of food, skipped meals, food stamps.
• Financial Strain: Inability to afford necessities. Trigger words: cannot afford medication, unemployed, utilities shut off.
• Social Isolation: Lack of meaningful social contact. Trigger words: lives alone, no family contact, widowed.

Each entity type must be mutually exclusive to prevent overlapping annotations.

Beyond the entity span, annotators must label key contextual dimensions that determine clinical relevance.

• Temporality: Is the risk current, historical, or future? A past episode of homelessness has different clinical implications than an active crisis.
• Experiencer: Is the patient the one experiencing the risk, or is it a family member/caregiver? 'Patient's son is homeless' must not be attributed to the patient.
• Certainty: Is the risk affirmed ('patient is homeless'), negated ('denies homelessness'), or uncertain ('possible food insecurity')?

These attributes enable downstream phenotyping algorithms to filter noise from actionable signals.

Precise rules for token-level boundary selection ensure consistency in the training data.

• Minimal Span: Tag the smallest contiguous text that captures the complete concept. For 'severe chronic food insecurity', tag the entire phrase, not just 'food'.
• Discontinuous Mentions: Define a policy for when a concept is split by punctuation or other words (e.g., 'food, and often housing, insecurity'). Specify whether to create one or two annotations.
• Conjunction Handling: 'Patient lacks food and shelter' should yield two distinct annotations: one for Food Insecurity and one for Housing Insecurity.

Inconsistent span boundaries introduce noise that degrades token classification model performance.

A formal process for resolving disagreements between annotators to create the final ground truth.

• Double Annotation: Every document is independently annotated by two reviewers.
• Adjudication Trigger: A third, senior annotator (often a clinical informaticist) reviews all disagreements where F1 overlap < 0.90.
• Guideline Amendment: Recurring disagreements signal a flaw in the guidelines, not the annotators. The guideline must be iteratively updated and version-controlled.
• Consensus Set: Only adjudicated, agreed-upon annotations enter the gold standard corpus.

This process is essential for defensible evaluation-driven development.

Annotation guidelines are living documents that evolve through a structured feedback loop.

• Pilot Round: Annotate a small sample (50-100 notes) to surface ambiguous cases before full-scale annotation begins.
• Error Analysis: Categorize all annotator errors as boundary errors, missed entities, or spurious entities.
• Active Learning Integration: Use model predictions on unlabeled data to find uncertainty regions where the guideline is most likely to be ambiguous.
• Version History: Maintain a changelog for every guideline update to ensure reproducibility of annotation runs.

This cycle directly supports active learning for SDOH and prevents model drift over time.

The foundational NLP subtask that annotation guidelines are designed to teach. NER involves identifying and categorizing specific mentions of social risk factors—such as 'homeless', 'unemployed', or 'food insecure'—within free-text clinical documents. Guidelines define the exact entity types (e.g., HOUSING_STATUS, FOOD_INSECURITY), their boundaries (span of text), and attributes (e.g., temporality). Without precise NER guidelines, annotators cannot produce consistent, machine-readable labels for model training.

A critical contextual analysis task governed by annotation guidelines. Negation detection distinguishes whether a social risk factor is affirmed ('patient is homeless') or negated ('patient denies homelessness'). Guidelines must explicitly define how to handle linguistic cues like 'denies', 'no evidence of', and 'without'. Misclassifying a negated mention as affirmed introduces false positives into the gold standard corpus, directly degrading model precision and clinical trustworthiness.

An NLP task that determines the chronological status of a social risk mention. Annotation guidelines must specify how to label whether a housing crisis is a current, historical, or future concern. For example, 'lost housing in 2019' is historical, while 'facing eviction next month' is future. This temporal framing is essential for clinical actionability—a resolved past issue requires different intervention than an active crisis. Guidelines define the temporal categories and the textual evidence required to assign them.

A contextual NLP task that identifies who is experiencing the social risk mentioned in a clinical note. The patient may not always be the subject—a note might state 'patient's spouse lost their job' or 'mother is homeless'. Annotation guidelines must define rules for assigning the experiencer to the correct entity (e.g., PATIENT, FAMILY_MEMBER, CAREGIVER). This prevents the erroneous attribution of a family member's social risk to the patient's own record.

A subset of diagnosis codes (Z55-Z65) used to document social determinants of health in a patient's structured medical record. Annotation guidelines often map extracted SDOH entities to these standardized codes. For instance, Z59.0 represents homelessness, while Z59.4 indicates lack of adequate food. Guidelines define the mapping logic between free-text mentions and specific Z-codes, ensuring the extracted data can be integrated into billing workflows and population health analytics.

A statistical measure of the consistency between multiple human annotators applying the same guidelines to the same text. High IAA (typically measured via Cohen's Kappa or Krippendorff's Alpha) validates that the guidelines are clear, unambiguous, and reproducible. Low IAA signals guideline defects—ambiguous edge cases, poorly defined entity boundaries, or conflicting rules. Iterative guideline refinement based on IAA analysis is the core mechanism for building a high-quality gold standard corpus.