Token Classification

The standard annotation scheme for training token classifiers. Each token receives a B (Beginning), I (Inside), or O (Outside) prefix to demarcate entity spans precisely.

• B-DRUG: First token of a medication name
• I-DRUG: Continuation tokens of the same medication
• O: Tokens outside any clinical entity

This scheme converts unstructured text into a structured sequence labeling problem, enabling models to learn exact entity boundaries rather than just presence.

A discriminative probabilistic model that serves as a decoding layer on top of neural token classifiers. CRFs model transition probabilities between adjacent labels, preventing impossible sequences like an I-DISEASE tag following an O tag.

• Captures label dependencies across the entire sequence
• Uses Viterbi decoding to find the globally optimal tag path
• Often paired with BiLSTM encoders in clinical NER architectures

While modern transformer-based models can learn these dependencies implicitly, CRFs remain valuable for enforcing structural constraints in low-resource medical domains.

A modern alternative to token-level BIO tagging that directly classifies arbitrary text spans as entity types. This approach elegantly handles nested and overlapping clinical entities.

• Enumerates all possible spans up to a maximum length
• Classifies each span independently as an entity type or null
• Eliminates the need for complex BIO consistency rules

Example: In "severe chest pain," span categorization can simultaneously label "chest pain" as a SYMPTOM and "severe chest pain" as a FINDING with severity context.

The downstream task that maps a recognized token span to its unique identifier in a standardized ontology. After token classification identifies "heart attack," concept normalization links it to C0027051 in the UMLS Metathesaurus.

• Resolves lexical variability ("MI" = "myocardial infarction")
• Handles abbreviation expansion in clinical context
• Enables interoperability across different EHR systems

This step transforms surface-form entity mentions into computable, semantically grounded concepts suitable for clinical decision support and cohort analysis.

A rule-based algorithm that determines whether a token-classified clinical finding is negated or affirmed in narrative text. It uses regular expressions to identify negation triggers and their scope.

• Detects patterns like "patient denies [FINDING]"
• Identifies historical and hypothetical contexts
• Operates as a post-processing filter on NER output

Without negation detection, token classification alone would incorrectly extract "pneumonia" from "no evidence of pneumonia," leading to false positive clinical data capture.

The harmonic mean of precision and recall, serving as the primary evaluation metric for token classification systems. It balances the trade-off between missing entities and generating false positives.

• Precision: Of all tokens predicted as B-DRUG, what fraction are correct?
• Recall: Of all actual B-DRUG tokens, what fraction were found?
• F1: 2 × (Precision × Recall) / (Precision + Recall)

Clinical NER systems typically target F1 scores above 0.90 for high-stakes applications like medication extraction, where both false negatives and false positives carry patient safety risks.