Named Entity Recognition (NER)

The dominant approach frames NER as a token-level classification task using the BIO tagging scheme. Each token is labeled as Beginning, Inside, or Outside of an entity. This transforms the span detection problem into a sequence prediction task solvable by models like Conditional Random Fields (CRF) or fine-tuned transformers. The CRF layer adds a crucial structural constraint by modeling transition probabilities between adjacent labels, preventing invalid sequences like an I-ORG tag following an O tag.

Modern NER performance is driven by the quality of contextualized token representations. BERT-NER architectures fine-tune pre-trained transformers by feeding the final hidden states into a linear classification head. The key insight is that attention mechanisms capture long-range dependencies, allowing the model to disambiguate 'Washington' as a Person, Location, or Organization based on surrounding context. SpanBERT further improves this by pre-training on contiguous span masking rather than random tokens.

The final layer transforms model logits into valid entity predictions. Viterbi decoding is the optimal algorithm for linear-chain CRFs, efficiently computing the most probable label sequence using dynamic programming. For span-based models, decoding involves thresholding span scores and applying non-maximum suppression to remove overlapping predictions. Mention-Level F1 is the standard evaluation metric, requiring exact matches on both span boundaries and entity type—a single token offset counts as a complete miss.

High-precision systems often combine neural models with structured knowledge. Gazetteers—dictionaries of known entities stored in efficient trie data structures—provide lookup features that guarantee recall on critical terms. Fuzzy matching using Levenshtein distance catches typographical errors in entity mentions. Distant supervision automatically generates training data by aligning text against knowledge bases like Wikidata, while weak supervision uses programmatic labeling functions managed by frameworks like Snorkel to create noisy but scalable training signals.

The standard token-level annotation scheme for sequence labeling. Each token is tagged as B (Beginning), I (Inside), or O (Outside) of a named entity span.

• B-PER: First token of a person entity
• I-LOC: Continuation token of a location entity
• O: Token not part of any entity

This scheme enables models to learn entity boundaries and types simultaneously, forming the backbone of most flat NER systems.

A discriminative probabilistic graphical model that serves as the classic decoding layer in NER pipelines. A linear-chain CRF models the conditional probability of an entire label sequence given the input tokens.

• Captures transition dependencies between adjacent labels (e.g., I-PER cannot follow B-LOC)
• Uses Viterbi decoding to find the globally optimal tag sequence
• Prevents invalid BIO transitions that independent classifiers might produce

A modern NER paradigm that bypasses token-level BIO tagging entirely. Instead, the model directly enumerates and classifies arbitrary text spans as entities.

• Scores all possible start-end token pairs using biaffine or multiplicative attention
• Naturally handles nested entities without complex label schemes
• Architectures like Global Pointer and SpanBERT are purpose-built for this approach

The primary evaluation metric for NER that computes the harmonic mean of precision and recall based on exact span matches.

• A prediction is correct only if both the span boundaries and the entity type match the ground truth exactly
• Computed at the entity level, not the token level
• Standardized by the CoNLL-2003 shared task, which established the benchmark using PER, LOC, ORG, and MISC types

A method for automatically generating noisy labeled training data by aligning a text corpus with an existing knowledge base or gazetteer.

• Matches entity strings from a dictionary against raw text to create weak labels
• Produces high recall but low precision annotations due to ambiguity
• Often combined with weak supervision frameworks like Snorkel to manage label noise

The process of adjusting a NER model trained on a source domain (e.g., newswire) to perform well on a target domain (e.g., medical records) with distinct entity types and linguistic patterns.

• Few-shot NER uses minimal labeled examples in the target domain
• Multi-task learning jointly trains on NER and auxiliary tasks like relation extraction
• Cross-domain NER tackles zero-shot transfer where no target labels exist