Named Entity Recognition (NER)

NER is fundamentally framed as a sequence labeling task where each token in a text is assigned a class label. The most common encoding scheme is the BIO (Begin, Inside, Outside) or BILOU tagging format. For example, in the sentence 'Tim Cook visited London,' the tokens are labeled as B-PER, I-PER, O, B-LOC. This token-level approach allows the model to precisely identify multi-word entity spans like 'San Francisco' without relying on gazetteers. Modern systems use conditional random fields (CRFs) on top of neural encoders to model the dependencies between adjacent labels, preventing invalid sequences like an I-ORG tag directly following an O tag.

The shift from static word embeddings like Word2Vec to deep contextualized models represents the single largest performance leap in NER history. Architectures like BERT, RoBERTa, and DeBERTa generate token representations that are dynamically conditioned on the entire surrounding sentence. This allows the model to disambiguate polysemous terms: 'Apple' will have a different vector representation in 'Apple released a new iPhone' versus 'I ate a green apple.' Fine-tuning a pre-trained transformer on a labeled NER corpus remains the standard baseline, with models like Flair and spaCy's transformer pipeline providing production-ready implementations.

A critical characteristic of NER systems is their extreme sensitivity to domain shift. A model trained on newswire text like the CoNLL-2003 dataset will catastrophically fail when applied to clinical notes, legal contracts, or social media. Medical NER must recognize entities like MEDICATION, DOSAGE, and PROCEDURE, which are entirely absent from general-domain schemas. Adaptation strategies include:

• Fine-tuning on in-domain annotated data
• Data augmentation using synonym replacement and back-translation
• Prompt-based learning where the entity extraction task is reformulated as a text generation problem for large language models

Traditional NER assumes entities are non-overlapping flat spans, but real-world text frequently contains nested entities. For instance, in 'the University of California, Berkeley campus,' both [University of California, Berkeley] as an ORG and [Berkeley] as a LOC are valid. Handling nested structures requires moving beyond simple sequence labeling to architectures like:

• Layered/Biaffine decoders that predict spans independently
• Constituency parsing approaches that treat entities as tree nodes
• Generative models that output entities as a structured sequence This characteristic is essential for biomedical NER, where genes, proteins, and mutations are routinely nested within longer descriptive phrases.

Despite the dominance of deep learning, external knowledge integration remains a defining characteristic of high-precision industrial NER systems. A gazetteer—a structured dictionary mapping surface forms to entity types—provides a strong prior that neural models can leverage. For example, a financial NER system might use a gazetteer of all publicly traded company names and their ticker symbols. Integration methods include:

• Feature concatenation: Gazetteer match signals are embedded and concatenated to token representations
• Retrieval-augmented generation: Candidate entities are retrieved from a knowledge base and provided as context
• Hybrid architectures: Rule-based components handle high-precision dictionary lookups while neural models handle ambiguous or novel mentions

Emerging NER systems are characterized by their ability to recognize novel entity types without explicit training examples. This is achieved through instruction-tuned large language models like GPT-4 and specialized models like GLiNER. Instead of fine-tuning on labeled spans, the user provides a natural language definition of the target entity type. For example: 'Extract all medical conditions and their associated treatments from this clinical note.' This paradigm eliminates the need for costly annotation pipelines and enables rapid prototyping across diverse schemas. However, zero-shot NER still lags behind fine-tuned supervised models on established benchmarks, particularly for complex, domain-specific entity boundaries.