Named Entity Recognition (NER)

NER is fundamentally a token-level classification task where each word or subword in a sequence is assigned a label using schemes like BIO (Begin, Inside, Outside) or BILOU. Modern architectures employ transformer-based encoders (e.g., BERT, RoBERTa) that generate contextualized embeddings, followed by a linear classification layer or a Conditional Random Field (CRF) to model dependencies between adjacent labels. The CRF layer is critical for preventing impossible sequences, such as an I-ORG tag following an O tag.

NER systems range from coarse-grained to fine-grained entity detection. Standard categories include:

• Person (PER): Individuals, fictional characters.
• Organization (ORG): Companies, agencies, institutions.
• Location (LOC): Geopolitical entities, facilities, natural features.
• Miscellaneous (MISC): Events, products, laws. Domain-specific systems extend this to include drug names, gene sequences, patent numbers, or financial instruments, requiring custom ontologies aligned with the target knowledge graph.

Early NER systems relied heavily on gazetteers—curated lists of known entities—combined with hand-crafted rules. While fast, they fail on unseen entities and suffer from high maintenance costs. Contextual NER using deep learning models infers entity type from surrounding syntax and semantics, enabling generalization to novel mentions. Production systems often use a hybrid approach, where model predictions are augmented by a gazetteer for high-precision lookups of critical entities like internal product codes or legal entity identifiers.

NER is a prerequisite for Entity Linking (EL), which disambiguates a recognized mention (e.g., 'Washington') and maps it to a canonical identifier in a knowledge base (e.g., Q1223 for the city, Q30 for the president). This two-stage pipeline—recognition then disambiguation—is essential for constructing knowledge graphs. End-to-end systems are emerging that jointly perform NER and EL, using the knowledge base as a direct supervision signal to improve recognition of rare entities.

NER systems are evaluated using strict span-level matching. A prediction is correct only if both the entity's boundary (start and end tokens) and its type are exactly right. Key metrics:

• Precision: Of all entities predicted, how many are correct?
• Recall: Of all true entities, how many were found?
• F1 Score: The harmonic mean of precision and recall. State-of-the-art models on benchmarks like CoNLL-2003 achieve F1 scores exceeding 93%, though performance degrades significantly on out-of-domain text.

A model trained on newswire text (e.g., CoNLL, OntoNotes) performs poorly on specialized corpora like clinical notes or legal contracts. Domain adaptation techniques include:

• Fine-tuning a pre-trained language model on in-domain, labeled NER data.
• Weak supervision using labeling functions and gazetteers to generate noisy training data.
• Few-shot learning where the model learns from only a handful of annotated examples per entity type. The vocabulary shift and unique entity formats in specialized domains remain a primary engineering hurdle.