Named Entity Recognition (NER)

NER systems operate by mapping surface-form text strings to a predefined set of entity classes. While generic models often use the CoNLL-2003 categories (PER, ORG, LOC, MISC), production systems in specialized domains rely on custom ontologies.

• Domain-Specific Labels: Medical NER uses UMLS semantic types (e.g., Disease or Syndrome, Pharmacologic Substance).
• Fine-Grained Typing: Modern systems go beyond coarse types to distinguish Politician from Actor or Country from City.
• Hierarchical Taxonomies: Entities are often nested within a parent-child structure, allowing queries at different levels of specificity.

NER is fundamentally a token-level classification task. The system processes an input sequence and assigns a label to each token using a tagging scheme that captures multi-word entity boundaries.

• BIO / IOB2 Tagging: The standard scheme uses B- (Beginning), I- (Inside), and O (Outside) prefixes. For example, "San Francisco" is tagged as B-LOC I-LOC.
• BILOU Scheme: A more expressive variant adding L- (Last) and U- (Unit) tags for single-token entities, providing explicit boundary information.
• Contextual Ambiguity: The same word can have different labels depending on context (e.g., "Washington" as PER vs. LOC).

Modern NER relies on deep transformer models that generate contextualized word representations, moving beyond static word vectors like Word2Vec. The meaning of a token is a function of its surrounding context.

• BERT-based Encoders: Models like BERT, RoBERTa, and DeBERTa process text bidirectionally, allowing the representation of "Apple" to differ based on whether the context is "ate an apple" or "Apple released a new iPhone."
• Subword Tokenization: Using WordPiece or SentencePiece, models handle out-of-vocabulary terms by breaking them into known subword units, crucial for scientific or technical entities.
• Domain-Adaptive Pre-training: Generic language models are further trained on domain-specific corpora (e.g., biomedical papers, legal briefs) to learn specialized entity distributions.

While token-level classifiers make independent decisions, a Conditional Random Field (CRF) layer is often stacked on top to model the dependencies between adjacent output labels.

• Constraint Learning: The CRF learns transition probabilities, such as the fact that an I-PER tag cannot follow a B-ORG tag. This prevents structurally invalid entity sequences.
• Global Optimization: Instead of greedy per-token prediction, the CRF finds the globally optimal tag sequence for the entire sentence using the Viterbi algorithm.
• Hybrid Architectures: The standard high-accuracy recipe is a BiLSTM or Transformer encoder feeding into a linear-chain CRF decoder.

High-fidelity NER pipelines do not stop at classification; they perform entity linking (EL) to disambiguate the detected mention against a canonical knowledge base entry.

• Wikidata Q-Node Resolution: The string "Paris" is resolved to either Q90 (the capital of France) or Q11204 (the mythological figure) based on context.
• Knowledge Graph Grounding: This step is critical for transforming unstructured text into structured, queryable knowledge graph triples.
• End-to-End Systems: State-of-the-art models like GENRE perform retrieval-augmented generation to directly output canonical entity IDs in an autoregressive fashion.

NER systems are evaluated using the strict exact-match F1 score, which is unforgiving to boundary errors. A prediction is only correct if both the span boundaries and the entity type match the ground truth perfectly.

• Precision: The percentage of predicted entities that are correct.
• Recall: The percentage of ground truth entities that were successfully predicted.
• F1 Score: The harmonic mean of precision and recall, balancing the trade-off between missing entities and generating false positives.
• Human Parity: On standard newswire datasets like CoNLL-2003, state-of-the-art models now exceed human annotator F1 scores (approx. 97% vs. 96%).

The practice of anchoring language model outputs to structured, verifiable facts stored in a knowledge graph. NER serves as the extraction layer that identifies entities in unstructured text, which are then validated against a graph database. This entity-to-graph alignment provides a factual backbone for generative AI, ensuring that claims about persons, organizations, and locations are traceable to authoritative records.

The computational task of resolving which specific real-world entity a textual mention refers to when the name has multiple meanings. For example, 'Apple' could be the technology company or the fruit. Modern disambiguation uses contextual embeddings from transformer models to analyze surrounding words, achieving high accuracy by understanding that 'Apple released a new iPhone' refers to the organization, not the produce.

A predefined, restricted list of authorized terms used to ensure consistent entity tagging across a content ecosystem. Unlike open-domain NER, which classifies entities into broad categories (PERSON, ORG, GPE), a controlled vocabulary enforces domain-specific precision. In medical NER, for instance, entities must map to SNOMED CT or ICD-10 codes rather than generic 'medical condition' labels.

Algorithmic evaluation of the quality and trustworthiness of sources cited by an AI system. NER plays a foundational role by extracting the authors, publishers, and institutions mentioned in a source document. These extracted entities are then cross-referenced against reputation databases to compute a composite trust score, helping retrieval-augmented generation systems prioritize high-authority references over low-quality content.