Named Entity Recognition (NER)

The foundational step where raw text is decomposed into atomic units (tokens) and sentence boundaries. NER systems rely on precise segmentation to define the search space for entity spans. Errors here cascade downstream.

• Word Tokenization: Splitting on whitespace and punctuation.
• Subword Tokenization: Using algorithms like Byte-Pair Encoding (BPE) or WordPiece to handle out-of-vocabulary terms and morphologically rich languages.
• Sentence Splitting: Resolving ambiguous punctuation (e.g., periods in 'Dr. Smith Inc.') to establish context windows.

The task of identifying the start and end indices of a potential entity mention within a token sequence. This is often framed as a sequence labeling problem using BIO (Begin, Inside, Outside) or BILOU tagging schemes.

• B-Tag: Marks the beginning token of an entity.
• I-Tag: Marks tokens inside a multi-token entity.
• O-Tag: Marks tokens outside any entity.
• Span-based Models: Modern architectures directly predict entity spans without intermediate tagging, reducing error propagation.

The mechanism by which the model encodes the linguistic context surrounding a candidate entity. This disambiguates 'Apple' (ORG vs. FRUIT) based on surrounding tokens.

• Static Embeddings: Legacy models like word2vec provide a single vector per word, failing at disambiguation.
• Contextual Embeddings: Transformer models (BERT, RoBERTa) generate dynamic token representations where the vector for 'Apple' differs based on the sentence.
• Character-level CNNs: Capture morphological patterns like prefixes and suffixes useful for unknown words.

Assigning a semantic category to a detected entity span. Standard categories include PERSON, ORGANIZATION, LOCATION, and GPE (Geopolitical Entity). Domain-specific taxonomies extend this to drug names, patent numbers, or financial instruments.

• Flat Classification: A single label per entity (e.g., PERSON).
• Hierarchical Classification: Fine-grained typing where 'Barack Obama' is tagged as PERSON → POLITICIAN → PRESIDENT.
• Few-Shot Classification: Using prompt engineering to classify entities into novel categories without retraining.

The post-recognition step that resolves a textual mention to a unique, canonical entry in a knowledge base like Wikidata or Wikipedia. This transforms the string 'Paris' into the distinct entity ID Q90 (city) or Q4115189 (mythological figure).

• Candidate Generation: Retrieving possible matching entities using alias tables.
• Contextual Ranking: Scoring candidates by computing the semantic similarity between the document context and the entity's knowledge graph description.
• Nil Prediction: Classifying a mention as unlinkable if no suitable target exists in the reference knowledge base.

Identifying semantic relationships between pairs of recognized entities within a text. This builds structured triples (Subject-Predicate-Object) from unstructured prose.

• Predefined Relations: Extracting specific links like 'founded_by' or 'headquartered_in'.
• Open Information Extraction: Discovering arbitrary relation phrases without a fixed schema.
• Joint Extraction: Models that simultaneously perform NER and relation extraction to leverage mutual information and reduce error propagation between the two tasks.