Wikification

The initial phase identifies text spans likely to represent named entities. Modern systems use transformer-based token classification or span-based architectures to detect mention boundaries with high precision. Unlike simple dictionary lookups, neural approaches handle polysemous terms and nested entities—for example, distinguishing 'Washington' as a person, city, or sports team based on context. Tools like DBpedia Spotlight combine phrase spotting with part-of-speech tagging to filter out common nouns, while TagMe uses anchor text probabilities from Wikipedia's internal link graph to identify meaningful surface forms.

Once a mention is detected, the system retrieves a set of possible Wikipedia article matches. This relies on prior probability tables built from Wikipedia's extensive internal link structure—the same surface form 'Mercury' might map to the planet, the element, the Roman god, or the car brand. Candidate generation uses:

• Anchor text dictionaries mapping phrases to article titles
• Redirect and disambiguation page resolution
• Approximate string matching for spelling variations
• Acronym expansion using context clues The goal is high recall: the correct entity must be in the candidate set for disambiguation to succeed.

The core algorithmic challenge: selecting the correct entity from candidates. Systems compute semantic relatedness between each candidate and the surrounding context using:

• Vector similarity in embedding spaces like BERT or Wikipedia2Vec
• Graph-based coherence models that maximize collective entity agreement across a document
• Commonness priors—how often a surface form links to a specific entity in Wikipedia Advanced implementations like AIDA and REL use iterative collective disambiguation, where entity decisions influence each other until a globally coherent solution emerges. This mirrors how humans resolve ambiguity through discourse-level reasoning.

Not every mention corresponds to a Wikipedia article. Robust wikification systems must identify NIL entities—mentions that have no valid target in the knowledge base. This prevents forced, incorrect mappings. Techniques include:

• Threshold-based confidence scoring on disambiguation scores
• Binary classification heads trained to distinguish linkable from unlinkable mentions
• Emerging entity detection using temporal signals from news corpora Systems like BLINK and GENRE incorporate explicit NIL prediction, critical for real-world applications where new people, products, and events constantly appear outside Wikipedia's coverage.

Traditional pipeline approaches separate mention detection, candidate generation, and disambiguation into distinct stages. Modern systems like GENRE (Generative Entity Retrieval) unify these tasks using sequence-to-sequence transformers that directly generate entity names from input text. This autoregressive approach:

• Eliminates error propagation between pipeline stages
• Handles overlapping and nested mentions naturally
• Learns implicit entity representations without explicit candidate dictionaries Other architectures like LUKE and ERNIE inject entity embeddings directly into pretrained language models, enabling joint learning of entity linking alongside downstream tasks like relation extraction and question answering.

Advanced systems extend entity linking beyond English by leveraging Wikipedia's interlanguage links—explicit mappings between articles on the same entity across 300+ languages. Cross-lingual wikification enables:

• Zero-shot transfer: linking mentions in low-resource languages to English Wikipedia entities
• Multilingual knowledge graph population using language-agnostic entity identifiers
• Cross-lingual information retrieval where queries in one language retrieve documents in another Tools like mGENRE and X-EL use multilingual transformer encoders to project mentions from any language into a shared entity representation space, enabling true language-independent entity resolution.

The broader NLP task of connecting textual mentions to unique identifiers in a knowledge base. While wikification specifically targets Wikipedia, entity linking can target any authoritative catalog.

• Disambiguation: Resolves which 'Paris' (France vs. Texas) is meant
• NIL prediction: Identifies mentions with no corresponding KB entry
• End-to-end systems: Combine NER and linking in a single pass

The prerequisite step that identifies and classifies text spans into categories like PERSON, ORG, or GPE. Wikification cannot proceed without accurate NER to locate the mentions that need linking.

• Sequence labeling: Token-level classification using BIO tagging
• Span-based models: Predict entity boundaries directly
• Fine-grained NER: Extends beyond the standard 4-type ontology to 100+ types

Resolves pronouns and nominal references to their named antecedents. Critical for wikification because a single entity may be referenced by multiple surface forms across a document.

• Example: 'Apple announced... It will release... The Cupertino company'
• All three mentions should link to the same Wikipedia entity
• Improves entity salience scoring by consolidating distributed mentions

Low-dimensional vector representations of entities and relations that preserve structural properties of the graph. Used to improve wikification accuracy by providing semantic similarity signals.

• TransE: Models relations as translations in embedding space
• RotatE: Captures symmetric and compositional patterns
• Use case: Disambiguating entities based on their graph neighborhood proximity

Quantifies how contextually important an entity is within a document. Not all linked entities are equally relevant to the document's core topic.

• TF-IDF baseline: Frequency weighted by corpus rarity
• Graph centrality: Apply TextRank or PageRank to entity co-occurrence graphs
• Positional weighting: Entities in titles and opening paragraphs score higher
• Guides which wikification links should be surfaced in AI-generated summaries