Candidate Generation

The most common approach uses a precomputed surface form dictionary mapping textual mentions to candidate entity IDs. This dictionary is built by mining anchor text from Wikipedia hyperlinks, page titles, and redirect pages. For the mention "Paris," the dictionary returns a list including the French capital, Paris Hilton, and Paris, Texas. The quality of this dictionary directly determines the upper bound of recall for the entire linking pipeline. Key considerations include:

• Handling case sensitivity and punctuation normalization
• Incorporating acronym expansion for mentions like "NLP"
• Managing out-of-vocabulary mentions not seen during dictionary construction

For zero-shot candidate generation where surface form dictionaries fail, dense retrieval using bi-encoder architectures encodes the mention context and entity descriptions into a shared vector space. Systems like Dense Passage Retrieval (DPR) compute the similarity between the mention embedding and pre-indexed entity embeddings using approximate nearest neighbor (ANN) libraries such as FAISS or ScaNN. This approach excels at:

• Handling rare or unseen surface forms not in any dictionary
• Capturing semantic similarity beyond exact string matching
• Scaling to knowledge bases with millions of entities via efficient indexing

A robust candidate generator maintains an alias table that maps multiple name variants to the same canonical entity. This includes transliterations across scripts (e.g., Москва → Moscow), common misspellings, historical names, and abbreviations. The table is constructed from:

• Redirect pages in Wikipedia that point to the canonical article
• Multilingual interwiki links connecting the same entity across language editions
• Phonetic encoding algorithms like Soundex or Metaphone for fuzzy name matching Without comprehensive alias coverage, recall drops sharply for entities with multiple common designations.

Each candidate in the generated shortlist is often assigned a prior probability P(e|m) — the likelihood that mention m refers to entity e based on frequency statistics alone, without considering context. These priors are derived from:

• Anchor text click-through rates in Wikipedia
• Entity popularity in large text corpora
• Domain-specific frequency distributions for vertical applications For the mention "Mercury," the planet has a higher prior in astronomy texts, while the element dominates in chemistry contexts. These priors serve as a strong baseline signal for the downstream disambiguation ranker.

When linking against massive knowledge bases with billions of entities, exhaustive comparison is infeasible. Blocking techniques partition the search space using cheap heuristics before candidate retrieval. Common blocking keys include:

• Token overlap between the mention and entity names
• Entity type constraints — only consider entities matching the NER type label
• Locality-sensitive hashing (LSH) on character n-grams Blocking dramatically reduces the number of pairwise comparisons while maintaining high recall, making candidate generation tractable at web scale.

A critical capability is nil prediction — recognizing when a mention refers to an entity absent from the target knowledge base. The candidate generator must signal this possibility rather than forcing a false link. Strategies include:

• Setting a confidence threshold on the top candidate's prior probability
• Generating a special NIL candidate with a learned representation
• Using open-world classification techniques that model the distribution of unseen entities Without proper NIL handling, systems silently introduce errors by linking mentions to incorrect but superficially similar entities.

String similarity algorithms that tolerate typographical errors, abbreviations, and formatting inconsistencies during dictionary-based candidate lookup. Essential when surface forms don't exactly match knowledge base labels.

• Levenshtein distance for edit-based similarity
• Jaro-Winkler optimized for name matching
• TF-IDF character n-grams for robust tokenization-free matching

A global optimization approach that jointly disambiguates all mentions in a document by maximizing semantic coherence among linked entities. Instead of independent candidate ranking, it evaluates the pairwise compatibility of entity assignments.

• Uses PageRank-style random walks over entity graphs
• Exploits topical consistency: linked entities should relate
• Graph neural networks learn entity co-occurrence patterns

The critical capability to recognize when a mention refers to an entity absent from the target knowledge base. Prevents false positive links that cascade into downstream errors. Systems output a special NIL identifier with a confidence threshold.

• Threshold-based: link only if top candidate score exceeds τ
• Learned NIL classifier trained on out-of-KB mentions
• Essential for real-world corpora with long-tail entities