Wikification

The initial phase of wikification involves span detection to identify text segments that refer to entities or concepts. This step uses Named Entity Recognition (NER) and noun phrase chunking to locate candidate mentions.

• Identifies surface forms like 'Apple' (company vs. fruit)
• Handles multi-word expressions and nested entities
• Filters out common words and non-entity phrases
• Often uses POS tagging to constrain candidate spans

Example: In 'Jobs founded Apple in Cupertino,' the system detects 'Jobs,' 'Apple,' and 'Cupertino' as candidate mentions.

For each detected mention, the system generates a set of possible Wikipedia article candidates. This leverages anchor text dictionaries built from Wikipedia's internal link structure and search indices over article titles and redirects.

• Uses prior probability P(entity|mention) from Wikipedia link statistics
• Considers redirect pages and disambiguation pages
• Employs approximate string matching for spelling variations
• Generates candidates from cross-lingual mappings when needed

The mention 'Mercury' might generate candidates for the planet, element, Roman god, and car brand.

The core algorithmic challenge: selecting the correct Wikipedia entity from candidates by analyzing semantic context. Modern systems compute cosine similarity between the mention's surrounding text and candidate article embeddings.

• Uses Transformer-based encoders to create contextual embeddings
• Compares document context against entity description vectors
• Applies graph-based collective disambiguation across all mentions
• Leverages coherence models that prefer topically related entities

Example: 'Mercury orbits closest to the Sun' disambiguates to the planet via astronomical context words.

A critical capability where the system determines that a mention does not correspond to any Wikipedia entity. This prevents forced incorrect links and maintains annotation quality.

• Uses a confidence threshold on disambiguation scores
• Trains classifiers on features like mention commonness and context fit
• Essential for handling emerging entities not yet in Wikipedia
• Prevents over-linking of generic terms and common nouns

Proper nil prediction avoids linking 'the company' to a specific company entity when no clear referent exists.

Wikification systems often incorporate fine-grained entity typing to improve disambiguation accuracy. Knowing a mention refers to a PERSON, ORG, or GPE constrains the candidate space.

• Uses hierarchical type taxonomies like FIGER or Ultra-Fine Entity Typing
• Filters candidates by type compatibility with the mention's predicted type
• Improves precision for ambiguous mentions with different type profiles
• Integrates with Wikidata type hierarchies for richer constraints

'Washington' typed as GPE eliminates the person candidate, leaving the city or state.

Traditional wikification systems rely heavily on features derived from Wikipedia's hyperlink graph. These statistical signals provide strong baselines before neural methods are applied.

• Commonness: How often a mention links to a specific entity
• Relatedness: Normalized Google distance or Jaccard similarity between candidate sets
• Context quality: TF-IDF overlap between source text and entity article
• Coherence: Pairwise semantic relatedness among all disambiguated entities in a document

These features feed into ranking SVMs or gradient-boosted trees for final entity selection.