Commonness

Commonness is the conditional probability P(e|s)—the likelihood that a specific surface form (s) refers to a particular entity (e) in a reference corpus. It is calculated by counting how often a hyperlink with anchor text 's' points to the Wikipedia page for entity 'e', divided by the total number of times 's' appears as anchor text. This provides a powerful, context-independent baseline for candidate entity ranking before any contextual analysis is performed.

The commonness score is derived from anchor text distributions in hyperlinked corpora:

• Primary Source: Wikipedia dump analysis (page-to-page links)
• Formula: commonness(e, s) = count(s → e) / count(s)
• Scale: Values range from 0.0 to 1.0, where 1.0 means a surface form is completely unambiguous
• Thresholding: Forms with commonness below a cutoff (e.g., < 0.05) are typically treated as highly ambiguous, triggering heavier reliance on contextual similarity for disambiguation

In modern Bi-Encoder and Cross-Encoder architectures, commonness serves as a critical filtering mechanism:

• Candidate Generation: Instead of scoring millions of entities, the system retrieves only the top-k entities by commonness for a given surface form
• Prior Fusion: The commonness score is often combined with entity embedding similarity via a weighted sum: final_score = α * commonness + (1-α) * contextual_similarity
• Fallback Strategy: When contextual signals are weak or noisy, the system defaults to the highest-commonness entity, mimicking human disambiguation behavior

A critical distinction in entity linking pipelines:

• Commonness is static and global—it does not change based on the surrounding text of a specific mention
• Contextual Similarity is dynamic and local—it measures the semantic relatedness between the mention's sentence and the entity's description
• Interaction: High commonness can override weak contextual signals for dominant entities (e.g., 'Java' almost always links to the programming language in tech articles, regardless of local context)
• Failure Mode: Over-reliance on commonness causes errors when a rare entity is discussed in a specialized document, necessitating robust Cross-Encoder Reranker models

Commonness probabilities are inherently tied to the training corpus:

• Wikipedia Bias: Probabilities reflect encyclopedic, not enterprise, link distributions. 'AWS' has high commonness for 'Amazon Web Services' in Wikipedia but may refer to 'American Welding Society' in an industrial corpus
• Temporal Drift: Anchor text distributions become stale as new entities emerge. A 2019 dump will have zero commonness for 'COVID-19'
• Mitigation: Production systems must periodically recalculate commonness tables from fresh dumps or use distant supervision to generate domain-specific priors from proprietary document collections

Commonness directly informs NIL prediction—the decision that a mention has no valid entity in the target knowledge base:

• Low Commonness Threshold: If the highest commonness score for a surface form falls below a configured threshold (e.g., 0.01), the system may flag the mention as a potential Out-of-KB Entity (OOKB)
• Confidence Calibration: The final linking confidence score is often a function of both commonness and contextual similarity; a low commonness prior increases the burden of proof on the contextual model
• Edge Case: A surface form with zero commonness (never seen in training) triggers a pure contextual retrieval path, relying entirely on entity embedding similarity