Contextual Similarity

Unlike static prior probability or commonness, contextual similarity is computed on-the-fly for each specific occurrence. It evaluates how well the words surrounding a mention align with the semantic profile of a candidate entity.

• Mechanism: Encodes the mention's context window and the entity's description into a shared vector space.
• Contrast: Static priors ignore the sentence; contextual similarity adapts to the specific usage.
• Example: The mention 'Apple' in 'Apple released a new chip' scores higher for the technology company than the fruit based on the surrounding terms 'released' and 'chip'.

A Bi-Encoder architecture independently encodes the mention context and each candidate entity description into dense vectors. The dot product between these vectors provides a fast, scalable contextual similarity score for initial candidate filtering.

• Efficiency: Enables sub-linear search over millions of entities using FAISS indexing.
• Representation: Uses pooled output from a transformer like BERT to create a fixed-size entity embedding.
• Trade-off: Sacrifices some precision for speed, as the context and entity are not jointly modeled.

A Cross-Encoder reranker computes a high-fidelity contextual similarity score by processing the concatenated mention context and entity description through full cross-attention. This captures deep, token-level interactions.

• Architecture: Feeds the pair [CLS] mention context [SEP] entity description [SEP] into a transformer.
• Output: A single scalar relevance score representing the true semantic fit.
• Usage: Applied only to the top-K candidates from a Bi-Encoder due to its computational cost.

The definition of the 'context' is a critical hyperparameter. It can range from a fixed token window around the mention to the entire document, and its composition directly impacts the similarity signal.

• Local Context: A window of 50-100 tokens surrounding the mention, capturing immediate syntactic cues.
• Document-Level Context: The full text, often summarized or chunked, providing a global topical signal.
• Coherence: In collective entity linking, the context of one mention includes the disambiguated entities of other mentions in the same document.

The candidate entity's side of the similarity equation is its textual description. The quality and structure of this description are paramount for accurate scoring.

• Sources: Wikipedia abstracts, structured knowledge graph properties, or concatenated aliases.
• Formatting: Descriptions are often prepended with the entity title for emphasis: 'Apple Inc. American multinational technology company...'
• Zero-Shot Linking: Enables linking to unseen entities by relying solely on their descriptive text, without any training examples.

Contextual similarity is rarely used in isolation. It is typically combined with a static prior probability to form a robust final score, balancing general popularity with specific local relevance.

• Formula: Final Score = log(P(entity|surface_form)) + Contextual_Similarity(mention, entity)
• Role: The prior acts as a strong baseline, while the contextual score provides the disambiguating signal.
• NIL Prediction: A low combined score across all candidates triggers a NIL prediction, indicating the entity is not in the knowledge base.

A high-precision model used to refine the contextual similarity scores of a shortlist of candidates. Unlike a Bi-Encoder, a Cross-Encoder processes the concatenated text of the mention and a single candidate entity jointly through full transformer attention. This allows the model to capture deep, token-level interactions between the context and the entity description, producing a highly accurate relevance score. Due to its computational cost, it is applied only to the top-k candidates retrieved by a faster Bi-Encoder.

A dense, low-dimensional vector representation of a knowledge base entity. These embeddings are trained to capture the semantic properties of the entity's description, types, and relationships. In a contextual similarity framework, the system compares the embedding of a textual mention's context against these pre-computed entity embeddings. The quality of these embeddings is paramount, as they define the semantic space in which similarity is measured.

The dominant training paradigm for models that compute contextual similarity. The model is trained on pairs of mentions and entities. It learns to pull the embeddings of a correct mention-entity pair close together in vector space while pushing apart the embeddings of incorrect pairs. This process teaches the model to distinguish subtle contextual differences, such as differentiating between 'Michael Jordan the athlete' and 'Michael Jordan the machine learning researcher'.

The core task that relies on contextual similarity. Disambiguation is the process of resolving the correct identity of an ambiguous mention by analyzing its surrounding text. The system computes the contextual similarity score between the mention and each candidate entity, typically selecting the entity with the highest score. A robust similarity metric is the primary mechanism for solving ambiguity.

The capability to link mentions to entities that were never seen during training. This is achieved by relying entirely on the contextual similarity between the mention's text and the textual description of the unseen entity. The model must learn a robust similarity function that generalizes to new entities based on their semantic descriptions alone, without relying on memorized entity IDs.