Entity-Aware BERT

Introduces dedicated entity tokens directly into the input sequence alongside word tokens. Unlike standard BERT, which treats entity names as plain text, models like LUKE and KnowBERT add a special token (e.g., [ENTITY]) that attends to and is attended by all word tokens. This allows the model to learn a joint representation of text and structured knowledge, grounding ambiguous mentions like 'Apple' in their correct context.

Modifies the self-attention mechanism to incorporate entity representations from an external knowledge base. KnowBERT integrates a Knowledge Attention and Recontextualization (KAR) component that uses an integrated entity linker to retrieve relevant entity embeddings and inject them into the attention computation. This allows the model to condition its predictions on factual world knowledge without requiring explicit entity annotations at inference time.

Combines standard masked language modeling with entity-centric tasks to learn a unified representation space. Common objectives include:

• Masked Entity Prediction: Predicting a masked entity based on its context and surrounding entities.
• Entity Descriptions: Training the model to predict the descriptive text of an entity, forcing it to encode factual knowledge.
• Hyperlink Prediction: Using Wikipedia hyperlinks as distant supervision to learn entity relatedness.

Generates dynamic, context-dependent representations for entities rather than using static embeddings. In LUKE, the final hidden state of an entity token is a function of the entire input context, meaning the representation of 'Paris' will differ when the text discusses 'Paris, France' versus 'Paris Hilton'. This resolves the polysemy problem inherent in static knowledge graph embeddings, providing a contextualized entity representation that is highly specific to the current mention.

Leverages the deep transformer architecture to perform fine-grained entity typing as an auxiliary task. By processing entity mentions with full bidirectional context, the model can assign highly specific types from a deep ontology (e.g., distinguishing a 'Politician' from an 'Athlete' for the mention 'Jordan'). This typing signal acts as a powerful regularizer, improving the quality of the learned entity representations and downstream disambiguation accuracy.

Introduced by Studio Ousia, LUKE treats entities as first-class tokens. It uses a novel entity-aware self-attention mechanism that computes attention scores over both words and entities independently, then fuses them. The model is pre-trained on a massive corpus of Wikipedia text aligned with Wikidata entities using a masked entity prediction objective.

• Entity tokens are embedded using Wikipedia anchor text statistics
• Pre-training objective: predict masked entities and words simultaneously
• Achieved SOTA on five entity-related benchmarks at release
• Handles both entity typing and relation classification natively

Developed by Allen AI, KnowBERT integrates multiple knowledge bases through a Knowledge Attention and Recontextualization (KAR) component. This module sits between transformer layers, injecting entity embeddings from a linked knowledge base and allowing the model to recontextualize word representations with structured facts.

• Uses WordNet and Wikipedia as integrated knowledge sources
• KAR module queries a KB and returns relevant entity vectors
• Supports plug-and-play knowledge base swapping
• Demonstrates strong gains on entity linking and relation extraction

Baidu's ERNIE (not to be confused with the later ERNIE 2.0/3.0 series) was one of the earliest BERT variants to incorporate entity-level masking. It masks multi-word entities as single units during pre-training, forcing the model to learn entity boundaries and semantic coherence rather than treating named entities as independent tokens.

• Phrase-level masking groups entity spans together
• Pre-training corpus includes Chinese Wikipedia and Baidu Baike
• Implicitly learns entity boundary detection
• Strong performance on Chinese NER and entity typing tasks

E-BERT takes a simpler architectural approach by concatenating entity embeddings directly to the token embeddings at the input layer. Entity embeddings are derived from a pre-trained knowledge graph embedding model like TransE, providing a dense vector representation of each entity's position in the knowledge graph structure.

• Input-level fusion: entity vectors appended to token embeddings
• Uses Wikidata5M for entity-annotated pre-training
• Lightweight modification to standard BERT architecture
• Effective for entity linking without major architectural changes

KEPLER unifies knowledge graph embedding and language modeling into a single objective. It encodes entity descriptions using a BERT-based text encoder, then uses the resulting [CLS] token as the entity embedding for link prediction tasks. This joint training creates entity representations that are both textually grounded and structurally coherent.

• Dual objective: masked language modeling + knowledge graph link prediction
• Entity embeddings are the [CLS] token of their description text
• Outperforms static KG embeddings on relation prediction
• Enables zero-shot entity linking via description similarity

K-BERT injects knowledge without modifying the core BERT architecture by constructing knowledge-enriched sentence trees. Relevant triples from a knowledge graph are inserted as additional branches into the sentence's syntax tree, and a soft-position encoding and visible matrix control attention flow to prevent knowledge noise from distorting the original meaning.

• Tree-based injection: triples become branches on the syntax tree
• Visible matrix restricts attention to semantically relevant paths
• No pre-training required: works with any fine-tuned BERT checkpoint
• Effective for domain-specific QA and text classification

A dense, low-dimensional vector that captures the semantic properties and relational context of a knowledge base entity. In entity-aware BERT architectures like LUKE, these embeddings are pre-trained jointly with word tokens, allowing the model to treat entities as first-class citizens in the attention mechanism rather than as post-hoc annotations.

• Learned from graph structure and entity descriptions
• Stored in a dedicated entity embedding matrix
• Updated during fine-tuning for downstream tasks

The NLP task of grounding ambiguous textual mentions to their unique canonical entries in a knowledge base. Entity-aware BERT models internalize this process by using an entity-aware self-attention mechanism that computes attention scores between word tokens and entity tokens simultaneously, enabling the model to resolve mentions using both local context and global entity coherence.

• Input: 'Apple released a new iPhone'
• Output: Apple_Inc. (not the fruit)

A structured semantic network where nodes represent real-world entities and typed edges represent their interrelationships. Entity-aware BERT models like KnowBERT integrate knowledge graphs by attending over entity descriptions stored in a knowledge base during the encoding process, enriching token representations with factual knowledge without requiring explicit graph traversal at inference time.

• Wikidata and Wikipedia are common sources
• Provides factual grounding for language models

A high-precision neural architecture that processes the concatenated text of a mention and a candidate entity jointly through full cross-attention. In entity-aware BERT systems, this is often used as a second-stage reranker after a fast Bi-Encoder retrieval step, allowing the model to compute deep contextual compatibility between the surrounding text and the entity's knowledge base description.

• Higher accuracy than Bi-Encoders alone
• Computationally more expensive per candidate pair

A specific form of entity linking that maps textual mentions to canonical Wikipedia page titles. This serves as both a benchmark task and a training objective for entity-aware BERT models. During pre-training, models like LUKE are trained on Wikipedia hyperlinks, learning to predict masked entities using the same masked language modeling objective applied to entity tokens.

• Standard evaluation benchmark for entity-aware models
• Leverages Wikipedia's dense hyperlink structure

A global disambiguation approach that resolves all mentions in a document jointly by maximizing semantic coherence among the resulting linked entities. Entity-aware BERT models enable this natively because their entity-aware attention mechanism allows entity representations to influence each other through the transformer's self-attention layers, capturing document-level entity coherence without separate graph algorithms.

• Contrasts with local, mention-by-mention disambiguation
• Improves accuracy on ambiguous short texts