Cross-Encoder Reranker

Unlike a Bi-Encoder that encodes a mention and candidate entity independently, a Cross-Encoder processes the concatenated sequence [CLS] mention [SEP] entity description [SEP] through a transformer. Every token in the mention attends to every token in the entity description across all layers. This joint encoding captures fine-grained lexical and semantic interactions—such as exact name overlap, acronym expansion, and contextual clues—that independent vector dot products miss. The final [CLS] token representation is fed into a linear classification head to produce a single relevance score between 0 and 1.

Cross-Encoders are too computationally expensive to run against an entire knowledge base of millions of entities. The standard architecture is a cascade:

• Stage 1 (Retrieval): A fast Bi-Encoder or sparse retriever (BM25) generates a candidate set of ~100 entities using approximate nearest neighbor search.
• Stage 2 (Reranking): The Cross-Encoder scores each mention-candidate pair from the shortlist, reordering results by precision. This hybrid approach delivers the speed of dual-encoders with the accuracy of full attention, making it the dominant paradigm in systems like BLINK and modern search engines.

The input to a Cross-Encoder is a carefully formatted text pair. For entity linking, the construction typically follows a template:

• Mention in context: The sentence containing the ambiguous surface form, with the mention wrapped in special tokens like [START_ENT] and [END_ENT].
• Entity description: The canonical title, definition paragraph, and type information from the knowledge base.
• Optional features: Entity popularity scores, prior probabilities, or type constraints can be appended as additional token segments. This structured input teaches the model to compare the mention's local discourse context against the entity's global semantic profile.

Cross-Encoders are trained using binary cross-entropy loss on positive and negative mention-entity pairs. The quality of negative examples is critical:

• Random negatives: Easy to classify, provide weak learning signal.
• In-batch negatives: Other entities in the same training batch, more challenging.
• Hard negatives: Entities with high Bi-Encoder similarity scores that are incorrect. These are the most valuable, forcing the model to learn fine-grained disambiguation beyond surface-level similarity. Top-performing systems use the Stage 1 retriever to mine hard negatives, creating a curriculum learning effect where the Cross-Encoder specializes in the difficult cases the Bi-Encoder gets wrong.

A Cross-Encoder requires a full transformer forward pass for every single mention-candidate pair, making it O(N) in the candidate set size. Key optimization strategies include:

• Dynamic batching: Grouping pairs of similar sequence length to minimize padding waste.
• Knowledge distillation: Training a smaller student Bi-Encoder to mimic the Cross-Encoder's scores, trading a small accuracy drop for 100x speedup.
• Early exit: Using the score after only a few transformer layers for obviously correct or incorrect pairs.
• Quantization: Reducing model precision to INT8 for faster matrix multiplications on inference hardware. Typical latency is 5-20ms per pair on GPU, making it viable for top-100 reranking but prohibitive for full-corpus scoring.

The raw relevance score from a Cross-Encoder is not inherently calibrated for NIL prediction—determining when no knowledge base entity matches the mention. Effective strategies include:

• Threshold tuning: Setting a minimum score threshold on a held-out validation set to balance precision and recall for NIL decisions.
• Null entity pairing: Training with an explicit [NIL] entity description, teaching the model to assign high scores to this class when no real entity fits.
• Score distribution analysis: Comparing the score of the top candidate against the distribution of scores for the top-k; a low margin between rank 1 and rank 2 suggests ambiguity or an out-of-KB entity. Proper calibration ensures the reranker doesn't force a link when the correct entity is absent from the knowledge base.