Cross-Encoder

Unlike a Bi-Encoder that processes a query and document independently, a Cross-Encoder concatenates the query-document pair into a single input sequence. This allows the Transformer's self-attention mechanism to compute attention weights across every token of both inputs simultaneously. Early token-level interactions enable the model to capture fine-grained semantic relationships, such as exact phrase matching, negation, and term dependency, that are lost when encoding each input in isolation.

The Cross-Encoder outputs a single scalar relevance score (typically passed through a sigmoid or softmax layer) representing the semantic relatedness of the pair. This is fundamentally different from the cosine similarity of independently generated embeddings. The model learns to weigh complex lexical and semantic interactions directly, making it exceptionally effective for tasks requiring precise judgment:

• Passage re-ranking after initial retrieval
• Natural Language Inference (NLI)
• Duplicate question detection

The primary trade-off is quadratic inference cost. Because the Cross-Encoder requires both inputs to be present at runtime, it is impossible to pre-compute document representations offline. For a corpus of N documents, scoring every candidate against a query requires N forward passes through the full Transformer. This makes Cross-Encoders unsuitable for first-stage retrieval over large datasets. They are deployed exclusively as re-rankers on a small candidate set (e.g., top-100 results) pre-filtered by a Bi-Encoder or BM25.

To maximize discriminative power, Cross-Encoders are typically fine-tuned using hard negative mining. Training data consists of positive pairs (query, relevant document) and carefully selected negative pairs that are topically similar but irrelevant. This forces the model to learn subtle distinction boundaries. Common training objectives include:

• Binary cross-entropy for relevance/non-relevance classification
• Ranking losses like LambdaRank for listwise optimization Without hard negatives, the model may fail to distinguish between a truly relevant document and one that merely shares keywords.

Cross-Encoders are highly versatile and can be applied to both asymmetric and symmetric semantic matching. In asymmetric tasks like question-answering, the query and document have different structures and lengths. In symmetric tasks like paraphrase identification, the two inputs are semantically and structurally similar. The architecture adapts naturally because the attention mechanism does not assume any fixed relationship between the two input segments, treating them as a unified token sequence.

A common production strategy is to use a Cross-Encoder as a teacher model to distill knowledge into a faster Bi-Encoder student. The Cross-Encoder generates high-quality relevance scores on training triplets, and the Bi-Encoder is trained via MarginMSE or KL-divergence loss to mimic these scores in the embedding space. This transfers much of the Cross-Encoder's accuracy to an indexable architecture, bridging the gap between precision and scalability.

The foundational training objective for Cross-Encoders in retrieval tasks. The model is trained to assign a high relevance score to positive query-document pairs and a low score to negative pairs.

• Binary cross-entropy: Common for pointwise training with labeled relevance data
• Margin ranking loss: Ensures the positive pair score exceeds the negative by a specified margin
• Knowledge distillation: Cross-Encoders can be used as teachers to train faster Bi-Encoders via score distillation

The strategic selection of deceptively similar but irrelevant documents to train Cross-Encoders. Hard negatives force the model to learn fine-grained distinctions that simple random negatives cannot provide.

• Source: Top-ranked false positives from a Bi-Encoder retrieval pass
• Impact: Dramatically improves the Cross-Encoder's ability to discriminate near-miss candidates
• Dynamic mining: Re-mining hard negatives as the retriever improves during training

The weight-sharing architectural paradigm that underpins both Bi-Encoders and Cross-Encoders. In a Cross-Encoder, the Siamese structure is implicit—the same transformer processes the concatenated input pair.

• Weight tying: Both inputs are processed by identical parameters before interaction
• Contrast with Bi-Encoder: Cross-Encoder applies Siamese weights but allows early cross-attention
• Training efficiency: Shared weights reduce parameter count and improve generalization

The primary scoring metric for Bi-Encoder outputs, contrasted with the direct relevance logit produced by a Cross-Encoder. While Bi-Encoders measure vector alignment, Cross-Encoders output a calibrated relevance probability.

• Bi-Encoder: score = cos(E_query, E_doc)
• Cross-Encoder: score = σ(W · CLS_token + b)
• Calibration: Cross-Encoder scores are typically better calibrated for threshold-based decision making