Cross-Encoder

Unlike bi-encoders that encode queries and documents in isolation, cross-encoders concatenate the query and document into a single input sequence. This allows every token in the query to attend to every token in the document through the transformer's self-attention layers. The result is a rich, token-level interaction that captures subtle semantic relationships, negations, and dependencies that independent encoding misses. This mechanism is the primary reason cross-encoders achieve superior accuracy on relevance benchmarks.

The input format is typically structured as [CLS] query [SEP] document [SEP]. The model processes this entire sequence simultaneously through multiple transformer layers. The final hidden state of the [CLS] token is fed into a linear classification head to produce a single relevance score between 0 and 1. This joint processing means the model cannot pre-compute document representations; every query-document pair requires a full forward pass, making it computationally expensive for large-scale retrieval but ideal for precise re-ranking of top candidates.

Cross-encoders are almost never used for initial retrieval over millions of documents due to their O(N) inference cost. The standard architecture is a two-stage cascade:

• Stage 1 (Retriever): A fast bi-encoder or BM25 fetches the top-K candidates (e.g., K=100 or 1000).
• Stage 2 (Re-ranker): A cross-encoder scores each candidate pair and re-orders the list by relevance. This combines the speed of approximate nearest neighbor search with the precision of deep interaction, optimizing the latency-accuracy trade-off for production systems.

To maximize discriminative power, cross-encoders are trained with carefully selected hard negatives—documents that are topically similar to the query but ultimately irrelevant. Standard training uses binary cross-entropy loss on positive and negative pairs. Advanced strategies include:

• Mining negatives from the top results of a bi-encoder retriever.
• Using Knowledge Distillation where a cross-encoder teacher scores pairs to train a more efficient student model.
• Incorporating graded relevance labels for nuanced scoring beyond binary relevance.

A single cross-encoder forward pass on a query-document pair typically takes 10-50 milliseconds on GPU, compared to < 1 millisecond for a pre-computed bi-encoder dot product. When re-ranking a candidate set of 100 documents, total latency can range from 1 to 5 seconds without optimization. Mitigation strategies include:

• Batching multiple pairs into a single GPU inference call.
• Model distillation into smaller architectures like MiniLM.
• Early exiting where low-confidence pairs are pruned after intermediate transformer layers.

Raw logit outputs from the classification head can be poorly calibrated, producing overconfident scores. Temperature Scaling applies a learned parameter T to the softmax function: softmax(logits / T). A T > 1 softens the probability distribution, improving the reliability of the relevance score. This is critical when cross-encoder scores are fused with other signals (e.g., BM25 scores or freshness boosts) in a final ranking formula, ensuring the scores represent true posterior probabilities.