Cross-Encoder Re-Ranking

Unlike a Bi-Encoder that produces independent vector representations, a Cross-Encoder concatenates the query and candidate document into a single input sequence. Every token in the query attends to every token in the document simultaneously. This token-level interaction allows the model to capture exact match signals, lexical overlap, and subtle semantic relationships that dot-product similarity misses. The joint encoding produces a single relevance score, typically derived from the [CLS] token representation, offering significantly higher precision at the cost of computational latency.

Cross-Encoders are too slow to score millions of documents directly. They are deployed as a second-stage re-ranker in a multi-stage retrieval pipeline:

• Stage 1: A fast Bi-Encoder or sparse retriever (BM25) retrieves a broad candidate set of 100-1000 documents.
• Stage 2: The Cross-Encoder scores only these top-k candidates, re-ordering them with high precision. This architecture optimizes the latency-relevance trade-off, applying expensive computation only where it matters most.

Cross-Encoders can be trained with different objectives:

• Pointwise (MonoBERT): The model processes a single query-document pair and outputs a relevance probability. This is simple and effective for initial re-ranking.
• Pairwise (DuoBERT): The model takes a query and two documents simultaneously, predicting which is more relevant. This is often applied as a third refinement stage.
• Listwise: The model optimizes the entire ordering of a document list, directly maximizing metrics like NDCG using losses like ListMLE.

Training a Cross-Encoder requires carefully selected negative examples. Hard negative mining selects documents that receive high scores from the first-stage retriever but are actually irrelevant to the query. These samples sit near the decision boundary and force the model to learn fine-grained discriminative features. Without hard negatives, the model only learns to separate obviously irrelevant documents, failing to improve precision in the top ranks where it matters most.

The precision of a Cross-Encoder can be transferred to a faster Bi-Encoder through knowledge distillation. The Cross-Encoder acts as a teacher, generating soft score distributions or margin signals on training data. A lightweight student Bi-Encoder is trained to mimic these outputs using KL divergence loss. The resulting student model approximates the teacher's scoring accuracy while retaining the low-latency vector search capabilities of a Bi-Encoder, enabling deployment at scale.

Cross-Encoder inference is computationally expensive, but several techniques reduce latency for real-time search:

• INT8 Quantization: Reduces model precision to 8-bit integers, cutting memory and compute by 4x with minimal accuracy loss.
• ONNX Runtime: Graph-level optimizations fuse operations and eliminate redundant computations.
• Dynamic Batching: Groups multiple query-document pairs into a single forward pass to maximize GPU utilization.
• Token Pruning: Early exit strategies drop irrelevant tokens mid-network to reduce computation.

A multi-stage retrieval architecture that chains a fast, high-recall vector search index with a precise Cross-Encoder. The initial retriever narrows the candidate set from millions to hundreds, and the Cross-Encoder reranks only the top-k documents.

• Stage 1: Bi-Encoder or BM25 for high recall
• Stage 2: Cross-Encoder for fine-grained precision
• Trade-off: Balances latency against relevance quality

The mechanism where every query token attends to every document token simultaneously via the transformer's self-attention layers. Unlike a Bi-Encoder's independent encoding, this joint processing captures rich token-level interactions, exact match signals, and lexical overlap.

• Enables fine-grained semantic matching
• Computationally expensive: O(n²) complexity
• Produces a single relevance logit from the [CLS] token

A compression technique where a computationally expensive teacher Cross-Encoder transfers its full-attention scoring distribution to a lightweight student Bi-Encoder. The student learns to approximate the teacher's precision at a fraction of the inference cost.

• Loss Function: KL divergence between teacher and student score distributions
• Benefit: Deploy Bi-Encoder speed with Cross-Encoder accuracy
• Training Data: Pairs scored by the teacher model

A training data curation strategy that selects negative document samples which receive high scores from the current retriever but are irrelevant to the query. These challenging negatives force the Cross-Encoder to learn fine-grained discriminative boundaries.

• Source: Top-k results from BM25 or dense retrieval
• Impact: Prevents the model from relying on superficial lexical overlap
• Contrastive Loss: Maximizes margin between positive and hard negative scores

Normalized Discounted Cumulative Gain is the standard listwise evaluation metric for re-ranking. It measures the quality of a ranked list by discounting relevance gains logarithmically by position and normalizing against the ideal ranking.

• Heavily weights top-ranked precision
• Scale: 0.0 to 1.0 (perfect ranking)
• Formula: DCG / IDCG, where gain is discounted by log₂(position + 1)

The process of adjusting the raw logit outputs of a Cross-Encoder so that the resulting scores reflect true empirical relevance probabilities. Overconfident predictions are corrected using:

• Platt Scaling: Logistic regression on validation set logits
• Temperature Scaling: Dividing logits by a learned scalar T > 1.0
• Purpose: Enables meaningful score thresholds across diverse queries