Cross-Encoder Re-ranking

Unlike bi-encoders that encode queries and documents independently, a cross-encoder processes the query-document pair as a single concatenated input sequence. This allows the model's self-attention mechanism to compute token-level interactions across both the query and the candidate document simultaneously, capturing fine-grained semantic relationships such as negation, word order, and context-dependent relevance that independent encoding architectures miss.

Cross-encoder re-ranking operates as the second stage in a cascade architecture:

• Stage 1 (Candidate Retrieval): A fast, scalable retriever — typically a bi-encoder with approximate nearest neighbor search — fetches the top-k candidates (e.g., k=100) from a large corpus.
• Stage 2 (Re-ranking): The cross-encoder scores each candidate jointly with the query, producing a precise relevance score used to reorder the list and return the top-n results (e.g., n=10). This division of labor balances recall from the retriever with precision from the re-ranker.

Cross-encoders are orders of magnitude slower than bi-encoders because they cannot pre-compute document embeddings. Each query requires a fresh forward pass for every candidate document. For a query with 100 candidates, the model runs 100 inferences. This makes cross-encoders impractical for full-corpus search but highly effective when constrained to a small candidate set. Typical latency ranges from 10-100ms per pair on GPU, compared to sub-millisecond vector comparisons in ANN search.

The cross-encoder outputs a scalar relevance score — typically a logit passed through a sigmoid activation — representing the probability that the document is relevant to the query. These scores are used to re-rank the candidate list in descending order. Unlike cosine similarity from bi-encoders, cross-encoder scores are calibrated to the specific query-document interaction, making them more reliable for threshold-based filtering and downstream decision logic.

Cross-encoders are typically fine-tuned from a pre-trained transformer (e.g., BERT, RoBERTa) on relevance-labeled query-document pairs. The training data consists of triples: (query, document, label) where labels are binary (relevant/non-relevant) or graded (e.g., 0-4 relevance scale). The model is trained with a cross-entropy loss or pairwise ranking loss (e.g., RankNet, LambdaRank) to optimize the ordering of documents by relevance.

Cross-encoder re-ranking is commonly deployed on top of hybrid retrieval pipelines that fuse results from dense vector search and sparse keyword retrieval (BM25) via Reciprocal Rank Fusion (RRF). The fused candidate set benefits from both semantic and lexical recall, while the cross-encoder provides a final, high-precision relevance judgment. This architecture is standard in production RAG systems and enterprise search platforms.

The first-stage retriever that cross-encoders are designed to refine. A bi-encoder encodes the query and all documents independently into dense vectors, enabling fast approximate nearest neighbor (ANN) search across millions of documents. Because there is no joint attention between query and document, relevance signals are weaker, producing a high-recall but lower-precision candidate set that requires re-ranking.

A retrieval paradigm that bridges the speed of bi-encoders and the precision of cross-encoders. ColBERT computes token-level embeddings for both query and document, then performs a cheap MaxSim operation at retrieval time. Unlike a full cross-encoder, the document representations can be pre-computed and indexed, avoiding joint re-computation. It serves as a direct architectural competitor to the two-stage bi-encoder + cross-encoder pipeline.

The upstream process that determines what a cross-encoder actually scores. Semantic chunking splits documents at natural topic boundaries rather than fixed character counts, ensuring each chunk passed to the re-ranker represents a coherent, self-contained idea. Poor chunking produces fragments that confuse the cross-encoder's joint attention mechanism, degrading re-ranking quality regardless of model sophistication.

A re-ranking algorithm that cross-encoders often replace or complement. MMR selects documents by balancing relevance to the query against novelty relative to already-selected documents, reducing redundancy. While MMR operates on existing similarity scores, a cross-encoder provides more accurate relevance signals, and the two are frequently combined: cross-encoder scores feed into an MMR diversity selection step.

An algorithm for merging ranked lists from multiple retrieval sources without requiring normalized scores. In a hybrid pipeline, RRF can combine the cross-encoder's re-ranked list with results from a sparse keyword retriever like BM25. Each document receives a score of 1 / (k + rank), and the fused ranking leverages the strengths of both semantic and lexical signals.

A query-side technique that can reduce the burden on cross-encoder re-ranking. HyDE instructs a language model to generate a synthetic ideal document in response to a query, then embeds that document for vector search. This bridges the semantic gap between short queries and long documents, potentially improving the quality of the initial candidate set and reducing the number of documents the expensive cross-encoder must process.