Cross-Encoder Scoring

Unlike bi-encoders that encode queries and documents independently, cross-encoders concatenate the query and document into a single input sequence: [CLS] query [SEP] document [SEP]. This allows the transformer's self-attention mechanism to model token-level interactions across both sequences simultaneously. Every word in the query attends to every word in the document, capturing nuanced relationships like negation, paraphrasing, and term dependency that independent encodings miss.

The defining architectural advantage of cross-encoders is full cross-attention between query and document tokens. In a bi-encoder, the query vector and document vector are compared only via a final cosine similarity or dot product. In a cross-encoder, every transformer layer computes attention weights between all query-document token pairs, enabling the model to learn complex relevance signals such as:

• Term proximity: How close query terms appear to each other in the document
• Contextual negation: Detecting when a document mentions a term but negates its relevance
• Passage-level entailment: Whether the document logically supports the query's premise

Cross-encoders append a linear classification head on top of the [CLS] token representation from the final transformer layer. This head outputs a single scalar relevance score, typically passed through a sigmoid or softmax function. During training, the model is optimized with a pairwise or listwise ranking loss (e.g., RankNet, LambdaRank) that teaches it to assign higher scores to relevant documents than irrelevant ones. The output is a calibrated relevance probability between 0 and 1.

The primary limitation of cross-encoders is their O(n·m) complexity for a query of length n and document of length m. Self-attention scales quadratically with the combined sequence length. This makes cross-encoders impractical for first-pass retrieval over millions of documents. The standard architecture uses them as a re-ranker: a fast bi-encoder or BM25 retrieves the top-k candidates (typically 100-1000), and the cross-encoder re-scores only that subset. This two-stage pipeline balances precision and latency.

Cross-encoder performance depends critically on exposure to hard negative examples during training. Hard negatives are documents that are top-ranked by a simpler retrieval model but are actually irrelevant. Without these challenging examples, the cross-encoder learns only coarse distinctions. Common training strategies include:

• In-batch negatives: Treating other queries' relevant documents as negatives within a training batch
• Mined negatives: Using a bi-encoder to retrieve top candidates and labeling the false positives as hard negatives
• Distillation: Training a cross-encoder from a larger teacher model's relevance judgments

Popular cross-encoder implementations include:

• monoBERT: A BERT-based model fine-tuned on MS MARCO passage ranking data, serving as a strong baseline
• monoT5: A T5-based model that casts ranking as text generation, outputting 'true' or 'false' tokens whose probabilities become relevance scores
• RankLLaMA and RankZephyr: Recent open-source models applying decoder-only LLMs to zero-shot and few-shot re-ranking
• Cohere Rerank: A commercial API offering optimized cross-encoding with support for documents up to 4K tokens

A retrieval architecture that encodes the query and each document independently into dense vectors. This allows documents to be pre-indexed offline, enabling extremely fast approximate nearest neighbor (ANN) search over millions of items. The trade-off is lower precision because the query and document never interact directly within the model, making it ideal for first-pass candidate generation before cross-encoder re-ranking.

A cascading pipeline that balances speed and accuracy. It typically involves:

• Stage 1 (Retrieval): A fast bi-encoder or BM25 model fetches ~100-1000 candidates.
• Stage 2 (Re-ranking): A slower, high-precision cross-encoder scores only the top-k candidates. This architecture is the standard for modern search systems, ensuring that the heavy compute of cross-encoders is applied only to a small, promising subset of documents.

Pairwise scoring compares a single query-document pair at a time, which is the standard cross-encoder approach. Listwise scoring evaluates an entire list of documents simultaneously, modeling their relative order directly. While listwise methods can optimize for the final ranked list more holistically, pairwise cross-encoders remain dominant due to their simpler architecture and easier training on standard relevance datasets like MS MARCO.

A technique to transfer the high accuracy of a cross-encoder into a faster bi-encoder. During training, the cross-encoder acts as a teacher model, generating soft relevance labels for query-document pairs. The bi-encoder student model is then trained to mimic these scores. The result is a single-stage retriever that approaches cross-encoder quality but retains the sub-50ms latency of a bi-encoder, a critical optimization for production systems.

A hybrid paradigm that bridges bi-encoders and cross-encoders. Models like ColBERT encode the query and document independently but store multiple token-level embeddings per document. At query time, a rich but efficient MaxSim operation computes fine-grained token interactions. This provides much of the precision of a cross-encoder with a fraction of the latency, as document representations can still be pre-computed and indexed.

Raw cross-encoder scores are often uncalibrated logits that cannot be directly compared across different queries. Techniques like min-max normalization, sigmoid calibration, or Platt scaling transform these raw scores into well-behaved probabilities. This is essential in multi-stage pipelines where cross-encoder scores must be merged with BM25 or bi-encoder scores to produce a final, globally consistent ranking.