Cascade Ranking

The foundational design pattern of cascade ranking, where a sequence of progressively more precise but computationally expensive models narrows the candidate set from millions to a final ranked list.

• Stage 1 (Retrieval): A lightweight Bi-Encoder or sparse model like BM25 retrieves a broad candidate set (e.g., top-1000) from the corpus with minimal latency.
• Stage 2 (Re-Ranking): A computationally intensive Cross-Encoder scores only the top-k candidates (e.g., top-100) with full query-document attention.
• Stage 3 (Optional): A pairwise DuoBERT or diversity re-ranker refines the final ordering of the top-10 documents.

This architecture ensures that the heavy full-attention computation is applied only to a tiny fraction of the corpus, making real-time semantic search feasible at scale.

Cascade ranking explicitly decouples the recall and precision objectives into separate stages, allowing each component to be optimized for its specific role.

• High Recall (Stage 1): The first-stage retriever is tuned to maximize recall@k, ensuring that all potentially relevant documents are included in the candidate set, even at the cost of including some irrelevant ones.
• High Precision (Stage 2): The Cross-Encoder re-ranker is trained to discriminate fine-grained relevance, pushing truly relevant documents to the top of the list and demoting false positives.
• Metric Alignment: The retriever is evaluated on recall@1000, while the re-ranker is evaluated on precision-oriented metrics like NDCG@10.

This separation prevents the system from being forced into a single model that is mediocre at both tasks.

The core advantage of the Cross-Encoder stage in a cascade is its ability to perform token-level interaction between the query and the candidate document.

• Full Self-Attention: Every token in the query attends to every token in the document simultaneously, enabling the model to weigh the importance of specific term alignments.
• Exact Match Signals: The Cross-Encoder can capture lexical overlap and exact keyword matches that are lost in the pooled vector representations of a Bi-Encoder.
• Contextualized Matching: The model understands that 'bank' in 'river bank' should not match 'financial bank' based on surrounding context, a distinction a single-vector Bi-Encoder often misses.

This granular semantic matching is what gives the cascade its final precision boost over a single-stage retrieval system.

The primary engineering challenge in cascade ranking is ensuring that the Cross-Encoder re-ranking stage does not violate the strict latency budgets of real-time search applications.

• INT8 Quantization: Reduces the precision of model weights and activations from 32-bit floating point to 8-bit integers, dramatically accelerating matrix multiplications on modern CPUs and GPUs.
• ONNX Runtime: Converts the PyTorch or TensorFlow model into an optimized computation graph that fuses operations and eliminates overhead.
• Dynamic Batching: Groups multiple query-document pairs into a single forward pass to maximize GPU utilization and amortize kernel launch overhead.
• Caching: Pre-computes and caches the document embeddings for the Bi-Encoder stage, and in some architectures, caches the key-value states for frequently accessed documents in the Cross-Encoder.

The effectiveness of a cascade ranking system depends critically on the quality of the training data for the Cross-Encoder re-ranker, specifically the selection of hard negative samples.

• Definition: Hard negatives are documents that receive a high score from the first-stage retriever but are actually irrelevant to the query.
• Mining Process: The current Bi-Encoder retriever is used to retrieve the top-k documents for a query. Documents that are not labeled as relevant but appear in this top-k are selected as hard negatives.
• Discriminative Training: Training the Cross-Encoder to correctly distinguish between true positives and these challenging hard negatives forces it to learn fine-grained relevance boundaries.
• Iterative Refinement: The process can be repeated, where the newly trained Cross-Encoder is used to mine even harder negatives for the next training iteration.

In a cascade, the final ranking often requires combining signals from multiple stages, necessitating score calibration to normalize disparate score distributions.

• The Problem: The raw logit from a Cross-Encoder is an uncalibrated confidence score that cannot be directly averaged with a BM25 lexical score or a Bi-Encoder cosine similarity.
• Platt Scaling: Fits a logistic regression model on a held-out validation set to map the raw Cross-Encoder logits to well-calibrated probabilities.
• Temperature Scaling: A simpler method that divides the logits by a learned temperature parameter to soften or sharpen the output distribution.
• Reciprocal Rank Fusion (RRF): A non-parametric alternative that combines ranked lists by computing a weighted sum of the reciprocal of each document's rank position, avoiding the need for score calibration entirely.