Two-Stage Retrieval

The fundamental principle is a sequential pipeline where each stage progressively refines the candidate set. Stage 1 uses a fast, lightweight retriever (e.g., bi-encoder, BM25) to scan the entire corpus and return a broad set of candidates (typically 100-1000). Stage 2 applies a computationally intensive re-ranker (e.g., cross-encoder) to this smaller subset, enabling deep semantic analysis that would be cost-prohibitive at scale.

• Stage 1 (Retrieval): Optimized for high recall with low latency.
• Stage 2 (Re-ranking): Optimized for high precision using full cross-attention.
• Trade-off: Accepts that some relevant documents may be missed in Stage 1 to achieve sub-second latency.

The two stages exploit a fundamental accuracy-efficiency trade-off in neural architectures. Bi-encoders encode queries and documents independently into dense vectors, enabling pre-computation and fast approximate nearest neighbor (ANN) search but lack token-level interaction. Cross-encoders process the query-document pair jointly through full self-attention, capturing fine-grained relevance signals at the cost of quadratic complexity.

• Bi-Encoder Latency: Microseconds via vector index lookup.
• Cross-Encoder Latency: Milliseconds per pair, requiring GPU inference.
• Synergy: The bi-encoder handles scale; the cross-encoder handles precision.

A critical design parameter is the recall budget—the number of candidates passed from Stage 1 to Stage 2. This directly governs the upper bound of end-to-end retrieval quality. If a relevant document is not in the candidate set, no re-ranker can recover it.

• Typical Depth: 100-1000 candidates for re-ranking.
• Recall Ceiling: Stage 1 recall at depth K is the maximum achievable final recall.
• Optimization: Use hard negative mining during bi-encoder training to push relevant documents into the top-K window.

After re-ranking, the system must produce a final ordering. This often involves score normalization and fusion if multiple signals are combined. The cross-encoder's relevance score may be blended with the original retrieval score, BM25 lexical score, or metadata-based authority signals.

• CombSUM: A simple linear combination of normalized scores.
• Reciprocal Rank Fusion (RRF): Merges ranked lists without requiring score calibration.
• Calibration: Temperature scaling can calibrate cross-encoder confidence before fusion.

Two-stage retrieval imposes a strict latency budget. Stage 1 must complete in single-digit milliseconds, often using vector databases with ANN indexes (HNSW, IVF). Stage 2 re-ranking is the dominant cost, requiring batched GPU inference on the candidate set. Infrastructure design must balance throughput and tail latency.

• Stage 1 Target: < 10ms for candidate generation.
• Stage 2 Target: < 100ms for re-ranking 100 candidates.
• Optimization: Continuous batching and model quantization reduce GPU overhead.

The two stages are often trained with knowledge distillation, where the cross-encoder acts as a teacher to the bi-encoder student. The cross-encoder's fine-grained relevance scores on query-document pairs are used as soft labels to train the bi-encoder, transferring interaction-level knowledge into a dual-encoder architecture.

• Teacher: Cross-encoder trained on graded relevance judgments.
• Student: Bi-encoder trained via margin-based or KL-divergence loss.
• Result: A faster retriever that better approximates deep relevance scoring.

The fast retriever that powers the first stage. Bi-encoders encode queries and documents independently into dense vectors, enabling pre-computed document indexing and rapid approximate nearest neighbor (ANN) search over millions of items. This independence sacrifices deep token-level interaction for sub-linear latency, making it the only viable architecture for initial candidate generation against large-scale corpora.

The precise re-ranker that powers the second stage. Cross-encoders process the query and document jointly through full self-attention, allowing every query token to attend to every document token. This deep interaction produces highly accurate relevance scores but is computationally prohibitive for full-corpus search. Applied only to the top-K candidates from Stage 1 to refine the final ordering.

A middle-ground architecture that bridges the efficiency gap between bi-encoders and cross-encoders. ColBERT indexes document token embeddings and computes MaxSim operations between query and document tokens at retrieval time. This late interaction preserves token-level expressiveness while maintaining practical indexing speed, often serving as a powerful single-stage or Stage 1.5 solution.

An unsupervised fusion algorithm that merges ranked lists from multiple retrieval sources into a single consensus ranking. RRF scores each document based on the reciprocal of its rank position across input lists: score = Σ 1/(k + rank). It requires no training data or score calibration, making it ideal for combining dense vector results with sparse BM25 results in hybrid retrieval pipelines.

The probabilistic lexical retriever that anchors the sparse leg of hybrid Stage 1 pipelines. BM25 ranks documents using term frequency saturation and inverse document frequency, excelling at exact keyword matching and rare term identification. It complements dense retrieval by catching precise terminology that semantic embeddings might miss, ensuring high recall for specialized queries.

A training optimization critical for Stage 1 retriever quality. Standard negative sampling produces easy negatives that don't challenge the model. Hard negative mining selects negative documents with high BM25 or embedding similarity to the query, forcing the bi-encoder to learn fine-grained distinctions. This dramatically improves the recall ceiling of the candidate generation phase.