Multi-Stage Retrieval

Multi-stage retrieval operates as a sequential funnel where each successive stage is more computationally expensive but more precise than the last. The pipeline typically begins with a lightweight candidate generation phase using fast vector search or inverted index lookup, then passes the top-k results to a re-ranking stage using cross-encoders or more sophisticated models.

• Stage 1 (Recall): Retrieves 100-1000 candidates using bi-encoders or BM25
• Stage 2 (Re-rank): Scores top candidates with a cross-encoder
• Stage 3 (Optional): Applies metadata filtering or business logic

The initial retrieval stage prioritizes high recall over precision, casting a wide net to ensure no relevant documents are missed. This stage typically employs bi-encoders for dense vector search or BM25 for sparse lexical matching, both optimized for speed via Approximate Nearest Neighbor (ANN) algorithms like HNSW.

• Latency target: typically < 50ms for the first pass
• Index structures are pre-built offline for sub-linear search
• Trade-off: accepts some noise to guarantee relevant candidates survive

The downstream re-ranking stage applies a cross-encoder that processes the query and candidate document jointly through full transformer attention, producing a fine-grained relevance score. Unlike bi-encoders that encode independently, cross-encoders capture nuanced semantic interactions at the cost of being too slow to run over the entire corpus.

• Cross-encoder models: BERT-based architectures fine-tuned for relevance
• Computes query-document pair scores with full cross-attention
• Typically re-ranks 50-200 candidates from the first stage

When multiple retrieval sources contribute candidates, their raw scores must be normalized onto a common scale before merging. Techniques include Min-Max normalization, Z-Score normalization, or rank-based fusion like Reciprocal Rank Fusion (RRF) which avoids score calibration entirely by using only rank positions.

• RRF: Assigns score = 1/(k + rank) for each result list
• Weighted Sum Fusion: Combines normalized scores with learned weights
• Critical for combining dense, sparse, and metadata-filtered results

Each stage in the pipeline consumes a portion of the total latency budget, requiring careful allocation to meet end-to-end response time targets. Techniques like caching frequent query embeddings, pre-computing document representations, and using Matryoshka embeddings for flexible dimensionality reduction help stay within budget.

• Stage 1 budget: 30-50ms for ANN search
• Stage 2 budget: 100-300ms for cross-encoder re-ranking
• Caching strategies: Query embeddings, popular results, session context

Training effective multi-stage systems requires careful negative sampling strategies. Hard negative mining identifies documents that are superficially similar to the query but irrelevant, forcing embedding models to learn finer-grained distinctions. These hard negatives are often sourced from the first-pass retrieval results of a previous model iteration.

• In-batch negatives: Reuse other queries' positives as negatives
• Hard negatives: Top-ranked irrelevant docs from current model
• Critical for preventing collapse in bi-encoder training