Ensemble Retrieval

The core strength of an ensemble lies in combining systems with orthogonal failure modes. A typical ensemble fuses:

• Dense Vector Search: Captures semantic similarity and paraphrasing.
• Sparse Lexical Retrieval (BM25): Excels at exact keyword and entity matching.
• Metadata Filtering: Enforces hard constraints like date ranges or access controls. When one system fails due to vocabulary mismatch, another compensates, increasing overall recall.

Raw relevance scores from different retrievers are not directly comparable. A fusion normalization step is mandatory. Common techniques include:

• Min-Max Normalization: Scales scores to a [0,1] range based on observed minimums and maximums.
• Z-Score Normalization: Centers scores around a mean of 0 with a standard deviation of 1.
• Reciprocal Rank Fusion (RRF): Ignores absolute scores entirely, using only the rank position to calculate a merged score, making it robust to uncalibrated systems.

Not all retrievers are equally reliable for every query. Weighted Sum Fusion assigns a static or dynamic coefficient to each system's normalized score.

• Static Weights: Determined via offline evaluation on a golden dataset.
• Dynamic Weights: Predicted by a lightweight Intent Classification model that analyzes the query and adjusts weights in real-time (e.g., boosting BM25 weight for acronym-heavy queries). This allows the ensemble to adapt its behavior based on query characteristics.

Ensemble retrieval is often implemented as a Multi-Stage Retrieval pipeline to manage latency costs:

1. Candidate Generation: Each retriever independently fetches a broad set of candidates (e.g., top-100 each).
2. Fusion: Results are merged and deduplicated into a single candidate pool.
3. Re-ranking: A more expensive and precise Cross-Encoder Reranker scores the fused candidates to produce the final, highly accurate ordering. This architecture balances the high recall of the ensemble with the high precision of a final re-ranker.

A primary design goal is resilience. A single Dense Passage Retrieval (DPR) system may fail on a query requiring a specific, rare entity name. An ensemble mitigates this:

• Lexical Fallback: If the dense retriever returns low-confidence results, the BM25 component reliably finds the exact string match.
• Semantic Generalization: Conversely, if a keyword query yields zero results, the dense vector retriever can find a conceptually related document. This redundancy ensures a graceful degradation of performance rather than a catastrophic failure.

The success of an ensemble is measured primarily by its recall at the candidate generation stage. The goal is to ensure the relevant document exists somewhere in the fused pool before re-ranking.

• Recall@K: The standard metric, measuring the percentage of queries for which the correct answer is in the top-K fused results (e.g., Recall@100).
• Ablation Studies: Systematically disabling one retriever at a time to quantify its marginal contribution to overall recall. The final precision is then the responsibility of the downstream re-ranker.

The most common algorithm for merging ranked lists in an ensemble. RRF assigns a score to each document based on the reciprocal of its rank in each constituent system.

• Formula: score(d) = Σ 1 / (k + rank_i(d)), where k is a constant (typically 60)
• Key Benefit: Does not require score calibration between systems
• Use Case: Merging results from BM25 and dense vector search where raw scores are not comparable
• Behavior: Boosts documents that appear consistently across multiple lists, even if not at the very top of any single one

The critical pre-processing step required before weighted sum fusion. Since retrieval systems output uncalibrated scores on different scales, normalization maps them to a common range.

• Min-Max Normalization: Rescales scores to [0, 1] using the min and max from each result set
• Z-Score Normalization: Centers scores around zero with unit variance, useful when score distributions are Gaussian
• Rank-Sim Normalization: Converts scores to a uniform distribution based on rank position
• Pitfall: Poor normalization causes one system to dominate the ensemble, negating the benefit of diversity

A linear combination method that assigns a pre-defined weight to each retrieval system's normalized scores. The final score for a document is the weighted sum of its scores from all systems.

• Formula: final_score(d) = w1*s1(d) + w2*s2(d) + ... + wn*sn(d)
• Tuning: Weights are typically set via grid search on a validation set with ground-truth relevance judgments
• Example: A legal search system might weight BM25 at 0.7 for exact clause matching and dense retrieval at 0.3 for conceptual similarity
• Limitation: Linear combination cannot capture non-linear interactions between system scores

The overarching architecture pattern that ensemble retrieval fits into. A cascading pipeline where each stage trades recall for precision using progressively more expensive models.

• Stage 1 - Candidate Generation: Fast, high-recall retrieval using ANN and BM25 in parallel (the ensemble)
• Stage 2 - Re-ranking: A cross-encoder or ColBERT model scores the fused candidate set with fine-grained attention
• Stage 3 - Final Filtering: Apply metadata filtering and business rules to the top-N documents
• Latency Budget: Stage 1 typically targets < 100ms, Stage 2 < 500ms, Stage 3 < 50ms

The process of transforming raw model outputs into well-calibrated probabilities that reflect the empirical likelihood of relevance. Critical when ensemble components must contribute scores that are meaningfully comparable.

• Platt Scaling: Fits a logistic regression model on raw scores to output calibrated probabilities
• Isotonic Regression: A non-parametric method that learns a monotonic mapping from scores to probabilities
• Temperature Scaling: A single parameter T that softens or sharpens the softmax distribution of neural rankers
• Evaluation: Measured by Expected Calibration Error (ECE), which bins predictions and compares average confidence to accuracy

A supervised approach that trains a model to optimally combine multiple relevance signals from an ensemble. Instead of manually tuning fusion weights, LTR learns the combination function from labeled data.

• Pointwise: Predicts absolute relevance score for each query-document pair independently
• Pairwise: Learns to correctly order pairs of documents, e.g., LambdaMART
• Listwise: Optimizes a ranking metric like NDCG directly over the entire result list
• Features: Input features include BM25 score, cosine similarity, PageRank, click-through rate, and document freshness