Late Fusion

In a Late Fusion architecture, the sparse index (e.g., BM25 over an inverted index) and the dense index (e.g., a vector database for embeddings) are built and queried as completely separate subsystems. Each pipeline processes the query independently, generating its own ranked list of document IDs and relevance scores without any cross-communication during the retrieval phase. This contrasts with Early Fusion, where sparse and dense signals are combined at the index level. The independence allows teams to scale, update, or even swap out one retriever without affecting the other, providing significant operational flexibility.

Because sparse and dense retrievers produce scores in fundamentally different ranges and distributions, a critical step in Late Fusion is score normalization. A BM25 score is an unbounded term-frequency statistic, while a cosine similarity score from a dense vector search is typically bounded between -1 and 1. Without normalization, one system's raw scores can dominate the merged ranking. Common techniques include:

• Min-Max Normalization: Rescales scores to a [0, 1] range.
• Z-Score Normalization: Standardizes scores based on mean and standard deviation.
• Rank-Based Normalization: Discards raw scores entirely and uses only the rank position.

When normalized scores are available, linear combination methods offer a straightforward fusion approach:

• CombSUM: The final score for a document is the simple sum of its normalized scores from all retrieval systems. A higher sum indicates stronger consensus.
• CombMNZ: This method multiplies the CombSUM score by the number of systems that retrieved the document. This explicitly boosts documents found by multiple independent sources, penalizing results that only appeared in a single list. CombMNZ is particularly effective when the retrievers are diverse and their agreement is a strong relevance signal.

A primary advantage of Late Fusion is the operational decoupling it provides. The dense vector index, which may require GPU-accelerated ANN libraries like FAISS or ScaNN, can be scaled horizontally on specialized hardware. Simultaneously, the sparse inverted index can run on optimized CPU-based search engines like Apache Lucene. Each subsystem can be tuned, re-indexed, or upgraded independently without downtime for the other. This modularity simplifies A/B testing of new embedding models against a stable BM25 baseline, as the pipelines do not share a critical runtime path.

The end-to-end latency of a Late Fusion system is bounded by the slowest independent pipeline. Since both the sparse and dense retrievals are executed in parallel, the total retrieval time is max(latency_sparse, latency_dense) plus the negligible time for the fusion algorithm. This fan-out pattern provides a latency advantage over serial multi-stage architectures. However, it doubles the computational load per query, as every search hits two full indexes. This trade-off is acceptable when precision gains from fusion outweigh the increased infrastructure cost.

The most common algorithm used in the final merge step of a Late Fusion architecture. RRF combines multiple ranked lists by summing the reciprocal of each document's rank position: score = Σ 1/(k + rank). It requires no score normalization, making it robust when sparse and dense systems produce scores on different scales. Documents appearing consistently near the top across both lists receive the highest fused scores.

A critical preprocessing step often required before Weighted Sum Fusion in Late Fusion pipelines. Because BM25 scores are unbounded and cosine similarity ranges from -1 to 1, raw scores cannot be directly combined. Techniques include:

• Min-Max Normalization: Rescales all scores to [0, 1]
• Z-Score Normalization: Centers scores around zero using standard deviation Without normalization, one retrieval system's score magnitude can dominate the merged ranking.

The architectural opposite of Late Fusion. In Early Fusion, sparse and dense signals are combined before or during retrieval rather than after. This often involves enriching sparse queries with semantic embeddings or using learned sparse representations. Early Fusion can be more computationally efficient but less modular—changing the dense encoder requires re-indexing, whereas Late Fusion allows independent subsystem iteration.

A cascading pipeline where Late Fusion often serves as the first-stage candidate generation step. The fused result set from sparse and dense retrieval is passed to a computationally expensive Cross-Encoder Re-Ranker for final precision scoring. This architecture balances the recall benefits of hybrid retrieval with the precision of full-attention scoring, applying the heavy compute only to the top N candidates.

The deduplication step that occurs immediately after sparse and dense indexes return their independent result sets. Each subsystem returns a list of document IDs with scores. The merger:

• Creates a union of all unique document IDs
• Preserves per-subsystem scores for downstream fusion
• May apply pre-filtering constraints (e.g., date ranges) before fusion Efficient merging is critical for latency in high-throughput search systems.

An advanced alternative to fixed formulas like RRF or weighted sums. A machine learning model is trained on labeled relevance data to learn optimal, non-linear combinations of sparse and dense scores. Features may include BM25 score, cosine similarity, document recency, and click-through rate. This approach can capture complex interaction patterns but requires ongoing training data and introduces model serving latency to the fusion step.