Multi-Stage Retrieval Architecture

The fundamental architecture pattern where each stage applies a progressively more expensive and precise model to a shrinking candidate set. The pipeline typically flows: Candidate Retrieval → Bi-Encoder Scoring → Cross-Encoder Re-Ranking.

• Stage 1: Approximate Nearest Neighbor (ANN) search retrieves ~1000 candidates from millions in milliseconds
• Stage 2: Bi-Encoder dot-product scoring narrows to top-100
• Stage 3: Cross-Encoder full-attention scoring produces final top-10 ranking

This cascading design optimizes the latency-relevance trade-off by reserving expensive computation for only the most promising candidates.

A dual-tower neural architecture that independently encodes queries and documents into dense vector representations. The query tower and document tower produce embeddings that can be compared via cosine similarity or dot product.

• Document embeddings are pre-computed and indexed offline, enabling sub-millisecond lookup
• Query encoding happens at inference time with minimal latency
• Trained with contrastive loss using in-batch negatives to push relevant pairs closer and irrelevant pairs apart

Bi-Encoders sacrifice token-level interaction for speed, making them ideal for the high-recall first stage where the goal is to surface all potentially relevant documents.

A full self-attention transformer that processes the query and candidate document concatenated into a single input sequence. Every query token attends to every document token, enabling rich token-level interaction.

• Captures exact match signals, synonym relationships, and contextual nuance
• Outputs a single relevance logit via the [CLS] token representation
• Typically 10-100x slower than Bi-Encoder scoring per document pair

Applied only to the top-k candidates from earlier stages, the Cross-Encoder provides the fine-grained discrimination needed for high-precision final ranking.

A retrieval paradigm, exemplified by ColBERT, that delays costly query-document interaction to the final MaxSim computation step. Instead of full cross-attention, queries and documents are encoded into sets of contextualized token embeddings.

• Each query token finds its maximum similarity with any document token
• The sum of these maximums produces the relevance score
• Enables pre-computation of document token embeddings while preserving token-level matching expressiveness

Late interaction bridges the gap between Bi-Encoder efficiency and Cross-Encoder precision, offering a compelling middle ground in the multi-stage architecture.

The mechanism for combining ranked lists from multiple retrieval stages into a single coherent ordering. Reciprocal Rank Fusion (RRF) is the dominant algorithm, computing a weighted sum of the reciprocal of each document's rank position.

• RRF formula: score(d) = Σ 1/(k + rank_i(d)) where k is a constant (typically 60)
• Normalizes disparate score distributions without requiring calibration
• Can incorporate sparse lexical signals like BM25 alongside dense semantic scores

Effective fusion ensures that complementary signals from different retrieval paradigms are combined to maximize both recall and precision.

A compression technique where a computationally expensive teacher Cross-Encoder transfers its scoring distribution to a lightweight student Bi-Encoder. The student learns to approximate the teacher's precision at dramatically lower latency.

• Teacher generates soft labels (probability distributions) for query-document pairs
• Student trained with KL divergence loss to match teacher outputs
• Enables single-stage retrieval that approaches two-stage quality

Distillation collapses the multi-stage pipeline into a more efficient architecture while preserving much of the Cross-Encoder's discriminative power.