Dense Passage Retrieval (DPR)

DPR employs two independent BERT-based encoders—one for queries and one for passages—that map text into fixed-size dense vectors. Unlike cross-encoders, the query and passage are encoded separately, allowing passage embeddings to be pre-computed and indexed offline. At inference time, only the query encoder runs, making retrieval extremely fast. The encoders are trained jointly using a contrastive loss that pulls relevant query-passage pairs together and pushes irrelevant ones apart in the embedding space.

DPR's training efficiency relies on a clever sampling strategy. Within a mini-batch of B query-passage pairs, each query's positive passage is paired with the other B-1 passages in the batch as negative examples. This reuses computation already performed during the forward pass, dramatically increasing the number of training examples without additional encoding cost. The technique is particularly effective when combined with one or two hard negative passages—high-ranking but irrelevant results retrieved by a BM25 system—which sharpen the model's ability to distinguish subtle semantic differences.

At retrieval time, DPR computes relevance as the dot product between the query embedding and all pre-indexed passage embeddings. Because exhaustive search over millions of vectors is computationally prohibitive, DPR relies on Approximate Nearest Neighbor (ANN) indexes—typically implemented via FAISS or ScaNN. These indexes use techniques like product quantization and inverted file systems to achieve sub-linear search times, often returning top-k results in < 50ms over billion-scale corpora while maintaining >99% recall relative to exact search.

DPR is trained to minimize Noise Contrastive Estimation (NCE) loss, which frames retrieval as a classification problem over a candidate set. The loss function maximizes the softmax probability of the positive passage against all negatives:

L(q, p⁺, P⁻) = -log( exp(sim(q, p⁺)) / Σ_{p∈{p⁺}∪P⁻} exp(sim(q, p)) )

This formulation directly optimizes for the retrieval objective—ranking the correct passage above all distractors—rather than a proxy task like next-sentence prediction. The temperature parameter in the softmax controls the concentration of probability mass.

A defining strength of DPR is its ability to generalize across domains without fine-tuning. A model trained on Natural Questions or TriviaQA can retrieve relevant passages from biomedical literature, legal documents, or technical manuals—domains never seen during training. This works because the dual encoders learn a universal semantic matching function rather than domain-specific lexical patterns. In benchmarks, DPR outperforms BM25 on out-of-domain datasets by 15-30% in top-20 retrieval accuracy, making it ideal for enterprise RAG systems with diverse knowledge bases.

While DPR excels at semantic matching, it can miss exact keyword matches critical for entity-heavy queries. Production systems often deploy hybrid retrieval, combining DPR's dense vectors with BM25's sparse lexical scores using Reciprocal Rank Fusion (RRF). RRF merges rankings without requiring score calibration:

RRF(d) = Σ_{r∈R} 1/(k + rank_r(d))

This approach captures both semantic similarity and precise term matching, yielding 5-10% improvements in top-5 accuracy over either method alone. The fusion weight k (typically 60) controls the influence of high-ranked items.