Dense Passage Retrieval (DPR)

DPR employs a bi-encoder design with two independent BERT-based encoders: one for questions and one for passages. This separation allows passage embeddings to be pre-computed offline and indexed via FAISS, while only the query is encoded at runtime. This decoupling is the key to achieving sub-second latency on billion-scale corpora, as the computationally expensive passage encoding happens asynchronously during ingestion.

DPR is trained using a contrastive loss function that maximizes the similarity between a question and its correct passage while minimizing similarity with all other passages in the batch. The critical innovation is in-batch negative sampling: every other passage in the training batch serves as a negative example for a given question. This technique dramatically increases training efficiency by reusing computed embeddings, eliminating the need for an external negative mining pipeline. The model learns fine-grained distinctions between topically similar but irrelevant passages.

Unlike cross-encoders that require joint computation, DPR measures relevance using a simple dot product between the query vector and pre-computed passage vectors. This linear operation is the fastest similarity function available and is natively supported by Maximum Inner Product Search (MIPS) algorithms in FAISS. The dot product effectively captures directional alignment and magnitude, providing a strong signal for semantic relevance without the computational overhead of cosine normalization during retrieval.

DPR training relies on supervised datasets where each question is paired with a single relevant 'gold' passage extracted from a larger document corpus like Wikipedia. The model is not trained to retrieve documents broadly but to pinpoint the exact passage span that answers the question. This fine-grained supervision teaches the encoder to distinguish between a document that merely mentions a topic and a passage that directly addresses the specific information need, significantly boosting precision@k metrics.

DPR is designed for tight integration with FAISS (Facebook AI Similarity Search) for production deployment. After offline batch vectorization, passage embeddings are loaded into a FAISS index—typically an IVFPQ or HNSW structure—that enables approximate nearest neighbor search with logarithmic time complexity. This allows DPR to scale to corpora containing billions of passages while maintaining retrieval latency suitable for real-time question answering systems.

A DPR model trained on general-domain data like Natural Questions or TriviaQA exhibits strong zero-shot retrieval performance on unseen domains without fine-tuning. The dense representations capture transferable semantic patterns that generalize beyond the training distribution. This property makes DPR a practical off-the-shelf retriever for enterprise applications where labeled in-domain QA pairs are scarce, serving as a strong baseline before domain adaptation.

The dual-tower neural architecture that independently encodes queries and passages into separate dense vectors. Unlike cross-encoders, bi-encoders allow pre-computation of passage embeddings offline, enabling fast approximate nearest neighbor (ANN) lookups at query time. The trade-off is reduced interaction-based precision since the query and passage never attend to each other directly during encoding. DPR uses two independent BERT-based encoders—one for queries, one for passages—trained with contrastive learning to pull relevant pairs together and push irrelevant pairs apart in the shared vector space.

A training efficiency technique where other passages in the same mini-batch serve as negative examples for each query, rather than requiring explicit negative mining. For a batch of B query-passage pairs, each query gets 1 positive passage and B-1 negatives. This dramatically increases the number of training examples without additional computation. DPR's original implementation used gold passages from the same batch as negatives, creating a more challenging training signal than random negatives. The technique requires large batch sizes—often 128 or more—to provide sufficient negative diversity.

The mathematical objective that drives DPR training by maximizing the similarity between a query and its correct passage while minimizing similarity with all negatives. DPR typically uses InfoNCE loss (a variant of noise contrastive estimation):

• Computes cosine similarity between query embedding and all passage embeddings in the batch
• Applies softmax over similarity scores to produce a probability distribution
• Minimizes negative log-likelihood of the correct passage

The temperature parameter controls how sharply the model penalizes hard negatives—lower temperatures create stricter separation.

Once trained, the passage encoder generates embeddings for the entire corpus offline. These vectors are stored in FAISS (Facebook AI Similarity Search), which provides GPU-accelerated indexing for billion-scale collections. FAISS supports multiple index types:

• IndexFlatIP: Exact inner product search, 100% recall but slow
• IndexIVFPQ: Inverted file with product quantization, trading accuracy for speed
• IndexHNSW: Graph-based navigation for logarithmic-time search

DPR's separation of encoding from indexing means passage embeddings can be updated incrementally without retraining the query encoder.

DPR's dense vectors are often combined with BM25 sparse retrieval in a hybrid system to capture both semantic meaning and exact keyword matches. The fusion typically uses:

• Reciprocal Rank Fusion (RRF): Combines ranked lists without requiring score calibration
• Linear interpolation: Weighted sum of normalized dense and sparse scores
• Re-ranking pipeline: DPR retrieves top-K candidates, then a cross-encoder refines the final order

This addresses DPR's known weakness with rare entities and exact phrase matching, where lexical methods like BM25 excel.

DPR requires supervised query-passage pairs for effective training, typically sourced from:

• MS MARCO: Microsoft's large-scale reading comprehension dataset with real Bing queries
• Natural Questions: Google queries paired with Wikipedia passages containing answers
• TREC CAR: Synthetic queries generated from Wikipedia section headings
• Domain-specific logs: Click-through data from production search systems

Data quality is critical—DPR trained on mismatched or noisy pairs will learn poor representations. Hard negative mining, where negatives are chosen to be top-ranked by a baseline retriever, significantly improves model robustness.