Dense Passage Retrieval (DPR)

DPR uses two independent BERT-based encoders: a query encoder and a passage encoder. This separation allows all passages in the corpus to be encoded offline into a static vector index. At query time, only the query is encoded, and its vector is used to search the pre-computed index via cosine similarity or inner product. This is fundamentally different from cross-encoders, which require processing the concatenated query-passage pair for every candidate, making them computationally prohibitive for first-pass retrieval over millions of documents.

DPR is trained using a contrastive loss with in-batch negatives. For each relevant query-passage pair in a training batch, all other passages in that same batch are treated as negative examples. This is computationally efficient because it reuses the passage embeddings already computed for the batch. The model is optimized to maximize the similarity between the query and its positive passage while minimizing similarity with all negatives. Hard negative mining—adding high-BM25-scoring but irrelevant passages—is critical for teaching the model fine-grained semantic distinctions.

Once all passages are encoded into dense vectors, they are indexed using Facebook AI Similarity Search (FAISS) for efficient approximate nearest neighbor search. DPR typically uses Hierarchical Navigable Small World (HNSW) graphs or Inverted File with Product Quantization (IVFPQ) to compress vectors and accelerate retrieval. This enables sub-linear search time over billion-scale corpora, trading a small amount of accuracy for massive speed gains. The index can be stored entirely in memory or on disk with memory-mapped files.

DPR excels at semantic matching—finding passages that answer a question using different vocabulary than the query. However, it can struggle with precise lexical matches like rare entity names or numbers. This is why DPR is most powerful when combined with BM25 or learned sparse retrieval in a hybrid system. The two methods retrieve different result sets with high complementarity, and their outputs are fused using Reciprocal Rank Fusion (RRF) or weighted sum fusion to maximize overall recall.

DPR training requires a specific dataset structure: natural questions paired with positive passages containing the answer span. The original DPR paper used datasets like Natural Questions, TriviaQA, and SQuAD. Critically, the positive passage is the one actually containing the answer, not just a top-ranked BM25 result. This ensures the model learns to retrieve passages that genuinely answer the question, not just those that are topically related. Negative passages are sampled from top BM25 results that do not contain the answer.

During inference, similarity between the query vector and passage vectors is computed using either dot product or cosine similarity. DPR is typically trained with dot product, which is faster to compute and works well when vectors are L2-normalized. Cosine similarity is equivalent to dot product on normalized vectors. The choice affects ANN index configuration: FAISS supports both inner product search and L2 distance search, with inner product being the direct implementation of dot product scoring.