Dense Passage Retrieval (DPR)

DPR uses two independent BERT-based encoders: one for queries and one for passages. This separation allows passages to be pre-encoded and indexed offline, enabling sub-linear retrieval latency at query time. The query encoder maps a natural language question to a dense vector, while the passage encoder maps every document chunk to a fixed-size embedding in the same vector space.

Relevance is computed as the dot product between the query embedding and each passage embedding. This operation is highly optimized for GPU acceleration and Approximate Nearest Neighbor (ANN) search. Unlike cross-encoders that process query-passage pairs jointly, the inner product allows for maximum inner product search (MIPS) over millions of pre-computed passage vectors in milliseconds.

DPR is trained using a contrastive loss with hard negatives. For each relevant query-passage pair in a batch, all other passages in the same batch serve as negatives. This technique dramatically increases training efficiency by reusing computed embeddings. Additional hard negative mining—selecting high-BM25-score but irrelevant passages—further sharpens the model's disambiguation capability.

DPR overcomes the lexical gap that plagues sparse retrievers like BM25. Where BM25 relies on exact term overlap, DPR captures semantic similarity:

• Synonym handling: 'physician' matches 'doctor' without explicit mapping
• Paraphrase robustness: reworded queries retrieve the same relevant passages
• Cross-lingual potential: embeddings can align semantically across languages This makes DPR ideal for clinical text where the same concept may be expressed with highly variable terminology.

Pre-computed passage embeddings are indexed using Facebook AI Similarity Search (FAISS) for efficient retrieval. FAISS supports multiple index types:

• Flat Index: exact search, suitable for small corpora
• IVF (Inverted File): clusters embeddings for faster approximate search
• HNSW (Hierarchical Navigable Small World): graph-based indexing for high recall For biomedical knowledge bases with millions of entities, FAISS enables sub-10ms retrieval on commodity hardware.

In clinical entity linking pipelines, DPR serves as the candidate generation stage. A clinical mention (e.g., 'MI') is encoded by the query tower, and the top-k most similar UMLS concept descriptions are retrieved from a pre-indexed knowledge base. This dense retrieval step is often followed by a cross-encoder reranker for precise candidate scoring. DPR's semantic understanding is critical for resolving ambiguous abbreviations like 'MI' (myocardial infarction vs. mitral insufficiency) based on surrounding context.

The foundational dual-tower neural network that defines DPR. A query encoder and a context encoder independently map text into a shared dense vector space. Unlike cross-encoders, this separation allows for offline pre-indexing of millions of passages. At inference, only the query needs to be encoded, enabling sub-linear search times via Approximate Nearest Neighbor (ANN) indexes.

The self-supervised training paradigm used to optimize DPR. The model is trained to maximize the inner product between a query and its positive passage while minimizing it for negative passages. Key sampling strategies include:

• In-batch negatives: Other passages in the same mini-batch serve as negatives.
• Hard negative mining: Selecting high-BM25-score but incorrect passages to sharpen decision boundaries. This forces the encoder to learn fine-grained semantic distinctions.

DPR typically serves as the first-stage retriever in a two-stage pipeline. It efficiently fetches a small subset of candidates (e.g., top-100) from a large corpus. A more computationally expensive Cross-Encoder Reranker then processes the query and each candidate jointly to produce a high-fidelity relevance score. This hybrid approach balances the speed of dense retrieval with the precision of joint encoding.

The indexing algorithm that makes DPR viable at scale. Exhaustive vector comparison is computationally prohibitive. ANN libraries like FAISS or ScaNN cluster and compress embeddings, trading a small amount of recall for a massive gain in speed. This allows DPR to search billions of passages in milliseconds by focusing the search on the most promising regions of the vector space.

A critical data curation strategy for DPR training. Random negatives are too easy for the model to dismiss. Hard negatives are passages that share high lexical overlap with the query but are semantically irrelevant. Mining these using a sparse retriever like BM25 forces the dual-encoder to move beyond keyword matching and learn true semantic similarity, significantly boosting top-k retrieval accuracy.

The high-precision second-stage model paired with DPR. While DPR encodes the query and passage independently, a cross-encoder concatenates them and processes the pair through a full transformer. This allows for deep token-level attention but is too slow for full-corpus search. It re-ranks the DPR candidates, often using a BERT-based architecture, to push the most relevant passage to the top position.