Multilingual Dense Passage Retrieval (mDPR)

The core innovation of mDPR is training a dual-encoder architecture to map semantically identical texts into the same dense region, regardless of source language. Unlike pipelines that translate queries, mDPR uses contrastive learning with parallel corpora to minimize the distance between a query in Language A and its relevant document in Language B. This creates a language-agnostic representation where cosine similarity directly measures cross-lingual relevance, enabling zero-shot retrieval without machine translation.

mDPR employs two independent Transformer encoders—one for queries and one for passages—that are fine-tuned jointly. Key properties include:

• Asymmetric encoding: The query encoder is optimized for short, keyword-sparse inputs, while the passage encoder handles long-form documents.
• Late interaction: Query and document vectors are computed independently and compared only at the final dot-product stage, enabling offline indexing of billions of documents.
• Shared multilingual backbone: Both encoders typically initialize from a single multilingual model like XLM-RoBERTa to inherit cross-lingual transfer capabilities.

mDPR is trained using a contrastive loss function that pulls positive query-passage pairs together while pushing negative pairs apart. The training strategy leverages:

• In-batch negatives: Other passages within the same GPU batch serve as negative examples, dramatically increasing computational efficiency without explicit negative mining.
• Hard negatives: Passages that are top-ranked by BM25 but not relevant are added to improve the model's discriminative power.
• Cross-lingual positives: A query in English and its relevant passage in Japanese form a positive pair, forcing the model to ignore surface-level lexical overlap and focus on semantic meaning.

At inference time, mDPR relies on FAISS (Facebook AI Similarity Search) for sub-linear retrieval from million- or billion-scale document collections. The indexing pipeline involves:

• Offline passage encoding: All documents in the corpus are pre-encoded into dense vectors and stored in a FAISS index.
• Product Quantization (PQ): Vectors are compressed to reduce memory footprint while preserving high-fidelity similarity search.
• Inverted File Index (IVF): The vector space is partitioned into Voronoi cells, allowing the retriever to search only a small fraction of the index at query time. This architecture decouples encoding cost from retrieval latency, enabling sub-100ms response times across multilingual corpora.

mDPR's ability to process 100+ languages begins at the tokenization layer. It uses SentencePiece with a unigram language model to build a single, shared vocabulary that segments text into subword units. Critical design decisions include:

• No language-specific pre-processing: Text is fed raw, without stemming or lemmatization, preserving morphological information.
• Script normalization: Unicode normalization (NFC) ensures visually identical characters share the same token IDs.
• Shared subword inventory: A token like '##ing' can represent a morpheme in English, a syllable in Chinese Pinyin, or a transliterated fragment, enabling the model to share parameters across scripts.

A defining property of mDPR is its ability to retrieve documents in languages never seen during fine-tuning. This zero-shot transfer emerges from:

• Multilingual pre-training: The underlying encoder (e.g., mBERT or XLM-RoBERTa) was pre-trained on 100+ languages with masked language modeling, building a polyglot representation space.
• Semantic grounding via parallel data: Fine-tuning on high-resource language pairs (e.g., English-Arabic) teaches the model that translation equivalents should be neighbors in vector space.
• Cross-lingual generalization: The alignment generalizes to unseen languages because the encoder's multilingual representations are already structurally aligned from pre-training. This property eliminates the need for per-language fine-tuning data, critical for low-resource languages.