mBERT (Multilingual BERT)

mBERT is pre-trained on a concatenated corpus of 104 languages using a standard Masked Language Modeling (MLM) and Next Sentence Prediction (NSP) objective. Unlike monolingual BERT, it does not use language-specific embeddings, forcing the model to learn a shared, language-agnostic representation space. The model uses a WordPiece vocabulary of 110,000 tokens shared across all languages, which requires careful vocabulary balancing to prevent high-resource languages from dominating the subword allocation.

The defining capability of mBERT is performing tasks in a target language after being fine-tuned only on English (or another source language) data. This works because the shared multilingual representations cause semantically equivalent words and structures to cluster together in the vector space. For example, an mBERT model fine-tuned on English Named Entity Recognition (NER) can identify person and organization names in Swahili or Basque without ever seeing labeled examples in those languages during training.

mBERT uses a cased WordPiece vocabulary of 110,000 subword units. The tokenizer is applied identically to all languages, meaning a word like 'bank' in English and 'Bank' in German share the same token IDs if they are identical strings. This design choice creates a form of lexical overlap anchoring, where identical or similar strings across languages act as pivot points in the embedding space, facilitating cross-lingual alignment even without explicit parallel data.

mBERT shares the exact same architecture as BERT-Base: 12 transformer layers, 768 hidden dimensions, and 12 attention heads, totaling 110 million parameters. This architectural constraint means the model must encode the complexities of 104 languages into the same parameter budget as the monolingual English BERT. This capacity bottleneck is a known limitation, often referred to as the curse of multilinguality, where adding more languages degrades performance on individual languages due to representational interference.

Despite having no explicit cross-lingual training signal, mBERT produces surprisingly robust language-agnostic sentence representations. When sentence embeddings are extracted from the final hidden states, semantically equivalent sentences in different languages (e.g., 'The cat sits on the mat' and 'Le chat est assis sur le tapis') map to similar vector regions. This property is what enables effective Cross-Lingual Information Retrieval (CLIR) and zero-shot document classification.

The primary limitation of mBERT is the curse of multilinguality: the fixed 110M parameter budget is shared across 104 languages, leading to degraded per-language performance compared to monolingual models. Low-resource languages written in unique scripts (e.g., Armenian, Georgian) often suffer from severe under-representation in the shared vocabulary, resulting in excessive fragmentation into subword units. This limitation motivated the development of successor models like XLM-RoBERTa, which uses a larger unigram vocabulary and is trained on a significantly larger corpus.

The core capability enabled by mBERT: applying a model fine-tuned on a high-resource source language (e.g., English) to perform tasks in a low-resource target language (e.g., Swahili) without any target-language fine-tuning data. This works because mBERT's shared multilingual representations allow task-specific classifiers to generalize across language boundaries. For example, an NER model trained only on English CoNLL-2003 data can identify persons, organizations, and locations in German text with reasonable accuracy.

The pre-training objective that builds mBERT's shared representation space. The model is trained to predict randomly masked tokens in a concatenated stream of text from 104 languages simultaneously. Unlike translation-based approaches, no parallel data is required. The model learns language-agnostic features because:

• Shared subword vocabulary forces cross-lingual alignment at the token level
• Deep Transformer layers learn to identify language-invariant syntactic and semantic patterns
• The model implicitly learns that 'dog' in English and 'Hund' in German fill similar masked slots

The ideal output of a multilingual encoder: sentence vectors that are independent of the source language, such that semantically equivalent sentences in different languages map to identical vector regions. mBERT partially achieves this, but dedicated models like LaBSE and LASER are explicitly trained for this property using translation ranking objectives. In mBERT, language-agnosticism emerges as a byproduct of shared subword tokens and joint training, rather than being an explicit optimization target.

The task of grounding a textual mention to a knowledge base entry in a language the model has never explicitly seen during training. mBERT enables this by encoding mentions and entity descriptions into a shared space. For example:

• Mention: 'Angela Merkel' in a German news article
• Target: Q567 (Angela Merkel) in Wikidata, described in English
• mBERT encodes both into similar vectors despite the language mismatch
• This works because mBERT's representations capture entity-level semantics that transcend surface form