Bilingual Lexicon Induction

BLI relies on a small seed dictionary of known translation pairs (e.g., 5,000 word pairs) to learn a linear mapping between two vector spaces. The quality and size of this seed directly dictate alignment accuracy. Key considerations:

• A seed dictionary as small as 25 word pairs can bootstrap alignment in closely related languages
• Identical strings (e.g., named entities like 'Barack Obama') often serve as a high-precision initial seed
• The mapping is typically learned via Procrustes alignment, solving for an orthogonal transformation matrix W such that source embeddings multiplied by W approximate target embeddings

In high-dimensional spaces, certain vectors become universal nearest neighbors—they are the closest match to an abnormally large number of query points. This phenomenon, called hubness, severely degrades BLI precision. Mitigation strategies include:

• CSLS (Cross-Domain Similarity Local Scaling): Penalizes similarity scores by subtracting the average similarity to the k nearest neighbors in the target space
• Inverted Softmax: Normalizes similarity scores across the entire target vocabulary before selecting the nearest neighbor
• L2 normalization of all embeddings to constrain them to the unit hypersphere, reducing the intrinsic dimensionality of the space

The mapping matrix W learned during BLI is constrained to be orthogonal (W^T W = I). This preserves the monolingual structure of the embedding space—distances and angles between words within a language remain unchanged after mapping. Why orthogonality matters:

• Prevents degradation of the source language's semantic relationships during alignment
• Reduces the search space from arbitrary linear transformations to rotations and reflections
• Enables a closed-form solution via Singular Value Decomposition (SVD): W = U V^T, where U Σ V^T = X^T Y (the cross-covariance of source and target seed embeddings)

BLI performance is rigorously measured using Precision@k, which reports the percentage of source words whose correct translation appears within the top k retrieved candidates. Standard benchmarks include:

• Precision@1: The strictest metric—the correct translation must be the single nearest neighbor
• Precision@5: Allows the correct translation to appear within the top 5 candidates, accommodating polysemy
• MUSE datasets provide standard evaluation dictionaries for 110 language pairs, including distant pairs like English-Chinese and English-Arabic
• State-of-the-art supervised methods achieve >80% Precision@1 on similar language pairs (e.g., English-German)

A bootstrapping technique that progressively improves alignment quality without additional human-annotated data. The process alternates between two steps:

• Step 1 - Mapping: Learn an orthogonal transformation W using the current seed dictionary via Procrustes analysis
• Step 2 - Dictionary Induction: Apply W to all source words, retrieve nearest neighbors in the target space, and select high-confidence mutual nearest neighbors as a new, larger seed dictionary
• This loop repeats until convergence, often expanding a 5,000-pair seed to 15,000+ high-quality translation pairs
• Mutual nearest neighbor criterion: A pair (s, t) is only retained if s is the nearest neighbor of t AND t is the nearest neighbor of s

A critical post-processing step that mitigates the hubness problem in high-dimensional cross-lingual spaces. In high dimensions, certain vectors become 'hubs'—universal nearest neighbors to many query points—degrading bilingual lexicon induction accuracy from 90%+ to below 30%.

• CSLS (Cross-domain Similarity Local Scaling): The standard solution that penalizes hubs by subtracting the mean similarity to the k nearest neighbors in the target space
• Inverted Softmax: An alternative that normalizes similarity scores using a softmax over the entire target vocabulary
• Impact: Restores retrieval precision to near-supervised levels in unsupervised alignment scenarios

The upstream data foundation that enables bilingual lexicon induction. Parallel corpora are collections of sentence-aligned translations, while bitext mining is the automated process of extracting these pairs from noisy web-crawled data using multilingual sentence embeddings like LaBSE.

• LASER toolkit: Provides language-agnostic sentence embeddings used for margin-based bitext mining across 100+ languages
• CCAligned: A publicly available corpus of 100+ million aligned document pairs extracted from Common Crawl
• Quality filtering: Bitext mining pipelines typically apply a margin score threshold and length ratio filtering to remove noisy alignments

The standard benchmark for evaluating whether cross-lingual representations capture semantic entailment rather than just lexical translation. XNLI tests if a model can determine that a premise in one language entails, contradicts, or is neutral to a hypothesis in another language.

• Languages: 15 languages including low-resource languages like Swahili and Urdu
• Zero-shot transfer: Models trained only on English NLI data are evaluated on all 15 target languages
• Relevance to BLI: High XNLI accuracy indicates that the aligned embedding space preserves semantic relationships beyond word-level translation