Transliteration

Transliteration operates on sound correspondence, not character-by-character substitution. The goal is to enable a reader of the target script to approximate the original pronunciation.

• Maps phonemes (distinct units of sound) between languages
• Greek 'Ελένη' → Latin 'Eleni' preserves the /e.le.ni/ pronunciation
• Differs from transcription, which is a broader phonetic recording
• Often irreversible without context due to many-to-one mappings

A critical distinction in multilingual NLP: transliteration handles form, translation handles meaning.

• Transliteration: 'Ελένη' → 'Eleni' (sound preserved, meaning opaque)
• Translation: 'Ελένη' → 'Helen' (meaning preserved, sound altered)
• Search systems use transliteration for named entity matching across scripts
• Essential for cross-lingual information retrieval when entities lack direct translations

Each script pair requires a defined romanization or conversion standard. These are often governed by international bodies like ISO or national language authorities.

• ISO 9: Cyrillic to Latin with diacritics for scholarly precision
• Pinyin: Mandarin Chinese to Latin, the official romanization system
• Hepburn: Japanese to Latin, widely used by English speakers
• ITRANS: Indian scripts to Latin, common in digital text input
• Inconsistent application of rules leads to entity linking failures in knowledge graphs

Transliteration is a critical pre-processing and query expansion step for search engines operating across scripts.

• A user searching for 'Moscow' must also retrieve documents mentioning 'Москва'
• Query-side transliteration converts the user's Latin query into the target script
• Document-side transliteration indexes the romanized form of foreign-script content
• Failure to transliterate causes zero-recall problems for named entity queries
• Modern systems use learned transliteration models rather than static rule tables

Statistical and neural sequence-to-sequence models now outperform hand-crafted rules, especially for low-resource language pairs.

• Treat transliteration as a sequence transduction problem
• Encoder-decoder architectures with attention learn context-sensitive mappings
• Training data consists of name pairs (e.g., 'Mikhail' ↔ 'Михаил')
• Handles many-to-many alignments where one source character maps to multiple target characters
• Critical for zero-shot entity linking in languages without explicit training data

Converting a romanized word back to its original script is inherently ambiguous due to information loss during forward transliteration.

• Latin 'shi' could map to multiple Japanese or Chinese characters
• Requires language identification before back-transliteration can be attempted
• Often combined with contextual disambiguation from surrounding text
• A major source of error in cross-lingual entity linking pipelines
• Solved with candidate generation and ranking using language models

The process of transforming Unicode text into a standard canonical form (NFC or NFD) to ensure that visually identical characters with different byte representations are treated as the same string. This is a prerequisite for reliable transliteration. Common forms:

• NFC: Canonical composition (é as a single code point)
• NFD: Canonical decomposition (é as 'e' + combining acute accent)
• NFKC/NFKD: Compatibility normalization that also maps fullwidth Latin to ASCII Without normalization, a transliteration engine may fail to match input characters to its mapping tables.

The classification task of automatically detecting the natural language of a given text. This is a critical pre-processing step before transliteration, as the correct phonetic mapping depends entirely on knowing the source language. For example, the character sequence 'chat' would be transliterated differently if identified as French (/ʃa/) versus English (/tʃæt/). Modern approaches use:

• n-gram based models (fastText, CLD3)
• Transformer-based classifiers fine-tuned on 100+ languages
• Confidence thresholds to route low-certainty text for manual review

A mapping of monolingual word vector spaces into a single shared space, allowing a model to find the translation of a word by locating its nearest neighbor in the target language's embedding space. Connection to transliteration: Transliterated named entities (e.g., 'Moskva' → 'Moscow') often serve as anchor points for aligning these spaces. The phonetic similarity captured by transliteration provides a signal that helps models learn that 'Ελένη' in the Greek embedding space and 'Eleni' in the Latin space refer to the same entity.