Neural Machine Translation (NMT)

Unlike statistical systems that decompose translation into separately tuned sub-components, NMT uses a single deep neural network to directly model the conditional probability of a target sequence given a source sequence.

• Eliminates the need for hand-crafted features and alignment models
• Jointly optimizes all parameters via backpropagation on parallel corpora
• The model learns an internal, distributed representation of meaning

The foundational NMT framework consists of two recurrent or transformer-based networks working in tandem.

• Encoder: Processes the source sentence into a dense, fixed-length vector (context vector) that captures its semantic essence
• Decoder: Generates the target translation word-by-word, conditioned on the context vector and previously generated tokens
• The attention mechanism later augmented this by allowing the decoder to dynamically focus on different parts of the source sentence at each step

NMT systems address the rare word problem by operating on subword units rather than full words, enabling open-vocabulary translation.

• Byte-Pair Encoding (BPE) iteratively merges frequent character pairs to build a vocabulary of common word pieces
• Handles morphologically rich languages and out-of-vocabulary terms by decomposing them into known fragments
• Balances vocabulary size against sequence length for computational efficiency

During inference, NMT models generate translations by searching for the most probable output sequence rather than greedily selecting the single best token at each step.

• Maintains a fixed number (beam width) of candidate hypotheses at each decoding step
• Balances translation quality against computational cost; larger beams improve fluency but increase latency
• Often combined with length normalization to prevent the model from favoring short, incomplete translations

A critical innovation that resolves the information bottleneck of compressing an entire source sentence into a single fixed-length vector.

• Allows the decoder to compute a weighted sum of all encoder hidden states at each generation step
• Produces a soft alignment between source and target tokens, learned jointly with the translation task
• Modern multi-head self-attention (as in the Transformer) extends this by allowing the model to attend to information from multiple representation subspaces simultaneously

A single NMT model can be trained to translate between multiple language pairs, often demonstrating zero-shot translation capabilities for pairs never explicitly seen during training.

• A special language token is prepended to the source sequence to signal the desired target language
• Enables knowledge transfer from high-resource to low-resource language pairs
• Dramatically reduces the operational overhead of maintaining separate models for each language direction

The foundational neural network design behind modern NMT, introduced in the 2017 paper 'Attention Is All You Need.' It processes entire input sequences in parallel using a self-attention mechanism, which dynamically weights the relevance of each token in the source sentence to every other token. This replaces the sequential processing bottleneck of older recurrent models, enabling the capture of long-range dependencies and drastically faster training on massive parallel corpora.

Bilingual Evaluation Understudy is a precision-focused metric that measures the overlap of n-grams (typically 1 to 4) between a candidate translation and one or more human reference translations. A brevity penalty is applied to prevent overly short, high-precision outputs. While fast and language-independent, BLEU is a surface-level metric that correlates poorly with human judgment for morphologically rich languages and creative text.

A subword tokenization algorithm that is a prerequisite for open-vocabulary NMT. BPE starts with a vocabulary of individual characters and iteratively merges the most frequent adjacent symbol pairs to create new tokens. This elegantly solves the out-of-vocabulary problem by segmenting rare words into known subword units (e.g., 'unfortunately' → 'un' + 'fortunate' + 'ly'), allowing the model to translate any input string without an unknown token fallback.

The Cross-lingual Optimized Metric for Evaluation of Translation is a neural framework that predicts human judgment directly. Unlike BLEU, COMET uses a pre-trained cross-lingual model (like XLM-R) to embed the source, candidate translation, and a reference into a shared semantic space, then learns to regress a quality score. It addresses the core limitation of n-gram metrics by capturing semantic equivalence and paraphrasing, achieving state-of-the-art correlation with human evaluators.

A machine learning task that predicts translation quality without access to any human reference. QE models are trained on post-editing data to assign confidence scores at the word, phrase, or sentence level. This is critical for production pipelines where a reference is unavailable, enabling automated triage—routing low-confidence segments to human post-editors while publishing high-confidence output directly.

A secondary correction task where a dedicated model is trained to fix systematic errors in raw NMT output. The APE model takes the source text and the raw machine translation as input and generates a corrected version. This is often trained on human post-editing data—triplets of (source, raw MT, final corrected text)—and can be used to improve a black-box NMT system without retraining the primary model.