Neural Machine Translation (NMT)

Unlike traditional Statistical Machine Translation (SMT) which decomposes translation into separately tuned components, NMT uses a single, large neural network to directly model the conditional probability P(target | source). This joint optimization eliminates error propagation between sub-components. The entire pipeline—from source tokenization to target sequence generation—is trained simultaneously via backpropagation, allowing the model to learn a holistic mapping between languages without intermediate symbolic representations.

The attention mechanism allows the decoder to dynamically focus on different parts of the source sentence at each decoding step, rather than compressing the entire input into a fixed-length vector. This solves the information bottleneck of earlier encoder-decoder architectures.

• Soft Attention: Computes a weighted sum of all encoder hidden states, where weights are learned alignment scores.
• Self-Attention: Allows the Transformer architecture to compute relationships between all positions in a sequence simultaneously, enabling parallel processing.
• Multi-Head Attention: Projects queries, keys, and values into multiple subspaces, allowing the model to attend to different representation subspaces jointly.

NMT systems employ subword segmentation algorithms like Byte-Pair Encoding (BPE) or SentencePiece to handle rare and out-of-vocabulary words. By decomposing words into frequent subword units, the model can translate morphologically rich languages and generate novel words not seen during training.

• Example: "untranslatable" → ["un", "translat", "able"]
• This eliminates the <UNK> token problem that plagued earlier word-level models.
• Enables open-vocabulary translation without a fixed dictionary, critical for agglutinative languages like Turkish or Finnish.

At inference time, NMT models use beam search to find the most probable output sequence. Rather than greedily selecting the single highest-probability token at each step, beam search maintains k candidate hypotheses (the beam width) and expands them simultaneously. The final translation is the hypothesis with the highest cumulative log-probability, often normalized by length penalty to prevent favoring overly short outputs. Typical beam widths range from 4 to 8, balancing translation quality against computational cost.

Modern NMT architectures support multilingual models trained on many language pairs simultaneously. A single model can translate between dozens of languages, including zero-shot translation—translating between language pairs never explicitly seen during training. This is achieved by prepending a special language token to the source sequence, conditioning the entire network on the target language. Multilingual training enables knowledge transfer from high-resource languages (e.g., English-French) to low-resource pairs (e.g., Swahili-Hindi), dramatically improving performance where parallel corpora are scarce.

While early NMT systems translated sentences in isolation, document-level NMT extends the context window to include surrounding sentences or the entire document. This resolves cross-sentence phenomena such as:

• Pronoun disambiguation: Correctly resolving "it" based on prior discourse.
• Lexical cohesion: Maintaining consistent terminology across a document.
• Discourse structure: Preserving rhetorical relationships and topic continuity. This is achieved by feeding inter-sentence context through extended attention mechanisms or by caching hidden states across sentence boundaries.

The foundational encoder-decoder framework upon which NMT is built. The encoder processes the entire source sentence into a fixed-length context vector, while the decoder autoregressively generates the target translation one token at a time.

• Encoder: A recurrent or transformer network that reads the input sequence and compresses it into a dense representation.
• Decoder: Generates the output sequence conditioned on the encoder's final hidden state.
• Bottleneck Problem: Early Seq2Seq models suffered from information loss when compressing long sentences into a single vector, a limitation directly addressed by the attention mechanism.

A critical innovation that allows the decoder to dynamically focus on different parts of the source sentence at each generation step, rather than relying on a single fixed context vector.

• Alignment: The model learns a soft alignment between source and target tokens, enabling it to handle long-range dependencies and word reordering.
• Bahdanau Attention: An additive form that computes an alignment score between the decoder's previous hidden state and each encoder output.
• Luong Attention: A multiplicative variant offering several scoring functions (dot, general, concat) for computing context vectors.

A subword tokenization algorithm essential for NMT that segments words into frequent subword units, solving the open-vocabulary problem and handling rare or unseen words.

• Vocabulary Construction: Iteratively merges the most frequent pair of bytes or characters in the training corpus until a target vocabulary size is reached.
• Out-of-Vocabulary Mitigation: By operating on subwords, BPE can encode any word as a sequence of known units, eliminating <UNK> tokens.
• Multilingual Utility: BPE is language-agnostic and learns segmentations that capture common morphemes across languages, making it ideal for multilingual NMT systems.

The dominant neural architecture for modern NMT, replacing recurrence with self-attention to process entire sequences in parallel. Introduced in the paper 'Attention Is All You Need' (Vaswani et al., 2017).

• Self-Attention: Computes a weighted representation of every token in a sequence relative to every other token, capturing contextual relationships without sequential processing.
• Multi-Head Attention: Runs multiple attention operations in parallel, allowing the model to attend to information from different representation subspaces simultaneously.
• Positional Encoding: Injects information about token order into the input embeddings using sinusoidal functions, since the architecture has no inherent notion of sequence.

The Bilingual Evaluation Understudy metric, the de facto standard for automatically evaluating NMT output quality by measuring n-gram overlap between candidate and reference translations.

• Precision Focus: Computes a modified precision score for n-grams of size 1 to 4, penalizing translations that over-generate common words.
• Brevity Penalty: Multiplies the precision score by a factor that penalizes translations shorter than their references, preventing artificially high scores from overly short outputs.
• Limitations: BLEU does not account for semantic equivalence, synonymy, or grammatical correctness, and correlates poorly with human judgment for high-quality systems.

A paradigm where a single NMT model is trained to translate between multiple language pairs simultaneously, often without parallel data for every pair (zero-shot translation).

• Language Tokens: A special token (e.g., <2en>) is prepended to the source sequence to indicate the target language, conditioning the entire generation process.
• Transfer Learning: Knowledge gained from high-resource language pairs improves translation quality for low-resource pairs sharing the model.
• Zero-Shot Translation: The model can translate between language pairs never seen together during training by leveraging a shared interlingua representation in the latent space.