Subword Regularization

Unlike deterministic tokenizers that always produce the same output, subword regularization randomly samples from multiple valid segmentations during training. For a word like 'unhappiness', the model might see 'un-happiness', 'un-happi-ness', or 'un-ha-ppi-ness' across different epochs. This exposes the model to tokenization variance, preventing it from overfitting to a single arbitrary segmentation boundary.

The primary implementation of subword regularization is BPE-Dropout, which randomly discards a percentage of merge rules during each training step. Key properties:

• Dropout rate (typically 0.1) controls regularization strength
• Higher rates produce more fragmented, varied segmentations
• At inference, the full merge table is used deterministically
• Implemented natively in SentencePiece and Hugging Face Tokenizers

In the Unigram tokenization framework, subword regularization is achieved by sampling segmentations from the probability distribution over all possible token sequences. The likelihood of each segmentation is computed using the unigram probabilities of constituent tokens. The Viterbi algorithm finds the most probable segmentation, but during training, the forward-filtering and backward-sampling algorithm draws diverse candidates proportional to their probability.

A critical practical benefit: models trained with subword regularization handle typographical errors and orthographic variation more gracefully. When 'accommodate' is misspelled as 'acommodate', the stochastic training has already exposed the model to unusual character n-gram combinations, preventing catastrophic OOV failures. This is essential for user-generated content and multilingual text with inconsistent spelling conventions.

An alternative to dropout-based methods: generate the top-N most probable segmentations for each input and randomly select one per epoch. This approach:

• Provides more controlled variance than random dropout
• Ensures all sampled segmentations are linguistically plausible
• Can be combined with temperature scaling to sharpen or flatten the sampling distribution
• Used in machine translation and cross-lingual transfer tasks

Subword regularization intentionally creates a train-test mismatch: stochastic during training, deterministic during inference. This is analogous to dropout in neural networks—the noise acts as a regularizer that prevents co-adaptation to specific token boundaries. The model learns to compose meaning from subword fragments rather than memorizing whole-token representations, improving compositional generalization to rare and unseen words.

The primary algorithm implementing subword regularization. During training, it randomly drops a percentage of possible merge rules before segmenting each sentence. This forces the model to see multiple valid segmentations of the same word—for example, 'lowest' might be segmented as ['low', 'est'] in one epoch and ['l', 'ow', 'est'] in another. The dropout rate, typically set between 0.1 and 0.3, controls the intensity of regularization. Unlike standard BPE which produces deterministic segmentations, BPE-Dropout samples from the true posterior distribution of all possible subword sequences, making the downstream model robust to tokenization variance at inference time.

An alternative subword regularization strategy that explicitly computes the top-N most probable segmentations for a given word using the Unigram language model. Instead of randomly dropping merges, it scores all candidate segmentations by their joint probability and samples from this distribution. Key characteristics:

• Produces more controlled variance than BPE-Dropout
• The n_best_size parameter (typically 64 or 128) defines the sampling pool
• A smoothing parameter (alpha) flattens or sharpens the probability distribution
• Particularly effective when the true segmentation ambiguity is concentrated in a few high-probability candidates

The probabilistic foundation underlying many subword regularization implementations. Unlike BPE's deterministic merge-based approach, the Unigram model assumes each token is generated independently and assigns a probability to every possible segmentation. The model is trained by:

• Starting with a large seed vocabulary (often all characters plus common subwords)
• Iteratively removing tokens that least impact the training data likelihood
• Estimating token probabilities using the Expectation-Maximization (EM) algorithm This probabilistic nature makes it inherently suited for stochastic sampling, as it directly models the distribution over segmentations that subword regularization requires.

The SentencePiece tokenization framework provides native support for subword regularization through its --subword_regularization_sampling_rate and --unigram_model training flags. SentencePiece implements the Unigram language model approach and exposes a sampling interface that can be toggled at training versus inference time. Key implementation details:

• Training: enable sampling with a rate (e.g., 0.1) to expose the model to varied segmentations
• Inference: disable sampling (rate=0) for deterministic, maximum-probability segmentation
• The framework handles all Unicode normalization internally, ensuring the regularization operates on a consistent character space
• Lossless encoding guarantees that any sampled segmentation can be perfectly decoded back to the original text

A critical downstream benefit of subword regularization is improved handling of orthographic noise and morphological variation. By training on multiple segmentations, the model learns that tokens like 'color' and 'colour' share overlapping subword representations. Empirical results show:

• 2-5% improvement on noisy text benchmarks compared to deterministic BPE
• Better generalization to misspelled queries in search and question-answering tasks
• Increased recall for morphologically rich languages (e.g., Turkish, Finnish) where a single root can have dozens of surface forms
• The model develops distributed representations that are inherently invariant to token boundary placement, treating subword units as fuzzy rather than discrete

Subword regularization pairs naturally with contrastive representation learning objectives. When training dense retrieval models, applying different dropout masks to the same text passage produces two slightly different token sequences. These become a natural positive pair for contrastive loss functions like InfoNCE. This technique:

• Acts as a form of data augmentation without altering the underlying semantics
• Teaches the encoder that token boundary variations should map to the same semantic vector
• Reduces the representation collapse problem where models overfit to specific tokenization patterns
• Is particularly effective in multilingual settings where the same concept may tokenize differently across languages