Unigram Language Model

Unlike BPE's bottom-up merge operations, Unigram begins with a heuristic-based large vocabulary—often all possible substrings or characters—and then applies a lossy pruning process. During each iteration, the algorithm calculates the loss in corpus likelihood if a specific token were removed. Tokens with the lowest impact on the overall probability are discarded. This continues until the target vocabulary size is reached, ensuring only the most statistically useful subword units survive.

A core innovation of Unigram is that it does not enforce a deterministic segmentation during training. Given a trained model, a word can have multiple valid segmentations, each with an associated probability. The most likely segmentation is found using the Viterbi algorithm, which efficiently computes the path through the token lattice that maximizes the product of token probabilities. This probabilistic foundation directly enables subword regularization.

The optimization objective is to maximize the log-likelihood of the training corpus. For a given vocabulary, the probability of a sentence is the product of the probabilities of its constituent tokens. The pruning step estimates the marginal loss for each token by calculating how much the overall log-likelihood would decrease if that token were removed and its occurrences were replaced by the next-best segmentation. Tokens with the smallest marginal loss are pruned.

Because Unigram models the probability distribution over all possible segmentations, it can be used for subword regularization during training. Instead of always using the Viterbi-best segmentation, tokens can be stochastically sampled from the distribution. This exposes the model to multiple tokenization variants of the same word, acting as a form of data augmentation that improves robustness to spelling variations and rare morphological forms.

The Unigram algorithm is natively supported in the SentencePiece library, which treats the input as a raw Unicode stream without requiring language-specific pre-tokenization. SentencePiece's implementation pairs Unigram with a lossless encoding scheme by escaping whitespace into a meta-character, ensuring the original text can be perfectly reconstructed from the token sequence. This makes it a popular choice for multilingual models.

Unigram differs fundamentally from frequency-based methods:

• BPE: Merges the most frequent pair of tokens iteratively.
• WordPiece: Merges the pair that maximizes training data likelihood.
• Unigram: Starts with all tokens and prunes the least useful ones. This top-down approach often yields a more statistically principled vocabulary that better captures rare morphological patterns, though it can be computationally more expensive to train than BPE.

The broader category of segmentation strategies that the Unigram model belongs to. Instead of treating whole words or single characters as units, subword tokenization breaks text into meaning-bearing fragments like morphemes or frequent character sequences. This balances vocabulary size with the ability to represent rare, misspelled, or unseen words by composing them from known subword units.

A training technique that leverages the probabilistic nature of the Unigram model. By stochastically sampling different segmentations for the same input text during training, the model learns to be robust to tokenization variance. This is implemented via BPE-Dropout or by exploiting the multiple segmentation candidates that the Unigram model naturally provides, improving generalization.

A critical hyperparameter directly controlled by the Unigram algorithm's pruning process. The model starts with a massive seed vocabulary and iteratively removes tokens that least impact the corpus likelihood until the target size is reached. This represents a trade-off: a larger vocabulary captures more semantics per token but increases the model's embedding matrix, while a smaller vocabulary forces more compositional encoding.

A property guaranteed by the Unigram model when implemented within frameworks like SentencePiece. Because the tokenizer operates on a complete probability distribution over all possible segmentations and the vocabulary is derived directly from the corpus, the original input text can be perfectly reconstructed from the token sequence. This eliminates the information loss common in heuristic normalization pipelines.