SentencePiece

SentencePiece guarantees lossless tokenization by treating the input as a raw sequence of Unicode characters, including whitespace. The original text can be perfectly reconstructed from the token sequence by simple concatenation. This is achieved by escaping whitespace with a meta symbol _ (U+2581), making the process fully reversible without ambiguity. This property is critical for tasks where exact text reproduction is required, such as machine translation and text generation.

Unlike rule-based tokenizers that rely on language-specific pre-tokenization (e.g., splitting on whitespace for English), SentencePiece operates directly on the raw byte stream. This makes it inherently suitable for languages that do not use spaces to separate words, such as Chinese, Japanese, Korean, and Thai. By treating all languages uniformly, it eliminates the need for custom segmentation logic per language, simplifying multilingual model pipelines.

SentencePiece implements two core subword algorithms:

• Byte Pair Encoding (BPE): Starts with characters and iteratively merges the most frequent adjacent symbol pairs. It builds a vocabulary bottom-up, effectively capturing common subword units.
• Unigram Language Model: Starts with a large seed vocabulary and iteratively prunes tokens that least increase the overall likelihood of the training corpus. This probabilistic approach often yields better segmentation for morphologically rich languages. Users can choose the algorithm that best fits their data distribution.

During training, SentencePiece can apply subword regularization by probabilistically sampling different segmentations for the same input string based on the Unigram model's likelihood. Instead of a deterministic segmentation, the model sees multiple tokenization variants, acting as a form of data augmentation. This technique improves model robustness to rare words and spelling variations, and is particularly effective for neural machine translation and low-resource language scenarios.

SentencePiece manages a fixed-size vocabulary where each token is assigned a unique integer ID. The model file stores the mapping between tokens and IDs, along with their learned scores. The vocabulary size is a user-defined hyperparameter that directly controls the granularity of segmentation. A smaller vocabulary yields shorter sequences of larger subwords, while a larger vocabulary captures finer-grained character n-grams. This explicit control is essential for balancing model size and sequence length.

SentencePiece is a self-contained, end-to-end system. It accepts raw text sentences and outputs token ID sequences, handling the entire pipeline from normalization to detokenization. The training process requires only raw text files and a target vocabulary size. The resulting model file is a single, portable artifact that encapsulates the full tokenization logic, ensuring deterministic and consistent preprocessing across training and inference environments without external dependencies.

An iterative data compression algorithm that merges the most frequent pair of bytes or characters into a new token. In SentencePiece, BPE starts with characters and repeatedly merges the most frequent adjacent symbol pairs to build a vocabulary of subwords.

• Deterministic: Produces the same segmentation for a given vocabulary.
• Greedy merging: Merges based on frequency counts in the training corpus.
• Reversibility: The original text can be perfectly reconstructed from the BPE token sequence.
• Handles OOV: Rare and unseen words are decomposed into known subword units.

A probabilistic subword segmentation algorithm that assumes each token is generated independently. SentencePiece trains a unigram LM by starting with a large seed vocabulary and iteratively removing tokens that least impact the overall likelihood of the training corpus.

• Probabilistic: Assigns a probability to multiple possible segmentations.
• Lossy pruning: Removes low-utility tokens to reach a target vocabulary size.
• Viterbi decoding: Finds the most probable token sequence for a given input.
• Regularization: Supports subword sampling during training for robustness.

A data augmentation technique that randomly samples different subword segmentations during neural network training. By exposing the model to multiple valid tokenizations of the same input string, it learns robust, segmentation-invariant representations.

• Improves generalization: Reduces overfitting to a single deterministic segmentation.
• On-the-fly sampling: Uses the unigram LM's probability distribution to generate varied token sequences.
• Better for NMT: Demonstrated to significantly improve neural machine translation quality, especially on low-resource languages.
• Hyperparameter α: Controls the smoothness of the sampling distribution.

A critical design property where the tokenization process is fully reversible without information loss. SentencePiece achieves this by treating whitespace as a normal character (escaping it as a meta-symbol like '▁') rather than using it as a word boundary delimiter.

• No pre-tokenization: Raw text is fed directly to the tokenizer without language-specific splitting.
• Deterministic detokenization: The original byte sequence is recovered by simply concatenating tokens and replacing the whitespace meta-symbol.
• Language independence: Essential for languages like Chinese and Japanese that lack explicit word delimiters.
• End-to-end: Enables a pure subword pipeline from raw text to model input.

The process of learning a subword vocabulary from a raw training corpus. SentencePiece's spm_train utility accepts unprocessed text and jointly optimizes the vocabulary and segmentation model.

• Input: A single plain-text file with one sentence per line.
• Key parameters: vocab_size (e.g., 32k), model_type (bpe or unigram), and character_coverage.
• Character coverage: Specifies the fraction of unique characters to include, with a default of 0.9995 for languages with rich character sets.
• Special tokens: Automatically reserves IDs for control tokens like <unk>, <s>, and </s>.

A built-in text normalization engine that applies deterministic string transformations before tokenization. SentencePiece uses a custom rule-based normalizer to standardize input text, improving consistency.

• NFKC Unicode normalization: Converts compatibility characters to their canonical equivalents.
• Case folding: Optional lowercasing of the input text.
• Custom rules: Users can define replacement patterns using a TSV format (e.g., mapping full-width digits to half-width).
• Idempotency: Normalization is designed to be applied safely multiple times without altering the result.