Pre-tokenization

The most common pre-tokenization strategy splits text on whitespace and isolates punctuation as separate units. For example, the string "Hello, world!" becomes ["Hello", ",", "world", "!"]. This ensures that punctuation marks like periods and commas are never merged with adjacent characters during BPE or WordPiece training, preserving their syntactic role. Libraries like Hugging Face Tokenizers implement this via the pre_tokenizers.Whitespace or pre_tokenizers.Punctuation components, which can be chained together in a tokenization pipeline.

Advanced pre-tokenizers use Unicode character classes to isolate digits from letters. The regex pattern \d+ splits contiguous numbers into standalone tokens, preventing a BPE merge like "file123" from fusing letters and digits. This is critical for models handling code, financial data, or scientific text where numeric values must remain distinct. The GPT-2 pre-tokenizer uses a regex that isolates contractions, letters, digits, and punctuation, ensuring "100km" is split into ["100", "km"] before BPE merges are applied.

Pre-tokenization must account for language-specific morphology. In languages like Chinese or Japanese that lack whitespace delimiters, a pre-tokenizer must perform word segmentation using dictionaries or statistical models. The SentencePiece framework bypasses this entirely by treating the input as a raw Unicode stream, making it language-agnostic. For languages with clitics (e.g., French "l'homme"), a pre-tokenizer may split the apostrophe to yield ["l'", "homme"], preserving the morpheme boundary for the subword model.

The pre-tokenization step defines the maximum token length and the set of characters that can appear within a single token. If a pre-tokenizer never splits on a hyphen, the subword model can learn merges like "state-of-the-art" as a single token. Conversely, aggressive splitting on every punctuation mark prevents multi-word compounds. This directly impacts the compression ratio and the model's ability to memorize common phrases. The choice of pre-tokenization regex is a hyperparameter that trades vocabulary expressiveness for granularity.

A well-designed pre-tokenizer combined with a subword model like SentencePiece or Byte-level BPE can guarantee lossless tokenization. This means the original input string can be perfectly reconstructed from the token sequence without ambiguity. The pre-tokenizer must preserve all whitespace and casing information. For instance, the MetaSpace pre-tokenizer in SentencePiece replaces spaces with a special character ▁ (U+2581), ensuring that leading spaces and word boundaries are explicitly encoded and recoverable during decoding.

Pre-tokenization is distinct from normalization. Normalization transforms the text (e.g., lowercasing, Unicode NFKC normalization, stripping accents) before any splitting occurs. Pre-tokenization then segments the normalized string. The order is critical: normalization can merge characters that pre-tokenization would otherwise split. For example, normalizing the Unicode ligature fi (U+FB01) into fi before pre-tokenization changes the character sequence and thus the token boundaries. The tokenization pipeline executes normalization first, then pre-tokenization, then the subword model.

Reserved vocabulary entries added during post-processing that serve control functions rather than representing textual content.

• [CLS] — Classification token prepended to sequences in BERT
• [SEP] — Separator between sentence pairs
• [MASK] — Placeholder for masked language modeling
• [PAD] — Padding token for batch uniformity
• [UNK] — Fallback for out-of-vocabulary sequences (rare in subword models)

Pre-tokenization boundaries determine where these tokens can be inserted. For instance, [SEP] is typically placed at a pre-token boundary between sentences.

The linguistic process of decomposing words into constituent morphemes — the smallest meaning-bearing units. This provides an alternative to purely statistical pre-tokenization.

• Stemming — Heuristic stripping of affixes (e.g., 'running' → 'run')
• Lemmatization — Dictionary-based reduction to canonical form (e.g., 'better' → 'good')
• Morpheme segmentation — Splitting 'unhappiness' into 'un-happy-ness'

While BPE and WordPiece discover subword units statistically, morphological analyzers use linguistic rules. Some hybrid tokenizers combine both approaches for improved semantic coherence.