Normalization

Maps all alphabetic characters to their lowercase equivalents to collapse case-based distinctions. This is a lossy transformation that destroys information useful for Named Entity Recognition (e.g., distinguishing 'Apple' the company from 'apple' the fruit). Modern subword tokenizers often defer case handling to the model, but lowercasing remains common in sparse retrieval and BM25 systems to improve recall by ensuring 'Query' and 'query' map to the same term.

Standardizes Unicode text into a canonical form to ensure visually identical characters have the same byte representation. The NFKC (Compatibility Composition) form is widely used as it decomposes ligatures (e.g., 'fi' becomes 'fi') and converts full-width characters to their ASCII equivalents. This prevents the vocabulary from being polluted by multiple token IDs for semantically identical glyphs, a critical step for multilingual models.

Removes leading/trailing whitespace and collapses multiple consecutive whitespace characters (spaces, tabs, newlines) into a single space. This prevents the tokenizer from generating spurious tokens for empty space and ensures consistent segmentation regardless of source formatting. In SentencePiece, whitespace is treated as a meta-symbol and escaped, enabling lossless reconstruction.

Removes diacritical marks from characters, mapping 'é' to 'e' and 'ñ' to 'n'. This is a domain-specific optimization commonly applied in English-centric search systems to handle queries typed without accents. It is a lossy operation that can conflate distinct words in languages where diacritics carry phonemic weight (e.g., Spanish 'año' vs. 'ano'), and is generally avoided in modern multilingual tokenizers.

Strategies range from stripping all punctuation to normalizing it to a canonical set. Stripping is common in traditional information retrieval to prevent mismatches between 'end-of-sentence.' and 'end-of-sentence'. However, modern subword tokenizers like BPE preserve punctuation as distinct tokens because it carries syntactic signals. A compromise is normalizing Unicode smart quotes and dashes to their ASCII equivalents.

Strips or escapes HTML tags, XML markup, and Markdown syntax from web-scraped corpora. This prevents structural markup from being tokenized as content. For example, '<div class="header">' should be removed or replaced with a space rather than generating tokens for '<div', 'class', and '="header">'. This step is essential in RAG pipelines that ingest crawled documentation.

The initial splitting of raw text into coarse units, typically by whitespace and punctuation, before a subword model applies its merge rules. This step defines the 'words' that BPE or WordPiece will further segment.

• Example: Splitting "Hello, world!" into ["Hello", ",", "world", "!"]
• Function: Sets the upper boundary for token boundaries
• Impact: Prevents subword merges from crossing explicit word boundaries

A technique that stochastically samples different segmentations during training to improve model robustness. Implemented via BPE-Dropout, it randomly skips merge rules to expose the model to varied token boundaries.

• Goal: Reduce sensitivity to tokenization variance
• Mechanism: Drops merges with probability p during training
• Benefit: Better handling of misspellings and rare morphological forms

A property where the original input text can be perfectly reconstructed from the token sequence. SentencePiece achieves this by treating text as a raw Unicode sequence and encoding whitespace as a special meta-symbol.

• Key Feature: decode(encode(text)) == text
• Mechanism: Whitespace is preserved as '▁' (U+2581)
• Contrast: Standard BPE with pre-tokenization is lossy with whitespace

A metric evaluating tokenization efficiency, calculated as the number of raw bytes divided by the number of tokens produced. Higher ratios indicate more efficient encoding.

• Formula: len(text_bytes) / len(token_ids)
• Example: GPT-4 achieves ~2.5 bytes/token on English text
• Trade-off: Larger vocabularies increase compression but expand embedding parameters

Reserved vocabulary entries with specific control functions added during post-processing. These tokens are never split by the tokenizer and signal structural boundaries to the model.

• [CLS]: Classification token, aggregates sequence representation
• [SEP]: Separates segments in pair-input tasks
• [MASK]: Denotes tokens for masked language modeling
• [PAD]: Ensures uniform batch dimensions

A structured formatting script that converts a sequence of chat messages into a single tokenized string. It inserts the appropriate control tokens for a specific instruction-tuned model.

• Function: apply_chat_template(messages) -> string
• Example: Llama 3 wraps user turns with <|start_header_id|>user<|end_header_id|>
• Importance: Ensures correct role delineation for conversational models