Tokenization

Tokenization operates on a spectrum from coarse to fine granularity, each with distinct trade-offs:

• Word Tokenization: Splits on whitespace and punctuation. Simple but creates large vocabularies and fails on unseen words.
• Subword Tokenization: Breaks words into meaningful fragments (e.g., 'un' + 'break' + 'able'). Balances vocabulary size with OOV handling.
• Character Tokenization: Treats each character as a token. Minimal vocabulary but produces very long sequences, challenging for transformers.

Modern LLMs overwhelmingly use subword algorithms like Byte-Pair Encoding (BPE) or SentencePiece to navigate this trade-off.

A primary function of tokenization is eliminating the Out-of-Vocabulary (OOV) problem. Word-level tokenizers fail catastrophically on novel terms like 'GPT-4o' or misspelled words, mapping them to a generic <UNK> token.

Subword tokenizers solve this by recursively decomposing unknown words into known constituent parts:

• 'unhappiness' → 'un' + 'happiness'
• 'happiness' → 'happi' + 'ness'

This guarantees that any input text can be represented, even if the full word was never seen during training. The Byte-fallback strategy in models like GPT extends this further by encoding raw bytes, ensuring universal coverage.

Byte-Pair Encoding (BPE) is the dominant vocabulary construction algorithm. It starts with a base vocabulary of individual characters and iteratively merges the most frequent adjacent pair until a target vocabulary size is reached.

Key properties:

• Data-driven: The vocabulary is learned from the training corpus, not predefined by linguistic rules.
• Frequency-based: Common words remain intact as single tokens; rare words get split.
• Reversible: The merge table allows perfect reconstruction of the original text from token IDs.

This is why 'the' is a single token in GPT models, while 'tokenization' might be split into 'token' + 'ization'.

Effective tokenization requires upstream text normalization to produce consistent tokens:

• Case Folding: Lowercasing ensures 'Apple' and 'apple' map to the same token, unless case carries semantic meaning (e.g., 'Apple' the company vs. 'apple' the fruit).
• Unicode Normalization: NFC or NFD forms ensure visually identical characters have identical byte representations.
• Contraction Expansion: Splitting 'don't' into 'do' + "n't" prevents a proliferation of contracted tokens.

Inconsistent preprocessing creates token mismatch, where semantically identical text produces different token sequences, degrading retrieval and generation quality.

Tokenization directly governs the economics of transformer inference. The context window—the maximum sequence length a model can process—is measured in tokens, not words or characters.

Implications:

• A 128k token context window may hold only ~96k English words due to subword splitting.
• Non-English languages often tokenize less efficiently; the same sentence in Burmese can consume 4-5x more tokens than in English.
• Token billing: API costs are per-token, making tokenization efficiency a direct cost driver.

Choosing a tokenizer optimized for your target language and domain can significantly reduce inference latency and expense.

A well-designed tokenizer is deterministic and losslessly reversible. Given the same input string and tokenizer, the output token sequence must be identical every time. The detokenization process must reconstruct the original text exactly.

Challenges to reversibility:

• Whitespace ambiguity: Tokenizers like BPE use special prefix symbols (e.g., 'Ġ' in GPT-2) to encode space location, enabling perfect reconstruction.
• Normalization drift: If the detokenizer applies different Unicode normalization than the tokenizer, the output text will differ from the input.

This property is critical for code generation, where a single character error breaks syntax, and for retrieval systems where exact string matching is required post-generation.

The process of assigning a grammatical category—such as noun, verb, or adjective—to each token. POS tagging is a critical prerequisite for lemmatization and syntactic parsing, as the correct lemma depends on the word's part of speech.

• Enables disambiguation of homographs like 'lead' (noun vs. verb)
• Typically uses a Hidden Markov Model or transformer-based architecture
• Essential for accurate dependency parsing

The removal of high-frequency, low-information words—such as 'the', 'is', and 'at'—from a token stream. This reduces dimensionality and noise in the feature space, improving computational efficiency for sparse retrieval models like BM25.

• Stop word lists are language-specific and domain-dependent
• Modern dense retrieval models often retain stop words for context
• Over-aggressive filtering can destroy phrase semantics

A contiguous sequence of N tokens from a text sample. Moving beyond unigrams to bigrams and trigrams captures local word order and multi-word expressions like 'New York' or 'machine learning'.

• Unigram: single token
• Bigram: two-token sequence
• Trigram: three-token sequence
• Essential feature for traditional text classification and language modeling

The normalization step of converting all characters to a single case, typically lowercase, to ensure that tokens like 'Apple' and 'apple' are treated identically. This reduces vocabulary size and improves recall.

• Critical for case-insensitive search engines
• Can destroy information for Named Entity Recognition (e.g., 'Apple' the company vs. 'apple' the fruit)
• Often combined with truecasing for downstream tasks

The challenge of managing tokens encountered during inference that were absent from the model's fixed vocabulary. Subword tokenization strategies like BPE and WordPiece mitigate OOV issues by decomposing unknown words into known subword fragments.

• Character-level tokenization eliminates OOV entirely but loses semantic granularity
• The [UNK] token is a fallback in word-level vocabularies
• OOV rate is a key metric for tokenizer quality