Tokenization

Modern tokenizers use subword algorithms like Byte-Pair Encoding (BPE) to split text into statistically frequent units. This balances vocabulary size against out-of-vocabulary issues.

• Rare words are decomposed into meaningful sub-units (e.g., 'tokenization' → 'token' + 'ization')
• Common words remain as single tokens to preserve semantic integrity
• Enables models to handle morphologically rich languages and novel compounds without an infinite vocabulary

A tokenizer's vocabulary is a fixed lookup table mapping integer IDs to specific byte sequences. This vocabulary is learned from the pre-training corpus and frozen before model training.

• Typical vocabularies range from 32,000 to 256,000 tokens
• Includes special tokens: <|endoftext|>, <|user|>, <|assistant|> for structural control
• The vocabulary size directly impacts embedding matrix memory and model throughput

Tokenization is a bidirectional process. Encoding maps raw text to integer IDs; decoding maps IDs back to text. Lossless reconstruction is critical for production systems.

• Encoding: "Hello world!" → [15496, 995, 0]
• Decoding: [15496, 995, 0] → "Hello world!"
• Byte-level tokenizers guarantee round-trip fidelity by operating on raw UTF-8 bytes, preventing character corruption

The context window is the maximum number of tokens a model can process in a single forward pass. Tokenization directly determines how much text fits within this budget.

• A 4,096-token window may hold only ~3,000 English words depending on token density
• Code and non-English languages often consume 2-3x more tokens per semantic unit
• Token counting is essential for prompt engineering and cost estimation in pay-per-token APIs

The tokenization strategy introduces subtle but measurable biases in model behavior. Splitting patterns affect arithmetic, spelling, and reasoning capabilities.

• Number tokenization: '123' may be split as '12' + '3', disrupting arithmetic understanding
• Whitespace sensitivity: Leading spaces can produce entirely different token sequences for the same word
• Language disparity: Languages with non-Latin scripts often require 4-5x more tokens than English for equivalent semantic content

Special tokens act as control codes that structure conversations, separate documents, and signal generation boundaries. They are integral to instruction-tuned model behavior.

• Chat templates use tokens like <|im_start|>user and <|im_start|>assistant to delineate turns
• Beginning-of-sequence (BOS) and end-of-sequence (EOS) tokens frame generation
• Misconfigured special tokens during inference cause format collapse and degraded output quality

The dominant subword tokenization algorithm used by GPT models. BPE starts with a vocabulary of individual characters and iteratively merges the most frequent adjacent pairs to form new tokens. This creates a deterministic vocabulary that efficiently handles open-vocabulary problems—rare words are decomposed into known subword units rather than replaced with an [UNK] token. The final vocabulary size is a hyperparameter, typically ranging from 50k to 100k tokens.

A language-independent tokenization framework that treats the input text as a raw byte stream, eliminating the need for language-specific pre-tokenization. It implements both BPE and the Unigram language model algorithm. Unlike standard BPE, SentencePiece normalizes text and manages whitespace by escaping spaces as a special meta-character '▁' (U+2581), ensuring lossless and reversible tokenization.

The subword tokenization algorithm originally developed for the BERT model. Unlike BPE, which merges based on frequency, WordPiece selects merges that maximize the likelihood of the training data. It uses a greedy longest-match-first strategy during encoding and prepends '##' to subword tokens that do not begin a word. This approach is highly effective for handling morphologically rich languages and technical terminology.

The maximum number of tokens a model can process in a single forward pass, defining its immediate working memory. The tokenizer directly determines the effective context length—a 4k token window may represent anywhere from 2,500 to 3,500 English words depending on text complexity. Exceeding this limit requires truncation or chunking strategies. Modern models extend this via RoPE Scaling to 128k tokens or more.

Reserved vocabulary entries that control model behavior rather than representing text. Critical examples include:

• <|endoftext|>: Marks document boundaries during pre-training.
• <|im_start|> / <|im_end|>: Delimit conversational turns in chat-tuned models.
• <|system|> / <|user|> / <|assistant|>: Define role-specific message segments. Improper handling of these tokens during fine-tuning or inference leads to format corruption and degraded output quality.