Tokenization

Modern tokenization relies on subword segmentation rather than whole-word or character-level splitting. This balances vocabulary size with the ability to represent open-vocabulary text.

• Byte-Pair Encoding (BPE) iteratively merges the most frequent adjacent byte or character pairs to build a vocabulary of subword units.
• WordPiece uses a likelihood-based merging criterion, selecting pairs that maximize the training data's probability.
• Unigram LM starts with a large vocabulary and prunes it by removing tokens that least impact the overall likelihood.

Subword tokenization ensures that rare, misspelled, or novel words can still be represented as sequences of known fragments, preventing the dreaded <UNK> token.

The tokenizer's output is not text but a sequence of integer IDs. The tokenizer maintains a fixed vocabulary—a bidirectional lookup table mapping each unique subword string to a unique integer index.

• The vocabulary is frozen after training; the model can only understand tokens within this set.
• The total vocabulary size (e.g., 50,257 for GPT-2, 32,000 for Llama) is a critical architectural hyperparameter.
• This mapping is the sole interface between raw human language and the model's embedding layer, which converts each integer ID into a high-dimensional vector.

Tokenizers inject special tokens that do not correspond to text but serve as structural control signals for the model.

• BOS (Beginning of Sequence) and EOS (End of Sequence) tokens demarcate utterance boundaries and act as generation halting signals.
• PAD (Padding) tokens ensure all sequences in a batch have uniform length for parallel processing.
• UNK (Unknown) tokens represent out-of-vocabulary characters, though modern subword tokenizers largely eliminate their need.
• Control tokens like <|user|> and <|assistant|> in chat models delineate conversational turns and roles.

The tokenization algorithm introduces inherent biases that propagate through the model's behavior.

• Integer splitting causes a single number like 380 to be tokenized as ['38', '0'], destroying its holistic numerical value and forcing the model to learn arithmetic via fragmented representations.
• Whitespace handling varies: GPT-style tokenizers treat leading spaces as part of the token, while others do not, causing inconsistent representations of the same word in different contexts.
• Language disparity arises because the training corpus for the tokenizer determines token fertility—the number of tokens needed to represent a concept. A word in a low-resource language may require 4-5x more tokens than its English equivalent, consuming more context window and increasing inference cost.

A common operational pitfall is conflating character count with token count. The relationship is non-linear and language-dependent.

• In English, a token averages roughly 4 characters or 0.75 words. The sentence "I love AI" is 3 tokens, while "I love artificial intelligence" is 5.
• Code and whitespace-heavy formats like JSON are token-inefficient; a single curly brace or indentation space is its own token.
• Non-Latin scripts (e.g., Hindi, Arabic) often require multiple tokens per character due to Unicode byte representation in the tokenizer's training data.
• This discrepancy is critical for context window budgeting: a 4,096-token limit may correspond to anywhere from 2,500 to 3,500 English words, but significantly fewer in other languages.

A tokenizer is a standalone, pre-trained module trained on a raw text corpus before the language model itself is trained.

• The training corpus determines the vocabulary; a tokenizer trained on English Wikipedia will perform poorly on Korean or Python code.
• The vocabulary size is a fixed hyperparameter: too small, and common words fragment into many tokens (increasing sequence length); too large, and the embedding matrix becomes memory-prohibitive.
• Once trained, the tokenizer is deterministic—the same input always produces the same token sequence. This determinism is essential for reproducible inference.
• Tokenizer choice is a binding architectural decision; switching from a GPT-2 tokenizer to a Llama tokenizer requires retraining the entire model from scratch.

The dominant subword tokenization algorithm that constructs a vocabulary by iteratively merging the most frequent adjacent pairs of bytes or characters in a corpus. BPE solves the out-of-vocabulary problem by decomposing rare words into known subword units.

• Training Phase: Starts with a base vocabulary of all unique characters; repeatedly merges the highest-frequency bigram until reaching a target vocabulary size (e.g., 50k tokens).
• Inference: Splits an unseen word like "tokenization" into known subwords such as ["token", "ization"].
• GPT Models: OpenAI's GPT series uses a variant of BPE with a 100k+ token vocabulary.

The core architectural component of the Transformer that computes a weighted relevance score for every token relative to every other token in the sequence. Tokenization directly determines the granularity of these attention calculations.

• Self-Attention: Each token attends to all other tokens, including itself, to build a context-aware representation.
• Token Granularity Impact: A word tokenized as ["un", "believ", "able"] forces the model to learn attention patterns across three subword units rather than one holistic representation.
• Quadratic Cost: The computational complexity is O(n²) where n is the number of tokens, making tokenization efficiency a direct cost multiplier.

The hard numerical ceiling on the number of tokens an LLM can accept as input or generate as output, governed by the model's architecture and serving infrastructure. Distinct from the context window, the token limit is a strict API or system constraint.

• Input vs. Output: Many models enforce separate limits for input tokens (prompt) and output tokens (completion).
• Tokenizer Variance: The same text produces different token counts across models. "Hello, world!" is 4 tokens in GPT-4 but may be 5–6 in other tokenizers.
• Truncation Strategy: When content exceeds the limit, systems must truncate—often losing critical context from the middle of the document.

A documented performance failure mode where information placed in the center of a long context window is significantly less likely to be retrieved or utilized compared to content at the beginning or end. Tokenization strategy influences how content is positioned within this attention gradient.

• Primacy/Recency Bias: Models exhibit strong recall for tokens at the start (primacy) and end (recency) of the context.
• Chunking Mitigation: Strategic tokenization and semantic chunking can ensure critical facts are positioned at context boundaries.
• Empirical Finding: In multi-document QA tasks, accuracy can drop by 20–30% when the relevant document is placed in the middle of a 10k+ token context.

A content segmentation strategy that divides documents into chunks based on semantic boundaries—paragraph breaks, topic shifts, or embedding similarity thresholds—rather than fixed token counts. The chunking strategy must align with the target model's tokenization scheme.

• Token-Aware Splitting: Chunkers often use the target model's tokenizer to ensure chunks don't exceed the embedding model's token limit (e.g., 512 tokens for text-embedding-ada-002).
• Overlap Strategy: Maintaining a 10–20% token overlap between chunks prevents meaning fragmentation at boundaries.
• RAG Integration: Properly chunked and tokenized content ensures retrieved context fits within the generator model's context window without truncation.