Tokenization

Modern tokenizers use subword algorithms like Byte-Pair Encoding (BPE) or WordPiece to balance vocabulary size and coverage. Instead of storing every word, they decompose rare terms into frequent subword units.

• Example: The word 'tokenization' might split into ['token', 'ization']
• Benefit: Eliminates out-of-vocabulary (OOV) tokens entirely
• Mechanism: Iteratively merges the most frequent adjacent byte pairs in a training corpus

The tokenizer's vocabulary is a fixed lookup table mapping subword units to unique integer IDs. This vocabulary is built during a training phase on a representative corpus and remains frozen during inference.

• Size: Typically 30,000 to 250,000 tokens for large language models
• Impact: A larger vocabulary encodes more information per token but increases embedding matrix size
• Special Tokens: Includes control tokens like [CLS], [SEP], [PAD], and [UNK]

The token count of input text directly consumes the model's fixed context window. Tokenization efficiency varies by language and domain, creating a hidden performance multiplier.

• English: ~1.3 tokens per word on average
• Code: Often more token-efficient due to repetitive syntax
• Non-English: Languages like Thai or Japanese can require 3-5x more tokens per semantic unit
• Strategy: Token-aware chunking prevents mid-thought truncation

Byte-level tokenizers like those used in GPT models operate directly on UTF-8 bytes rather than Unicode characters. This provides a universal foundation that can represent any text without a separate pre-tokenization step.

• Advantage: Robust handling of emoji, code, and malformed text
• Base Vocabulary: 256 byte values serve as the atomic units
• Result: A truly language-agnostic tokenization scheme

Mismatched tokenization between the embedding model and the generative model causes silent performance degradation. The same text can produce different token sequences depending on the tokenizer.

• Symptom: Poor retrieval quality despite correct semantic indexing
• Cause: Embedding model uses WordPiece, but LLM uses BPE
• Mitigation: Always pair models with their canonical tokenizer; never mix tokenizer artifacts across model families

Before subword splitting, text undergoes normalization steps that permanently alter the input. These transformations are lossy and irreversible.

• Lowercasing: Destroys case information (BERT uncased)
• Unicode Normalization: NFKC/NFKD forms standardize characters
• Whitespace Splitting: Many tokenizers split on whitespace first, making them blind to languages without spaces
• Critical Rule: Apply identical normalization at both indexing and query time

A neural network that maps tokenized input IDs into continuous, high-dimensional vector representations. The tokenizer and embedding model are tightly coupled—each model requires its specific tokenizer vocabulary. Using mismatched pairs causes catastrophic semantic drift.

• Sentence Transformers: Bi-encoders optimized for semantic similarity
• Vocabulary Size: Typically 30k-50k subword tokens
• Output: Fixed-length dense vectors (e.g., 768, 1024, or 1536 dimensions)

The maximum sequence length of tokens an embedding or language model can process in a single forward pass. Tokenization directly determines how much text fits within this budget—a 512-token context window may hold vastly different amounts of semantic content depending on the tokenizer's efficiency.

• GPT-4 Turbo: 128k token context window
• Claude 3: 200k token context window
• Impact: Defines the upper boundary for chunk size in retrieval pipelines

The methodology for segmenting documents into discrete, semantically coherent pieces that fit within an embedding model's context window. Tokenization is the fundamental constraint here—chunk boundaries must respect token counts, not just character or word counts.

• Fixed-size chunking: Splits at exact token count thresholds
• Semantic chunking: Uses embedding similarity to find natural break points
• Recursive splitting: Applies hierarchical separators (paragraph → sentence → word)

The subword algorithm used by BERT and its derivatives. Unlike BPE which merges by frequency, WordPiece selects merges that maximize the likelihood of the training data, using a greedy longest-match-first approach during tokenization.

• ## prefix: Indicates a subword continuation token
• Vocabulary: Typically 30,522 tokens for BERT-base
• Key difference: Likelihood-based merging vs. frequency-based merging