Tokenization

Word tokenization splits text into discrete words based on whitespace and punctuation. It is the most intuitive method, creating a vocabulary where each unique word is a token.

• Example: "Don't split this!" becomes ["Don't", "split", "this", "!"].
• Pros: Simple, human-interpretable.
• Cons: Leads to large vocabularies, poor handling of out-of-vocabulary (OOV) words (e.g., "tokenization" → <UNK>), and inconsistent treatment of contractions and compound words.

Subword tokenization splits text into units smaller than words but larger than characters, such as prefixes, suffixes, and common character sequences. This is the dominant method in modern LLMs (e.g., GPT, LLaMA).

• Key Algorithms: Byte-Pair Encoding (BPE), WordPiece, Unigram Language Model.
• Example: "tokenization" might be split into ["token", "ization"].
• Pros: Balances vocabulary size and semantic meaning, effectively handles OOV words by breaking them into known subwords, and improves model efficiency.

Character tokenization treats each individual character (including letters, digits, and punctuation) as a token.

• Example: "cat" becomes ["c", "a", "t"].
• Pros: Extremely small vocabulary (e.g., < 1000 tokens), eliminates OOV problems entirely.
• Cons: Produces very long sequences for the model to process, obscuring semantic and morphological relationships. It is computationally inefficient for standard language modeling but can be useful for specialized tasks like spelling correction or generative work on rare scripts.

Sentence tokenization (or sentence segmentation) splits a document into its constituent sentences. This is a higher-level, often rule-based, preprocessing step that typically precedes word or subword tokenization.

• Example: "Hello world. How are you?" becomes ["Hello world.", "How are you?"].
• Key Challenge: Sentence boundary disambiguation (e.g., distinguishing periods that end sentences from those in abbreviations like "Dr." or decimals).
• Use Case: Critical for semantic chunking and sentence window retrieval in RAG systems, where preserving full sentence integrity maintains context.

Byte-level tokenization operates on the raw byte representation of text. A prominent example is the Byte-level BPE (BBPE) used in models like GPT-2.

• Mechanism: It applies BPE merges directly to UTF-8 byte sequences.
• Pros: Truly vocabulary-free; can represent any text in any language without predefining a character set. This makes it highly flexible for multilingual and code corpora.
• Cons: Can produce longer sequences than character-level tokenization for non-Latin scripts, as a single character may be encoded as multiple bytes.

Whitespace tokenization is a simplistic form of word tokenization that splits text only on whitespace characters (spaces, tabs, newlines), ignoring punctuation.

• Example: "Hello, world! How's it going?" becomes ["Hello,", "world!", "How's", "it", "going?"].
• Pros: Fast and trivial to implement.
• Cons: Punctuation remains attached to words, creating redundant vocabulary entries (e.g., "world" and "world!" are separate tokens). It is rarely used in production NLP systems but serves as a basic baseline.

Byte-Pair Encoding (BPE) is a subword tokenization algorithm that builds a vocabulary by iteratively merging the most frequent pairs of characters or character sequences in a training corpus. It starts with a base vocabulary of individual characters and creates new tokens for common subword combinations (like 'ing' or 'ed'), allowing the model to handle out-of-vocabulary words by breaking them into known subword units. This is the core algorithm behind tokenizers for models like GPT and LLaMA.

• Key Mechanism: Greedy, frequency-based merging.
• Primary Use: Balancing vocabulary size with the ability to represent any word.
• Example: The word 'playing' might be tokenized as ['play', 'ing'] if 'play' and 'ing' are in the learned subword vocabulary.

SentencePiece is a language-agnostic, unsupervised text tokenizer and detokenizer that implements subword units like BPE and Unigram Language Model directly on raw text. A key differentiator is that it treats the input as a raw Unicode sequence, allowing tokenization without pre-tokenization (like splitting on spaces), which is particularly useful for languages without clear word boundaries (e.g., Chinese, Japanese). It is widely used for models like T5 and many multilingual systems.

• Key Feature: Trains directly on raw sentences, enabling whitespace to be treated as a normal character.
• Algorithms: Supports both BPE and Unigram segmentation.
• Benefit: Simplifies preprocessing pipelines and improves consistency across diverse languages.

A context window is the fixed maximum sequence length of tokens that a language model can process in a single forward pass. This is a hardware and architecture-defined constraint (e.g., 128K tokens for Claude 3, 8K for GPT-3.5-Turbo) that directly governs chunking strategy. The total input—user query + system instructions + retrieved chunks + conversation history—must fit within this limit. Exceeding it requires truncation, which can discard critical context.

• Constraint: Dictates the maximum combined size of retrieved evidence.
• Implication for Chunking: Chunk sizes must be optimized to leave sufficient space for the query, instructions, and model output.
• Management: Techniques include sliding windows and hierarchical retrieval to work within this fixed budget.

Sentence Boundary Detection (SBD) is the natural language processing task of identifying where sentences begin and end in plain text. It is a critical preprocessing step for semantic and sentence-based chunking strategies, as splitting within a sentence can destroy its meaning and degrade retrieval quality. SBD is more complex than splitting on periods due to abbreviations (e.g., 'Dr.'), decimal points, and ellipses.

• Challenge: Distinguishing sentence-ending punctuation from punctuation used in abbreviations, numbers, and titles.
• Tools: Libraries like spaCy, NLTK, and specialized neural models provide robust SBD.
• Impact: High-quality SBD is essential for creating coherent, self-contained chunks that preserve semantic integrity.

Truncation is the process of cutting off tokens from a text sequence to fit it within a model's maximum context length. When the combined input (prompt + retrieved chunks) exceeds the context window, a truncation strategy must be applied. Common methods include removing tokens from the middle of long chunks or discarding the least relevant retrieved documents. Poor truncation can remove key information, leading to factual errors or hallucinations in the model's response.

• Strategies: Truncate from the start, end, or middle of long sequences.
• Risk: Indiscriminate truncation can sever critical causal dependencies in the text.
• Best Practice: Implement intelligent chunk selection and compression before resorting to brute-force truncation.

Text Normalization is a preprocessing step that standardizes raw text into a consistent, canonical format before tokenization and chunking. This reduces noise and ensures the same concept is represented identically across documents. It is crucial for improving the recall of lexical and hybrid search systems.

Common normalization operations include:

• Case Folding: Converting all text to lowercase.
• Unicode Normalization: Converting characters to a standard composed form (NFC).
• Accent Removal: Stripping diacritical marks (e.g., 'café' -> 'cafe').
• Contraction Expansion: Converting "don't" to "do not".
• Whitespace Standardization: Replacing multiple spaces/tabs with a single space.

Impact: Consistent normalization ensures the tokenizer and embedding models process equivalent text identically, improving retrieval consistency.