Tokenization

Tokenization converts raw, unstructured text into discrete tokens—the fundamental units of meaning that a model processes. These tokens are mapped to unique token IDs, which are integer indices used to look up dense vector representations in the model's token embedding layer. This conversion is the bridge between human language and machine computation.

The vocabulary is a fixed set of all known tokens, defined during training. Its vocabulary size is a critical hyperparameter representing a trade-off:

• Larger vocabularies capture more unique words but increase the model's embedding parameters and memory footprint.
• Smaller vocabularies are more compact but rely heavily on subword segmentation, potentially losing semantic granularity. The goal is to minimize out-of-vocabulary (OOV) occurrences, a problem where a word is absent from the vocabulary, which is largely solved by modern subword algorithms.

The process is a multi-stage tokenization pipeline that standardizes and segments text:

1. Normalization: Standardizes text via lowercasing, Unicode normalization (e.g., NFKC), and whitespace stripping.
2. Pre-tokenization: Splits text into coarse units, typically on whitespace and punctuation.
3. Model Application: A subword algorithm like Byte-Pair Encoding (BPE) or WordPiece applies merge rules to segment words into subword tokens.
4. Post-processing: Adds special tokens like [CLS], [SEP], and generates the attention mask.

These are the two fundamental operations of a tokenizer:

• Encoding: The forward process of converting a raw text string into a sequence of token IDs using the tokenizer's vocabulary and merge rules. This is the input to the model.
• Decoding: The inverse process of converting a sequence of token IDs back into a human-readable string. A key property of models like SentencePiece is lossless tokenization, where the original input can be perfectly reconstructed without information loss, as it treats the input as a raw Unicode sequence.

Neural networks process inputs in parallel batches, requiring uniform sequence lengths. Padding is the strategy of adding a special [PAD] token to shorter sequences in a batch to achieve this uniformity. A corresponding binary attention mask is generated to instruct the model's self-attention mechanism to ignore these padding tokens, ensuring they don't corrupt the contextual representations of genuine content tokens.

The compression ratio is a key metric for evaluating a tokenizer's efficiency. It is calculated as the number of raw bytes in the input text divided by the number of tokens produced. A higher ratio indicates a more efficient encoding, where fewer tokens are needed to represent the same amount of information. This directly impacts the context window a model can process and the computational cost of inference.