Encoding

Encoding is fundamentally a lookup operation. After the tokenizer segments text into tokens, each token is mapped to its unique Token ID from the vocabulary. This integer sequence is the model's sole linguistic input.

• The vocabulary is a fixed mapping of token strings to integer indices.
• A token not in the vocabulary results in an Out-of-Vocabulary (OOV) error, which subword algorithms like BPE are designed to prevent.
• The output is a list of integers, e.g., [101, 2054, 2003, 4829, 102].

Encoding is not a single step but a sequential pipeline of operations. Raw text passes through multiple stages before becoming token IDs.

• Normalization: Standardizes text (e.g., lowercasing, Unicode NFC normalization).
• Pre-tokenization: Splits text into coarse units, typically on whitespace and punctuation.
• Model Application: The core subword algorithm (e.g., BPE, WordPiece) applies merge rules to segment words into subwords.
• Post-processing: Adds Special Tokens like [CLS] and [SEP], and generates the Attention Mask.

Encoding injects control signals directly into the token stream. These reserved vocabulary entries structure the input for the model.

• [CLS]: A classification token prepended to the sequence; its final hidden state represents the entire input.
• [SEP]: A separator token used to demarcate sentence boundaries or turns in a conversation.
• [PAD]: A padding token used to make all sequences in a batch the same length for parallel processing.
• [MASK]: A token used in pre-training to hide a word for the model to predict.

For efficient processing, multiple encoded sequences are batched together. Encoding handles the necessary tensor alignment.

• Padding: Shorter sequences are extended with [PAD] tokens to match the length of the longest sequence in the batch.
• Attention Mask: A binary tensor is generated simultaneously. A value of 1 indicates a real content token, and 0 indicates a [PAD] token that the model's self-attention mechanism must ignore.
• This ensures the model does not learn from the padding structure.

Decoding is the reverse process of converting a sequence of token IDs back into a human-readable string. The fidelity of this process depends on the tokenizer.

• A standard BPE tokenizer may not be perfectly lossless, as whitespace information can be ambiguous.
• SentencePiece guarantees Lossless Tokenization by treating the input as a raw Unicode sequence and preserving whitespace as a special meta-symbol.
• This allows for perfect reconstruction of the original text from the token IDs.

The quality of an encoding strategy is measured by its Compression Ratio, which is the number of raw bytes divided by the number of tokens produced.

• A higher ratio means fewer tokens are needed to represent the same text, reducing the model's compute cost.
• Subword tokenization achieves high compression by encoding common words as single tokens and rare words as a few subword units.
• For example, 'unbelievably' might be encoded as ['un', 'believ', 'ably']—three tokens instead of one, but still far fewer than character-level encoding.