Vocabulary

A vocabulary is a static lookup table constructed during the tokenizer's training phase and remains frozen during model training and inference. It defines the complete universe of tokens the model can understand. Any input text must be segmented into sequences of these known tokens. The vocabulary maps each unique token to a Token ID, a unique integer index that serves as the numerical input to the model's embedding layer.

The vocabulary size is a critical hyperparameter representing a fundamental trade-off:

• Larger vocabularies: Allow for shorter sequence lengths (higher compression ratio) and can represent whole words, but increase the model's embedding matrix parameters and memory footprint.
• Smaller vocabularies: Reduce the number of trainable parameters but force the tokenizer to split text into more subword units, increasing sequence length and potentially losing semantic granularity. Typical sizes range from 32k (SentencePiece) to 50k+ (GPT models).

Vocabularies reserve specific integer IDs for control and structural tokens that are not part of natural language. These are essential for model formatting and task-specific behavior:

• [CLS]: A classification token whose final hidden state represents the entire sequence.
• [SEP]: A separator token used to demarcate sentence boundaries.
• [MASK]: A placeholder token for masked language modeling objectives.
• [PAD]: A padding token used to standardize sequence lengths within a batch.
• [UNK]: An unknown token representing out-of-vocabulary words (largely obsolete with subword tokenization).

An Out-of-Vocabulary (OOV) condition occurs when a word or character sequence is absent from the model's fixed vocabulary. In early word-level tokenizers, this was a critical failure mode, forcing the model to map all unknown words to a single [UNK] token, resulting in total information loss. Modern subword tokenization strategies like BPE and WordPiece largely mitigate this by segmenting unknown words into known subword units, ensuring every input can be represented.

The vocabulary is the central component in two inverse processes:

• Encoding: The conversion of a raw text string into a sequence of Token IDs by applying the tokenizer's merge rules and vocabulary lookup. This produces the numerical input for the model.
• Decoding: The inverse process of converting a sequence of Token IDs back into a human-readable text string. In lossless tokenizers like SentencePiece, the original input can be perfectly reconstructed without information loss.

Each Token ID in the vocabulary is associated with a learned, dense vector representation called a Token Embedding. This embedding matrix is the first layer of the neural network and is trained jointly with the rest of the model. The dimensionality of these vectors is the model's hidden size. The embedding layer captures the semantic and syntactic properties of each token, positioning similar tokens close together in the high-dimensional vector space.

The dominant subword algorithm that constructs the vocabulary by iteratively merging the most frequent adjacent pairs of bytes or characters. Starting with individual characters, BPE builds a set of subword units that efficiently represent a corpus. This directly determines the final vocabulary size and the list of token IDs the model will recognize, balancing encoding efficiency against the risk of fragmenting rare words into many tokens.

Reserved vocabulary entries with specific control functions, added to the core vocabulary derived from the training data. These are not natural language but structural signals for the model. Key examples include:

• [CLS]: Marks the start of a sequence for classification.
• [SEP]: Separates segments, like question and context.
• [MASK]: Hides a token for masked language modeling.
• [PAD]: Ensures uniform sequence length in a batch.

The two-way mapping process governed by the vocabulary.

• Encoding: Converts a raw text string into a sequence of token IDs by applying normalization, pre-tokenization, and the subword merge rules defined in the vocabulary.
• Decoding: The inverse process of converting a sequence of token IDs back into a human-readable string. A lossless tokenization model like SentencePiece guarantees perfect reconstruction of the original text.

A condition where a word or character sequence is absent from a model's fixed vocabulary. A core purpose of subword tokenization is to eliminate this issue. By segmenting unknown words into known subword units (e.g., 'un' + 'known'), the model can still process and represent any input text. A byte-level BPE strategy provides a definitive solution by operating on raw bytes, guaranteeing a base vocabulary of 256 tokens and zero unknown tokens.

Critical post-processing steps for efficient batch processing. Since sequences have different lengths, a [PAD] token (with a specific token ID) is appended to shorter sequences to create a uniform tensor. The attention mask is a binary tensor generated simultaneously, instructing the model's self-attention mechanism to ignore these padding tokens (value 0) and only process genuine content tokens (value 1), preventing computation on meaningless data.

A critical hyperparameter defining the total number of unique tokens. It represents a fundamental trade-off:

• Larger Vocabulary: Encodes text with fewer tokens (higher compression ratio), but increases the model's embedding matrix size and parameter count.
• Smaller Vocabulary: Reduces model parameters but forces longer token sequences to represent the same text, potentially losing nuanced meaning as words are heavily fragmented. Typical sizes range from 32k to 250k tokens.