Vocabulary Size

The vocabulary size directly determines the dimensions of the model's input and output embedding matrices. For a vocabulary of size V and hidden dimension d_model, the embedding layer contains V × d_model parameters. In large models, this can constitute a significant fraction of total parameters. For example, a vocabulary of 50,000 tokens with a hidden size of 4,096 results in over 200 million parameters in the embedding layer alone, before any transformer blocks are added.

A larger vocabulary enables more efficient encoding by representing common words and subwords as single tokens, reducing sequence length. However, it increases the risk of under-trained token embeddings for rare entries. A smaller vocabulary forces longer token sequences for the same text, increasing computational cost in the attention layers. The optimal size balances these factors for the target domain and language.

Vocabulary size is the primary defense against the Out-of-Vocabulary (OOV) problem. A vocabulary that is too small for a multilingual or code-heavy corpus will force the tokenizer to fall back to byte-level or character-level tokens, drastically increasing sequence length. Modern subword tokenizers like Byte-level BPE guarantee zero unknown tokens by including all 256 bytes in the base vocabulary, but the effective vocabulary size still governs encoding efficiency.

The final language modeling head projects the hidden state to a vector of size V for the softmax operation. A larger vocabulary directly increases the computational cost and memory footprint of this output layer. Techniques like sampled softmax or candidate sampling are often employed during training to make very large vocabularies tractable, but at inference time, the full projection may still be required for accurate next-token prediction.

When fine-tuning a pre-trained model on a specialized domain like medicine or law, the original vocabulary may lack critical domain-specific terms. These terms get fragmented into multiple subword tokens, degrading performance. Strategies to address this include vocabulary extension—adding new tokens and resizing the embedding matrix—followed by continued pre-training to learn meaningful embeddings for the new entries before task-specific fine-tuning.

In multilingual models, vocabulary size must be allocated across dozens or hundreds of languages. A fixed total vocabulary budget forces a trade-off in per-language representation. Languages with similar scripts (e.g., Latin-based) share tokens efficiently, while morphologically rich or logographic languages like Japanese or Arabic may be under-represented. SentencePiece with a large vocabulary (e.g., 250k) is a common approach to ensure adequate coverage across diverse scripts.

The most widely used algorithm for constructing a subword vocabulary. BPE starts with characters and iteratively merges the most frequent adjacent token pairs in a training corpus. The number of merge operations is a direct hyperparameter that determines the final vocabulary size. A larger number of merges results in a larger vocabulary with more common words kept intact, while a smaller number forces more fragmentation into subwords.

The frequency at which a tokenizer encounters input sequences absent from its fixed vocabulary. A primary goal of subword tokenization is to drive the OOV rate to zero. A smaller vocabulary size increases the risk of encountering OOV tokens, while a byte-level BPE strategy guarantees a 0% OOV rate by operating on a base vocabulary of 256 bytes, at the cost of longer token sequences for rare scripts.

A key efficiency metric for evaluating a tokenizer, calculated as the number of raw bytes divided by the number of tokens produced. A larger vocabulary size typically yields a higher compression ratio because common words are represented as single tokens. However, this comes at the cost of a larger embedding matrix. The trade-off is central to vocabulary size selection: maximizing compression while minimizing model parameters.

A learned lookup table of shape (vocabulary_size, embedding_dimension) that maps each token ID to a dense vector. The vocabulary size directly determines the number of rows in this matrix, making it a significant contributor to total model parameters. For a model with a 100,000-token vocabulary and 4,096 embedding dimensions, this single matrix contains over 400 million parameters, illustrating the memory cost of a large vocabulary.

A training technique that stochastically samples different tokenizations for the same input string. By applying BPE-Dropout, a model is exposed to multiple valid segmentations of a word, making it robust to tokenization variance. The vocabulary size influences the diversity of possible segmentations: a larger vocabulary offers fewer valid subword splits, while a smaller one creates more opportunities for regularization through varied merges.