Token ID

Each unique token in a model's vocabulary is assigned a distinct integer, typically ranging from 0 to the vocabulary size minus one. This mapping is deterministic and bidirectional: the tokenizer's encoding process converts text to a sequence of IDs, while decoding reverses the operation. For example, in GPT-2, the token 'the' maps to ID 464, while 'The' maps to a different ID due to case sensitivity.

• IDs serve as lookup indices into the token embedding matrix
• Special tokens like [CLS], [SEP], and [PAD] occupy reserved low-index positions
• The mapping is frozen after training; new tokens require vocabulary extension

Token IDs function as the direct input to a model's embedding layer, a learned matrix of shape (vocabulary_size, embedding_dimension). During a forward pass, each ID performs a row-wise lookup to retrieve its corresponding dense vector. This operation is mathematically equivalent to one-hot encoding followed by a matrix multiplication.

• The embedding matrix is randomly initialized and learned during training
• IDs with similar semantic roles often develop similar embedding vectors
• Subword token IDs allow compositional representation of rare words

The range of possible token IDs is defined by the vocabulary size hyperparameter, which directly impacts model architecture. A larger vocabulary (e.g., 50,000+ tokens) reduces sequence length but increases the embedding matrix parameter count. A smaller vocabulary produces longer sequences but fewer parameters.

• GPT-2 uses a vocabulary of 50,257 tokens
• BERT's WordPiece vocabulary contains 30,522 tokens
• Byte-level BPE guarantees exactly 256 base byte tokens plus merges

Specific integer ranges are reserved for special tokens that control model behavior rather than representing linguistic content. These IDs signal structural boundaries and task-specific operations to the model's attention mechanism.

• [PAD] (ID 0): Ensures uniform batch dimensions, masked by the attention mask
• [CLS] (ID 101 in BERT): Aggregates sequence-level representation for classification
• [SEP] (ID 102 in BERT): Delimits sentence boundaries in pair tasks
• [MASK] (ID 103 in BERT): Indicates tokens to predict during masked language modeling
• [UNK] (ID 100 in BERT): Represents out-of-vocabulary tokens when subword fallback fails

Token IDs directly determine the computational cost of processing text. A model's maximum context window is defined in tokens, not characters or words. Efficient tokenization minimizes ID count per semantic unit, measured by the compression ratio (raw bytes ÷ token count).

• GPT-4 Turbo supports 128,000 token IDs per sequence
• A higher compression ratio means fewer IDs encode more information
• Subword tokenization typically achieves 2-4x compression over character-level encoding

The fidelity of the ID-to-text round-trip depends on the tokenizer architecture. Lossless tokenization, as implemented by SentencePiece, guarantees that every sequence of token IDs can be perfectly decoded back to the original string. Lossy tokenizers may introduce irreversible transformations during normalization.

• SentencePiece treats input as raw Unicode, preserving whitespace and casing
• Standard BPE tokenizers may lose information through lowercasing or Unicode normalization
• Lossless encoding is critical for code generation and structured data tasks

The fixed set of unique tokens that a language model recognizes. Each entry in the vocabulary is assigned a unique Token ID, creating a bijective mapping between a human-readable string and its integer representation. The vocabulary size is a critical hyperparameter—GPT-3 uses ~50,000 tokens, while Llama 2 uses 32,000.

• Base vocabulary includes single characters and common subwords
• Extended vocabulary grows through merge rules during training
• Special tokens like [PAD] and [UNK] occupy reserved IDs

The process of converting a raw text string into a sequence of Token IDs using a tokenizer's vocabulary and merge rules. This is the critical bridge between human language and model-compatible numerical input.

• Input: "Hello, world!"
• Output: [15496, 11, 995, 0]
• Each integer maps directly to an embedding vector in the model's first layer
• Encoding is deterministic for a given tokenizer and vocabulary

A learned, dense vector representation associated with each Token ID in a model's vocabulary. The embedding layer acts as a lookup table: given a Token ID, it returns the corresponding vector that captures the token's semantic and syntactic properties.

• Dimensionality ranges from 768 (BERT-base) to 12,288 (GPT-3)
• Embeddings are initialized randomly and refined during training
• Similar tokens (e.g., 'king' and 'queen') develop proximally close vectors

The inverse process of converting a sequence of Token IDs back into human-readable text. In subword tokenizers like SentencePiece, decoding is lossless—the original input can be perfectly reconstructed from the token sequence.

• Handles merging of subword fragments (e.g., ['token', 'ization'] → 'tokenization')
• Manages whitespace restoration using special prefix markers
• Critical for text generation when model outputs must be rendered as readable strings

Reserved vocabulary entries with specific control functions that occupy fixed Token IDs. These tokens are never split by the tokenizer and serve as structural signals to the model.

• [CLS] (ID 101 in BERT): Aggregates sequence-level representation
• [SEP] (ID 102): Marks sentence boundaries
• [MASK] (ID 103): Indicates tokens to predict in masked language modeling
• [PAD] (ID 0): Fills shorter sequences for batch uniformity
• [UNK] (ID 100): Represents out-of-vocabulary tokens

A binary tensor generated during tokenization that instructs the model's self-attention mechanism to ignore padding tokens. It is constructed directly from Token ID sequences.

• 1 indicates a genuine content token to attend to
• 0 indicates a [PAD] token to be ignored
• Prevents the model from learning spurious correlations with padding
• Essential for efficient batch processing of variable-length sequences