Special Tokens

A special token prepended to the input sequence in models like BERT. The final hidden state corresponding to this token is used as the aggregate sequence representation for classification tasks. It is designed to capture the entire input's contextual meaning.

• Origin: Introduced in the BERT architecture (Devlin et al., 2018)
• Usage: The output embedding at the [CLS] position is fed into a linear classifier
• Models: BERT, RoBERTa, DistilBERT
• Note: Not used in GPT-style decoder-only models, which rely on the last token's hidden state

A delimiter token used to mark the boundary between two distinct segments of input, such as a question and a passage in question-answering tasks. It signals to the model that the preceding and following text belong to different logical units.

• Primary function: Segment differentiation in sentence-pair tasks
• Secondary function: Marks the end of a single sequence
• Used with: Token type IDs (segment embeddings) to distinguish segment A from segment B
• Example: [CLS] What is BPE? [SEP] Byte-Pair Encoding is... [SEP]

A placeholder token that replaces a percentage of input tokens during pre-training. The model's objective is to predict the original vocabulary ID of the masked token based on bidirectional context. This forces the model to learn deep bidirectional representations.

• Training strategy: Typically 15% of tokens are masked; of those, 80% replaced with [MASK], 10% with random tokens, 10% unchanged
• Inference use: Enables fill-in-the-blank prompting for models like BERT
• Contrast: Not used in autoregressive (left-to-right) models like GPT

A token used to extend shorter sequences to a uniform length within a batch, enabling efficient parallel processing on GPUs. The token ID for [PAD] is typically 0, and its embedding is ignored during computation via the attention mask.

• Attention mask role: A binary tensor sets positions with [PAD] to 0, preventing the model from attending to them
• Loss calculation: Padding positions are excluded from the loss function
• Critical for: Efficient batching of variable-length inputs in both training and inference

A fallback token representing any word or subword unit not present in the model's vocabulary. In modern subword tokenization schemes like BPE and WordPiece, the [UNK] token is rarely triggered because unknown words are decomposed into known subword units.

• Legacy relevance: Critical in word-level vocabularies where OOV rates were high
• Modern mitigation: Byte-level BPE (used in GPT-2 and later) eliminates [UNK] entirely by operating on raw bytes
• Indicator: A high frequency of [UNK] tokens signals a vocabulary mismatch between training data and inference text

Tokens that explicitly mark the beginning of sequence and end of sequence. These are essential for autoregressive decoder models that generate text token-by-token, as [EOS] signals the model to stop generation.

• [BOS]: Prepended to the input; also called <s> in SentencePiece and <|startoftext|> in some GPT variants
• [EOS]: Appended to the target; generation halts when this token is predicted
• Critical for: Preventing infinite loops in text generation and enabling the model to learn sequence termination
• Chat models: Often use <|im_start|> and <|im_end|> as structured variants

A binary tensor generated during tokenization that explicitly instructs the model's self-attention mechanism to ignore padding tokens and focus solely on genuine content.

• 1 indicates a real token to attend to
• 0 indicates a [PAD] token to be ignored
• Prevents padding from corrupting the contextual representation of the sequence
• Essential for efficient batch processing of variable-length inputs

A structured formatting script within a tokenizer that converts a sequence of chat messages into a single tokenized string with the appropriate control tokens for a specific instruction-tuned model.

• Inserts [CLS], [SEP], and role-specific tokens automatically
• Ensures the model receives the correct conversational structure
• Varies significantly between model families (LLaMA vs. Mistral vs. Vicuna)
• Critical for maintaining prompt format fidelity in production

The fixed set of unique tokens that a language model recognizes, mapping each token to a unique integer Token ID. Special tokens are explicitly reserved entries within this vocabulary.

• Built during training via algorithms like BPE or WordPiece
• Special tokens occupy the first indices (e.g., ID 0 for [PAD])
• Vocabulary size is a critical hyperparameter balancing encoding efficiency and embedding parameters
• Typical sizes range from 32k to 256k tokens

The strategy of adding a special [PAD] token to shorter sequences in a batch to ensure all input tensors have a uniform length for efficient parallel processing on GPUs.

• Enables dynamic batching without sacrificing throughput
• Padding can be applied to the left or right side depending on the architecture
• Decoder-only models typically use left-padding for autoregressive generation
• Always paired with an attention mask to nullify padded positions

The process of converting a raw text string into a sequence of Token IDs using a tokenizer's vocabulary and merge rules. Special tokens are injected during this step to provide structural signals.

• Input: "Hello world"
• Output: [101, 7592, 2088, 102] (with [CLS] and [SEP])
• Includes normalization, pre-tokenization, and subword splitting
• The resulting integer sequence feeds directly into the model's embedding layer

A strategy that segments words into smaller, meaning-bearing units like morphemes or frequent character sequences. This approach fundamentally reduces the Out-of-Vocabulary (OOV) problem that plagued early word-level tokenizers.

• Algorithms include BPE, WordPiece, and Unigram
• Enables models to represent rare and unseen words by composing known subwords
• Special tokens like [UNK] are rarely needed with byte-level variants
• Balances vocabulary size with encoding efficiency