WordPiece

Unlike BPE which merges based on raw frequency, WordPiece selects the pair of symbols whose merge maximizes the training data likelihood. It computes a score for each potential merge: score = count(pair) / (count(first) * count(second)). The pair with the highest score is merged, prioritizing combinations where the individual symbols are rare but their co-occurrence is frequent. This statistically-driven approach produces a vocabulary that better captures meaningful linguistic units.

WordPiece was developed specifically for BERT (Bidirectional Encoder Representations from Transformers) and remains the default tokenizer for BERT-family models. The original BERT vocabulary consists of approximately 30,000 tokens. A distinctive feature is the use of the ## prefix to denote subword continuations—for example, 'playing' might be tokenized as ['play', '##ing']. This prefix convention allows the model to distinguish between word-initial and word-internal occurrences of the same character sequence.

During tokenization, WordPiece applies a greedy longest-match-first algorithm. Starting from the beginning of a word, it iteratively finds the longest substring present in the vocabulary. If no match is found for the entire remaining substring, it falls back to unknown token handling. This deterministic encoding ensures that every input text is segmented consistently, producing the same token sequence for identical inputs across different runs.

WordPiece vocabularies reserve several special tokens with fixed semantic functions:

• [CLS]: Prepended to every sequence; its final hidden state is used for classification tasks
• [SEP]: Separates sentence pairs and marks sequence boundaries
• [MASK]: Used during pre-training for masked language modeling
• [PAD]: Ensures uniform sequence length within batches
• [UNK]: Represents characters or sequences not found in the vocabulary

The quality of a WordPiece vocabulary is directly tied to the representativeness of the training corpus. The algorithm learns merge rules that reflect the statistical properties of the text it was trained on. A vocabulary trained on general-domain text will perform poorly on highly specialized domains like biomedical literature or legal documents. This necessitates domain-specific vocabulary training for optimal performance in specialized applications.

While both are subword algorithms, key differences exist:

• Merge Criterion: BPE uses raw frequency; WordPiece uses a likelihood ratio
• Encoding: BPE applies merge rules sequentially; WordPiece uses longest-match-first
• Continuation Markers: WordPiece uses ## prefixes; BPE typically uses @@ or </w> suffixes
• Vocabulary Construction: BPE builds bottom-up from characters; WordPiece starts with all characters and iteratively merges WordPiece generally produces more linguistically motivated segmentations.

A critical hyperparameter defining the total number of tokens a model recognizes. WordPiece vocabularies for BERT typically range from 30,000 to 120,000 tokens, balancing encoding efficiency against model embedding parameters.

• Trade-off: Larger vocabularies reduce sequence length but increase embedding matrix size
• BERT-base: 30,522 tokens
• Impact: Directly affects memory footprint and inference speed

Reserved vocabulary entries with specific control functions essential to WordPiece-based models like BERT. These include [CLS] for sequence classification, [SEP] for sentence separation, and [MASK] for masked language modeling.

• [CLS]: Aggregates sequence-level representation
• [SEP]: Delimits sentence boundaries in input pairs
• [MASK]: Placeholder for masked token prediction during pre-training
• [UNK]: Represents unknown tokens (rarely used with subword tokenization)

A technique that stochastically samples different token segmentations during training to improve model robustness. While WordPiece uses a deterministic longest-match-first segmentation, applying BPE-Dropout or Unigram sampling can expose models to tokenization variance.

• Purpose: Reduces sensitivity to segmentation ambiguity
• Method: Randomly drops merges during training
• Benefit: Improved generalization to rare and misspelled words