Hugging Face Tokenizers

The foundational algorithm behind many Hugging Face tokenizers. BPE starts with a base vocabulary of characters and iteratively merges the most frequent adjacent pairs of tokens to build a subword vocabulary.

• Training: Learns merge rules from a corpus based on co-occurrence frequency.
• Greedy Encoding: Applies learned merges in order to segment new text.
• Byte-level BPE: A variant operating on raw bytes (UTF-8), guaranteeing a base vocabulary of 256 tokens and eliminating unknown tokens.
• Example: The word 'lower' might be segmented as 'low' + 'er' if that pair is frequent.

A subword tokenization algorithm used by BERT and DistilBERT, available in the Hugging Face Tokenizers library. Unlike BPE, WordPiece selects merge rules based on maximizing the likelihood of the training data.

• Selection Criterion: Chooses the pair that maximizes the training data likelihood, not raw frequency.
• Token Prefix: Uses ## to mark tokens that are continuations of a previous word (e.g., 'playing' -> 'play' + '##ing').
• OOV Handling: Decomposes unknown words into known subword units.

A probabilistic tokenization method that starts with a large initial vocabulary and iteratively prunes tokens that least increase the overall likelihood of the corpus.

• Bottom-Up Pruning: Unlike BPE's bottom-up merging, Unigram removes the least useful tokens.
• Probabilistic Encoding: At inference, it can sample multiple segmentations, enabling subword regularization.
• Lossless: Often used with SentencePiece for full reversibility.
• Example: The word 'university' might have multiple valid segmentations like 'uni' + 'versity' or 'university'.

A language-independent tokenization framework that treats input text as a raw Unicode sequence, eliminating the need for language-specific pre-tokenization.

• Lossless Decoding: Whitespace is encoded as a special meta-symbol (▁), allowing perfect reconstruction of the original text.
• Algorithm Agnostic: Can use BPE or Unigram as its underlying model.
• Direct Raw Text: No pre-tokenization step required, making it ideal for multilingual models like T5 and Llama.
• Example: 'Hello world' becomes ['▁Hello', '▁world'].

Critical pre-processing stages in the tokenization pipeline that standardize raw text before subword segmentation.

• Normalization: Applies transformations like Unicode normalization (NFKC), lowercasing, and whitespace stripping to ensure consistency.
• Pre-tokenization: Splits text into coarse units (words) using rules like whitespace and punctuation splitting.
• Pipeline Order: Normalization → Pre-tokenization → Subword Model → Post-processing.
• Impact: Inconsistent normalization leads to vocabulary mismatches and degraded model performance.

A structured formatting script within a tokenizer that converts a sequence of chat messages into a single tokenized string with appropriate control tokens.

• Role Formatting: Wraps user, assistant, and system messages with model-specific special tokens like <|im_start|>user.
• Generation Prompt: Automatically appends the assistant prefix to prime the model for response generation.
• Model-Specific: Each instruction-tuned model (Llama 3, Mistral, Gemma) requires its own template.
• Usage: tokenizer.apply_chat_template(messages, tokenize=True)