WordPiece Tokenization

WordPiece builds its vocabulary from the bottom up, starting with individual characters and iteratively merging the most frequent adjacent symbol pairs. This greedy algorithm ensures that the final vocabulary is composed of subword units that optimally represent the training corpus. For clinical text, this means common prefixes like 'hyper-' and suffixes like '-ectomy' become distinct, reusable tokens, allowing the model to decompose rare or unseen terms like 'hypercalcemia' into the known fragments 'hyper', 'cal', '##cemia'.

A primary failure mode of word-level tokenizers is the 'unknown' token for rare medical jargon. WordPiece solves this by design. Any word can be reconstructed from its constituent subwords. A novel term like 'pneumonoultramicroscopicsilicovolcanoconiosis' is not a single out-of-vocabulary token but a sequence of known fragments: 'pneumono', 'ultra', 'microscopic', 'silico', 'volcano', 'coniosis'. This guarantees zero out-of-vocabulary words, a critical requirement for processing clinical narratives with extensive morphological variants.

WordPiece uses a unique continuation marker (typically '##') to distinguish subword tokens that begin a word from those that continue it. For example, the word 'gastroenteritis' might be tokenized as 'gastro', '##enter', '##itis'. This prefix is crucial for the model's ability to reconstruct original word boundaries during decoding and provides a strong signal for the self-attention mechanism to learn that '##enter' is morphologically linked to the preceding token 'gastro', not a standalone word.

The effectiveness of WordPiece is directly tied to its training data. A vocabulary trained on general English will fail to segment medical terms efficiently. For clinical NLP, the tokenizer must be trained on a large corpus of biomedical literature (PubMed) and clinical notes (MIMIC-III). This ensures the vocabulary prioritizes high-utility medical subwords:

• Drug suffixes: '##mab', '##tinib', '##lol'
• Anatomical roots: 'hepat', 'nephr', 'cardi'
• Procedural terms: '##ectomy', '##plasty', '##scopy'

WordPiece offers a direct trade-off between vocabulary size and sequence length. A typical clinical WordPiece vocabulary contains 30,000 to 110,000 tokens. A larger vocabulary captures more complete medical terms as single tokens, reducing sequence length and computational cost. A smaller vocabulary forces more fragmentation, increasing sequence length but providing finer-grained morphological analysis. This hyperparameter is tuned to balance the 512-token input limit of models like BERT against the need to represent complex clinical concepts efficiently.

While both are subword algorithms, WordPiece differs from BPE in its selection criterion. BPE merges the most frequent pair of symbols. WordPiece merges the pair that maximizes the likelihood of the training data, calculated as the frequency of the pair divided by the product of the frequencies of its individual parts. This probabilistic approach tends to select morphemes that are more linguistically meaningful, potentially yielding better segmentations for complex medical terminology than purely frequency-based BPE.