Compression Ratio

The compression ratio is calculated as the number of raw bytes in the original text divided by the number of tokens produced by the tokenizer.

• Formula: Compression Ratio = Raw Bytes / Token Count
• Interpretation: A ratio of 4.0 means each token represents 4 bytes of text on average.
• Benchmarking: This metric allows for direct, apples-to-apples comparisons between different tokenization algorithms like Byte-Pair Encoding (BPE) and Unigram.

A higher compression ratio directly reduces operational costs and latency in large language models.

• Cost Efficiency: Most LLM APIs charge per token. A tokenizer that produces 20% fewer tokens for the same prompt reduces API costs by 20%.
• Latency Reduction: Fewer tokens mean fewer autoregressive generation steps, decreasing time-to-first-token and total inference time.
• Context Window Maximization: Efficient tokenization allows more raw text to fit within a model's fixed context window, improving performance on long-document tasks.

Compression ratios are highly sensitive to the language and script of the input text.

• English vs. Multilingual: Tokenizers trained predominantly on English corpora often show high compression ratios for English but degrade significantly for languages with non-Latin scripts or complex morphology.
• The Cyrillic Penalty: A Russian sentence can require up to 2-3x more tokens than its semantic equivalent in English, inflating costs for non-English users.
• Mitigation: Training on balanced multilingual corpora and using byte-level fallbacks helps normalize compression ratios across languages.

The compression ratio is directly governed by the vocabulary size hyperparameter, presenting a critical trade-off.

• Larger Vocabulary: A 100k-token vocabulary can encode common words and phrases as single tokens, achieving a high compression ratio.
• The Trade-off: Increasing vocabulary size expands the model's embedding matrix, consuming more GPU memory and compute for the embedding lookup.
• The Sweet Spot: Most modern LLMs settle on vocabulary sizes between 32k and 100k tokens to balance encoding efficiency with model parameter efficiency.

Evaluating compression ratio requires standardized test corpora to ensure meaningful comparisons.

• Common Benchmarks: Use datasets like Flores-200 for multilingual evaluation or domain-specific corpora (code, medical text) for specialized applications.
• Tokenizer Comparison: A BPE tokenizer might achieve a ratio of 4.5 on English news text, while a Unigram model might score 4.3, but the Unigram model could be more robust to noise.
• The tiktoken Advantage: OpenAI's tiktoken library is specifically optimized for high compression ratios on common code and English text patterns.

Certain data types expose the limits of subword tokenization, causing the compression ratio to plummet.

• Arbitrary Sequences: Random strings of characters or base64-encoded data force the tokenizer to fall back to single-byte tokens, collapsing the ratio to near 1.0.
• Repetitive Whitespace: Leading spaces in code can each become a separate token, wasting context window space.
• Numerical Data: A long integer like 847291038475 is often shattered into multiple digit tokens, making arithmetic reasoning difficult for the model.