Token Smuggling

Attackers leverage the difference between Unicode normalization forms (NFD, NFC, NFKC) to craft payloads that look identical to humans but have different byte representations. For example, the Latin 'A' (U+0041) and Cyrillic 'А' (U+0410) are visually indistinguishable but produce entirely different token IDs. When safety filters scan the raw text, they see harmless characters, but the model's tokenizer may normalize them into a malicious sequence. Normalization Form Canonical Decomposition (NFD) breaks precomposed characters into base characters and combining marks, creating additional smuggling surfaces. Attackers chain multiple normalization passes to create payloads that only become dangerous after the tokenizer's internal preprocessing.

Invisible Unicode characters are inserted between tokens to break pattern-matching filters without altering visual appearance. Key characters include:

• Zero-Width Space (U+200B): Splits keywords so 'malware' becomes 'mal​ware', evading string matching while the tokenizer may still reconstruct the original term.
• Zero-Width Joiner (U+200D): Forces adjacent characters to render as a joined form, potentially merging split tokens.
• Zero-Width Non-Joiner (U+200C): Prevents ligature formation, causing the tokenizer to fragment words differently than expected.
• Word Joiner (U+2060): Prevents line breaks but also interferes with token boundary detection. These characters are invisible in most text editors and chat interfaces, making manual detection nearly impossible.

This technique exploits mismatches between the safety filter's tokenizer and the model's tokenizer. By splitting a forbidden word across multiple tokens that individually appear benign, the attacker bypasses blocklists. For instance, if a filter blocks the token for 'explosive', the attacker sends 'explo' and 'sive' as separate tokens. The model's autoregressive generation then reconstructs the full concept from context. Subword tokenization algorithms like Byte-Pair Encoding (BPE) and WordPiece create predictable split points that attackers can reverse-engineer. Advanced variants use multi-lingual tokenization where a harmful English word is smuggled inside tokens from a low-resource language that the safety filter doesn't monitor.

Attackers replace ASCII characters with visually identical Unicode homoglyphs to evade keyword filters. The word 'hack' written with Cyrillic 'а' (U+0430) instead of Latin 'a' (U+0061) passes string-matching defenses but may be normalized to the intended meaning by the model's internal representations. Internationalized Domain Name (IDN) homograph attacks pioneered this technique for phishing, and it now extends to LLM prompt injection. Sophisticated smuggling chains combine homoglyphs with bidirectional text control characters like Right-to-Left Override (U+202E) to reverse the visual order of characters, making the rendered text appear completely different from the underlying byte sequence that reaches the tokenizer.

Advanced smuggling chains nest multiple encoding layers to defeat recursive sanitization. A payload may be:

1. Base64-encoded to hide keywords
2. Wrapped in Unicode escape sequences (\uXXXX)
3. Interspersed with zero-width characters to break pattern matching
4. Split across tokens using BPE boundary prediction When the model processes this, each layer is decoded sequentially, reconstructing the malicious instruction only at inference time. Recursive smuggling specifically targets systems that apply multiple sanitization passes by designing payloads where each decoding step reveals a new layer of obfuscation, exhausting the filter's recursion depth limits. This technique is particularly effective against defense-in-depth architectures that assume single-pass sanitization is sufficient.

Unicode bidirectional algorithm characters are weaponized to create visual deception in source code and prompts. Right-to-Left Override (RLO, U+202E) forces subsequent text to render right-to-left, reversing the visual order of characters. Left-to-Right Override (LRO, U+202D) restores left-to-right rendering. Pop Directional Formatting (PDF, U+202C) terminates the override. By strategically placing these control characters, attackers make a malicious payload appear as a harmless comment or variable name in code review tools while the compiler or interpreter processes the true character order. In LLM contexts, this causes the human reviewer and the automated safety filter to see different semantic content than what the model's tokenizer actually receives.