Prompt Normalization

Converts all text to a single, standard Unicode normalization form—typically NFKC (Normalization Form KC) —to collapse visually identical but technically distinct characters.

• Maps full-width Latin characters (e.g., Ａ U+FF21) to standard ASCII (A U+0041)
• Decomposes and recomposes ligatures and combined glyphs
• Eliminates ambiguity between compatibility equivalents

This directly defeats homoglyph attacks where pаy (with Cyrillic а) masquerades as pay.

Removes or escapes invisible Unicode control characters that attackers embed to manipulate tokenization boundaries or inject hidden instructions.

• Strips zero-width characters: U+200B (Zero-Width Space), U+200C (ZWNJ), U+200D (ZWJ)
• Removes bidirectional text override characters: U+202E (RLO), U+202A (LRE)
• Filters deprecated control codes like U+0085 (Next Line)

These characters can break delimiter injection defenses by altering how the model parses prompt boundaries.

Ensures that normalization does not inadvertently create or destroy token boundaries that alter the semantic meaning of the input.

• Preserves whitespace semantics critical for code and structured data
• Validates that canonicalization does not merge adjacent tokens into new, unintended words
• Applies normalization before tokenization to maintain a consistent pipeline

This prevents token smuggling attacks where normalization itself could be exploited to reassemble split payloads.

Identifies and rejects overlong UTF-8 encodings —byte sequences that represent a valid character using more bytes than necessary.

• Detects 2-byte encodings of ASCII characters (e.g., C0 AF for /)
• Detects 3-byte and 4-byte overlong sequences for code points that fit in fewer bytes
• Blocks these before they reach the model's tokenizer

Overlong encodings are a classic bypass technique against input sanitization filters that only check canonical byte representations.

Standardizes all whitespace characters to a single canonical form to prevent attackers from using alternative space characters to evade pattern-matching defenses.

• Converts non-breaking spaces (U+00A0), en spaces, and em spaces to standard spaces
• Collapses repeated whitespace to a single space where semantically safe
• Normalizes line endings (CRLF, CR, LF) to a single newline character

This closes gaps in perplexity filtering where unusual whitespace could lower the statistical anomaly score of an obfuscated payload.

Serves as the first stage in a defense-in-depth pipeline, operating before semantic analysis or guard model evaluation.

• Normalized output feeds into semantic filtering and guard model checks
• Reduces the attack surface for downstream defenses by eliminating encoding-based obfuscation
• Logs pre-normalization and post-normalization inputs for forensic analysis of blocked attacks

Without normalization, a prompt firewall may inspect a benign canonical form while the model processes a malicious obfuscated original.

A visual spoofing technique that replaces characters in a malicious command with visually identical Unicode equivalents to bypass text-based content filters. Prompt normalization directly neutralizes this threat by converting text to a canonical form (e.g., NFC or NFKC) before processing.

• Example: Replacing Latin 'a' (U+0061) with Cyrillic 'а' (U+0430)
• Defense: Unicode canonicalization maps confusable characters to their standard forms
• Impact: Without normalization, a model may interpret pаyment as a distinct, malicious command

An adversarial technique that exploits tokenization discrepancies to hide malicious commands within seemingly harmless text strings. Attackers craft inputs that tokenize differently than expected, embedding instructions that bypass input filters but are reconstructed by the model. Prompt normalization complements token-level defenses by removing the invisible control characters and zero-width spaces often used to manipulate token boundaries.

• Mechanism: Inserting zero-width joiners to split tokens unnaturally
• Defense: Combining normalization with perplexity filtering and token-level anomaly detection

A statistical defense that flags inputs with abnormally high perplexity scores, as obfuscated or encoded payloads often deviate from natural language distributions. While prompt normalization handles structural obfuscation (Unicode tricks, invisible characters), perplexity filtering catches the semantic unnaturalness of concatenated tokens, Base64-encoded strings, and adversarial suffixes.

• Threshold-based: Inputs exceeding a perplexity threshold are blocked
• Complementary: Works alongside normalization to catch what structural cleaning misses
• Limitation: Sophisticated attackers can craft low-perplexity adversarial prompts

A detection method that uses embedding models to understand the underlying intent of a prompt, blocking inputs that are semantically adversarial regardless of phrasing or obfuscation. Unlike prompt normalization, which operates at the character and encoding level, semantic filtering evaluates the meaning of the entire input.

• Embedding comparison: Measures cosine similarity against known attack patterns
• Intent classification: Uses a fine-tuned classifier to detect jailbreak and injection attempts
• Layered defense: Normalization cleans the text first, then semantic filtering evaluates intent

The defensive practice of reinforcing system prompts with explicit boundaries and fallback logic to resist adversarial manipulation. While prompt normalization secures the input pipeline, prompt hardening fortifies the instructions themselves against extraction and override attempts.

• Delimiter defense: Uses unique, non-guessable separators to mark untrusted input boundaries
• Instruction reinforcement: Repeats critical constraints at multiple points in the context window
• Fallback logic: Defines explicit refusal responses for out-of-scope requests