Token Manipulation

This technique replaces standard characters with visually similar ones from different Unicode blocks to create adversarial examples that appear normal to humans but are interpreted differently by the model's tokenizer.

• Homoglyphs: Using 'Cyrillic а' (U+0430) instead of Latin 'a' (U+0061).
• Zero-Width Characters: Inserting non-printing characters like U+200B (Zero-Width Space) or U+200C (Zero-Width Non-Joiner) to break tokenization.
• Impact: Can bypass keyword-based safety filters, cause tokenizer errors, and create inputs that are misaligned with the model's training distribution.

This method exploits differences in how various tokenizers (e.g., between a safety filter and the core model) segment text, creating inputs that are parsed inconsistently across system components.

• Multi-Token Splits: Crafting a word like 'kill' as 'k' + 'ill' to avoid a filter that looks for the whole token.
• Subword Ambiguity: Using punctuation or spacing to force a harmful word to be tokenized as benign sub-units.
• Defense Evasion: The safety layer may see a harmless token sequence, while the language model's decoder reconstructs the intended harmful meaning from the subwords.

An advanced, often white-box, attack that directly manipulates the continuous vector representations of tokens to find small perturbations that maximally alter the model's output logits.

• Gradient-Based Search: Using the model's gradients to find token substitutions that increase the probability of a target (e.g., harmful) output.
• Adversarial Suffixes: Automated discovery of token sequences, like the "Artemis" or "AutoDAN" suffixes, that act as universal triggers for compliance.
• Precise Control: Allows for highly optimized attacks that can be human-uninterpretable but are highly effective at steering model behavior.

This involves manipulating the raw byte-level encoding or data serialization format of a prompt before it is processed by the model's tokenizer.

• Base64/URL Encoding: Embedding instructions within encoded strings that a pre-processing step might decode.
• Markdown/HTML Injection: Using formatting syntax like [click here](javascript:alert()) in a context where it might be rendered and re-tokenized.
• Buffer Overflows in Tokenization: Crafting extremely long or malformed strings to cause errors in the tokenization pipeline, potentially leading to undefined behavior.

This technique targets models that process multiple data types by embedding adversarial instructions within non-text modalities, such as images or audio, which are then converted into token sequences.

• Image Steganography: Encoding text prompts within the pixel data of an image using tools like Stable Diffusion with custom embeddings.
• OCR Bypass: Creating images of text that, when read by an Optical Character Recognition (OCR) component, produce a malicious prompt.
• Audio Adversarial Examples: Using inaudible perturbations or specific phonemes in speech-to-text inputs to generate harmful text tokens.

A range of techniques designed to detect and mitigate token manipulation attacks, essential for securing production systems.

• Input Normalization & Canonicalization: Converting all text to a standard form (NFKC Unicode normalization), stripping zero-width characters, and verifying encoding.
• Perplexity Filtering: Flagging inputs with abnormally high perplexity scores, as adversarial examples often lie off the natural language manifold.
• Robust Tokenization: Using consistent, hardened tokenizers across all pipeline stages and employing allowlists/denylists for suspicious Unicode blocks.
• Adversarial Training: Fine-tuning the model on examples of token manipulation to improve its robustness against such attacks.

An adversarial suffix is a string of tokens, often gibberish or optimized characters, appended to a benign user query. Its purpose is to shift the model's internal representations, systematically inducing it to comply with harmful requests it would normally refuse. These suffixes are typically discovered through automated search algorithms that query the model repeatedly to find token sequences that maximize the probability of a harmful output.

• Automated Discovery: Often generated using gradient-based methods or LLM-as-attacker frameworks in a black-box attack setting.
• Universal Properties: Some suffixes demonstrate transferability, acting as universal adversarial prompts that work across multiple queries.
• Defense Challenge: These attacks are difficult to filter as the suffix tokens may appear nonsensical or innocuous in isolation.

Unicode exploits are a subset of token manipulation that uses the complexity of the Unicode standard to create adversarial inputs. Attackers employ visually similar characters (homoglyphs), zero-width spaces, or non-standard encodings to obfuscate prompt content.

• Homoglyph Attacks: Replace standard ASCII characters with visually identical Unicode characters (e.g., Latin 'a' vs. Cyrillic 'а') to bypass keyword-based safety filters.
• Invisible Characters: Insert zero-width joiners or spaces to break tokenization in unexpected ways, potentially causing the model to misinterpret the prompt.
• Normalization Bypass: Exploit differences in how systems normalize Unicode text, creating a mismatch between the filter's parsing and the model's tokenizer.

An embedding space attack operates directly on the continuous vector representations of tokens. Instead of manipulating discrete tokens, this method searches for small perturbations in the embedding space that, when added to a prompt's embeddings, cause a maximal change in the model's output behavior. This is a more fundamental input manipulation technique.

• Gradient-Based: Often requires white-box access to compute gradients of the loss with respect to the input embeddings.

• Automated Jailbreak Generation: Used in automated red teaming to efficiently discover adversarial prompts by optimizing in the continuous space before converting back to tokens.

• Tokenization-Agnostic: Works on the numerical features the model actually processes, bypassing some token-level heuristics.

A black-box attack is performed without any access to the target model's internal architecture, parameters, or gradients. The attacker can only query the model via its API and observe its outputs. Most practical adversarial prompting, including many jailbreak prompt discoveries, falls into this category.

• Query-Only: Relies on iterative trial and error, genetic algorithms, or using a secondary LLM to generate and test attack candidates.

• Real-World Relevance: Reflects the typical access level of an external attacker or red teaming practitioner.

• Includes: Adversarial suffix optimization and systematic boundary testing are common black-box methodologies.

Inference-time attack is the broad category for exploits that occur during a model's deployment and generation phase, as opposed to during its training. Token manipulation, prompt injection, and jailbreak prompts are all inference-time attacks. They exploit vulnerabilities in how the model processes input context at runtime.

• Contrast with Data Poisoning: Does not require corrupting the training set; it targets the deployed system.

• Dynamic: Attacks can be adapted in real-time based on model responses and updated safety patches.

• Core Mechanism: Manipulates the input sequence to cause model evasion or goal hijacking without altering the underlying model weights.

Boundary testing in adversarial prompting is the systematic practice of crafting inputs at the perceived edges of a model's safety guidelines and operational capabilities. The goal is to map the model's failure modes and discover adversarial examples that trigger harmful content generation or other unintended behaviors.

• Methodical Exploration: Involves testing paraphrases, role-playing scenarios, hypotheticals, and edge-case queries to find weaknesses.

• Proactive Security: A key activity in manual and automated red teaming to harden models before deployment.

• Discovers: Subtle triggers, conflicting instructions, and context-based vulnerabilities that simpler attacks might miss.