Adversarial Example

An adversarial example is defined by a small, intentional modification to a standard input. In computer vision, this is often an imperceptible change to pixel values. For language models, this manifests as subtle alterations to prompt wording, structure, or encoding. The core characteristic is that the perturbation is purposefully designed, not random, to exploit specific model vulnerabilities. For instance, rephrasing a harmful request using benign synonyms or inserting special delimiter characters can constitute an adversarial perturbation in text.

The primary objective is to induce a high-confidence error. The model does not merely become uncertain; it is fooled into providing an incorrect or harmful output with high-assurance. This differentiates adversarial examples from general model mistakes. Key aspects include:

• Misclassification: The model assigns the wrong label with high probability (e.g., classifying a panda as a gibbon).
• Policy Violation: For LLMs, the model complies with a request it is safety-aligned to refuse, often without typical refusal messages.
• The error persists despite the input being semantically identical or nearly identical to a non-adversarial version from a human perspective.

A critical and concerning property is transferability. An adversarial example crafted for one model often causes errors in other models, even those with different architectures or trained on different datasets. This occurs because many models learn similar decision boundaries in high-dimensional space. In practice, this means:

• A jailbreak prompt effective on one LLM (e.g., GPT-4) may also work on another (e.g., Claude).
• Transferability enables black-box attacks, where an attacker can use a surrogate model to develop adversarial prompts for a target model whose internals are unknown.
• This property underscores the systemic nature of the vulnerability.

Adversarial examples exploit a model's reliance on non-robust features. These are patterns in the data that are highly predictive during standard training but are not semantically meaningful to humans and are easily manipulated.

• In an image, these could be specific high-frequency textures.
• In text, these might be rare token combinations, syntactic structures, or embedding-space correlations that the model has learned to associate with certain outputs.
• The adversarial perturbation strategically amplifies or injects these non-robust features to steer the model's prediction, while leaving the human-interpretable robust features (the core semantic content) largely unchanged.

In deep neural networks, adversarial examples often exploit the piecewise linear nature of modern activation functions (like ReLU). In high-dimensional spaces, models can be surprisingly linear in the vicinity of data points. A small step in the direction of the gradient of the loss function with respect to the input can cause a large change in the output. This is formalized in attacks like the Fast Gradient Sign Method (FGSM). For LLMs, while the discrete nature of tokens complicates gradient-based attacks, the principle translates to searching for token sequences that create a similar linear "push" in the model's internal representations toward a desired (and often harmful) output.

The existence of adversarial examples highlights a fundamental disconnect between human and model perception. Humans and models process information differently.

• Humans perceive holistically and semantically.
• Models process inputs as points in a high-dimensional geometric space (like embedding space) and make decisions based on statistical correlations.
• An adversarial perturbation is significant in the model's geometric space but insignificant in human perceptual space. This mismatch is the root cause of the phenomenon and a major challenge for building robust and aligned AI. Defenses often aim to force models to base decisions on features that align more closely with human semantic understanding.

Prompt injection is an attack where malicious user input overrides or subverts the original system instructions. This can lead to goal hijacking, where the model performs a different task than intended.

• Direct Injection: Malicious instructions are placed directly in the user query.
• Indirect Injection: Instructions are hidden in external data (e.g., a retrieved webpage) that the model processes.
• Impact: Can lead to data exfiltration, system prompt leaks, or unauthorized actions.

An in-context attack manipulates the few-shot examples within a prompt's context window to adversarially steer the model's response. This corrupts the in-context learning process.

• Chain-of-Thought Poisoning: Injecting malicious or false reasoning steps into demonstration examples.
• Impact: Causes the model to adopt incorrect premises or harmful reasoning patterns for the target task.
• Challenge: Difficult to detect as the attack is embedded within seemingly legitimate demonstration data.

These attacks exploit the technical parsing of prompts and input encoding to break instruction boundaries.

• Delimiter Attacks: Using or manipulating special characters (e.g., ###, """) that separate system/user/assistant messages to inject instructions.
• Unicode Exploits: Using homoglyphs, zero-width spaces, or other non-standard Unicode to obfuscate text and bypass simple keyword filters.
• Token Manipulation: Altering input tokens at the embedding level to cause model misinterpretation.

Advanced techniques that target complex system architectures.

• Multi-Modal Injection: Embedding adversarial instructions within non-text inputs (images, audio) to influence a multimodal model's text generation.
• Recursive Injection: An initial injected instruction forces the model to generate further malicious prompts, creating a self-propagating chain.
• RAG Jailbreak: Poisoning a retrieval-augmented generation system's knowledge base so that retrieved documents contain injectable instructions.

An adversarial suffix is a string of tokens, often nonsensical to humans, that is algorithmically optimized and appended to a user query to systematically induce a language model to comply with harmful requests. Unlike handcrafted jailbreaks, these suffixes are typically discovered through automated gradient-based search or black-box optimization techniques against the model's API. Their effectiveness demonstrates the non-linear and brittle nature of model decision boundaries in high-dimensional token space.

A universal adversarial prompt (UAP) is a single, fixed input string that, when prepended or appended to a wide variety of benign user queries, reliably causes a model to generate harmful, biased, or otherwise policy-violating outputs. The existence of UAPs highlights a critical security vulnerability: a generalized failure mode that bypasses safety training across multiple domains. Research into UAPs focuses on the transferability of attacks across different models and the robustness of safety fine-tuning methods.

An embedding space attack operates on the continuous vector representations (embeddings) of tokens rather than discrete tokens themselves. By perturbing these vectors along calculated gradients, an attacker can find small changes that cause maximal divergence in the model's output. This technique is often used in white-box attack scenarios to automate the discovery of adversarial examples. For text, the challenge is projecting the perturbed embedding back into a sequence of valid, natural-looking tokens that retain the adversarial property.

An inference-time attack is any adversarial technique executed during the model's deployment and generation phase, as opposed to during its training. Prompt injection, jailbreaking, and the use of adversarial suffixes are all inference-time attacks. They exploit vulnerabilities in how the model processes concatenated instructions, parses context, or applies its safety filters at runtime. This contrasts with training-time attacks like data poisoning, which corrupt the model before deployment.

Model evasion is the overarching objective of causing a machine learning model to produce an incorrect or undesired output while avoiding detection by safety or monitoring systems. In the context of LLMs, this includes:

• Generating harmful content without triggering a refusal response.
• Bypassing content moderation filters through obfuscation.
• Causing goal hijacking without alerting the application layer. Evasion techniques test the model's alignment robustness and the effectiveness of its harm detection classifiers.

Boundary testing is a systematic adversarial prompting methodology that involves crafting inputs at the edges of a model's known capabilities, safety guidelines, and instruction formats. The goal is not just to cause a failure, but to map the failure surface and discover unexpected behaviors. This includes:

• Testing semantic boundaries of safety policies.
• Probing the limits of context window comprehension.
• Exploring how few-shot example formatting affects task adherence. It is a core practice in manual red teaming and automated robustness evaluation.