Model Evasion

This is an automated, gradient-based search technique to find a string of tokens (a suffix) that, when appended to a wide range of user queries, systematically bypasses a model's safety filters. Unlike manual jailbreaks, it uses the model's own loss function to iteratively craft a universal adversarial prompt. For example, appending a specific, seemingly nonsensical suffix can cause a model to comply with harmful requests it would normally refuse.

• Automated Discovery: Uses algorithms to search the token space.
• Universal Triggers: A single suffix can work across many query types.
• Black-Box Variants: Can be performed using only API access to query the model.

This vector involves crafting inputs to probe the model's operational boundaries and extract its foundational instructions. A system prompt leak attack tricks the model into outputting its own configuration, revealing proprietary rules or sensitive constraints. Boundary testing systematically crafts queries at the edges of known safety policies to discover failure modes, such as asking the model to role-play scenarios that gradually erode its refusal mechanisms.

• Information Disclosure: Reveals hidden system prompts and safety configurations.
• Failure Mode Discovery: Identifies specific phrasings or contexts where safeguards break down.
• Precursor to Hijacking: Leaked instructions can inform more targeted evasion attempts.

These are multi-stage attacks that poison the model's context indirectly. Indirect prompt injection embeds malicious instructions within data from an external source (e.g., a poisoned webpage or database record retrieved by a RAG system). The model then processes these instructions as part of its context, subverting its goal.

Recursive injection is a more advanced form where the initial injected payload forces the model to generate further adversarial prompts, creating a self-propagating chain of malicious behavior within a single session.

• Exploits Trusted Data: Attacks the model through its retrieved knowledge.
• RAG Jailbreak: A primary attack surface for Retrieval-Augmented Generation systems.
• Self-Replicating: Can amplify and persist an attack within a context window.

This technique corrupts the model's reasoning by manipulating the examples within its context window. In an in-context attack, the few-shot demonstrations are replaced with adversarial examples that steer the model toward incorrect or harmful outputs for a subsequent task.

Chain-of-thought poisoning specifically targets reasoning models by injecting false logical steps or premises into the example reasoning traces. This teaches the model to follow a corrupted reasoning process, leading to confidently held but erroneous conclusions.

• Corrupts Learning from Demonstrations: Manipulates the model's in-context learning.
• Targets Reasoning Integrity: Undermines step-by-step rationale generation.
• Difficult to Detect: The poisoned examples may appear benign in isolation.

These attacks exploit the technical processing layers of model input. Multi-modal injection targets models that process images, audio, or video by embedding adversarial instructions within non-text data (e.g., hidden text in an image) to influence the text generation.

Token manipulation operates at the text encoding level, using techniques like:

• Homoglyphs: Visually similar characters from different Unicode blocks.

• Zero-Width Joiner Characters: Invisible characters that can break tokenization.

• Encoding Tricks: Alternative text representations that bypass simple keyword filters.

• Bypasses Text-Only Filters: Exploits the full input pipeline.

• Obfuscation: Makes malicious intent difficult for automated scanners to parse.

These are high-level attacks that subvert the model's core objective. Goal hijacking is a successful prompt injection that completely redirects the model's task. For instance, a customer service chatbot is tricked into generating phishing emails instead of answering product questions.

Tool misuse occurs when a model with function-calling capabilities is manipulated to exploit external tools. An adversary might craft a prompt that causes the model to call a database deletion API, send unauthorized emails, or exfiltrate data through a search function.

• Objective Subversion: Changes the fundamental task of the AI agent.
• Real-World Impact: Leverages the model's ability to act on external systems.
• Privilege Escalation: Uses the model's access credentials to perform unauthorized actions.

A jailbreak prompt is a specific, crafted input designed to bypass a language model's built-in safety filters and content moderation policies. The goal is to elicit responses—such as generating harmful content or revealing restricted information—that the model is explicitly designed to refuse under normal operation.

• Objective: Circumvent refusal mechanisms.
• Method: Often uses creative scenarios, role-playing, or encoded instructions.
• Example: Asking the model to respond as a fictional character without ethical constraints.

Prompt injection is an adversarial attack where malicious user input is designed to override, subvert, or ignore a language model's original system instructions. This can lead to goal hijacking, data exfiltration, or unintended behavior.

• Direct Injection: Malicious instructions are placed directly in the user query.
• Indirect Injection: Malicious instructions are embedded within data retrieved from an external source (e.g., a compromised web page or database), which the model then processes.
• Primary Risk: Breaks the intended separation between instruction and data.

An adversarial example is an input crafted with small, often imperceptible perturbations to cause a machine learning model to make a high-confidence error. Originally from computer vision (e.g., altering pixels), the concept extends to NLP as adversarial prompts.

• Core Principle: Exploits the model's sensitivity to specific input features.
• In Text: Can involve character-level manipulations (Unicode exploits), synonym substitution, or adding distracting context.
• Automation: Gradient-based attacks or LLM-as-attacker frameworks can automate the search for these effective perturbations.

In AI security, red teaming is the systematic, proactive practice of simulating adversarial attacks to identify vulnerabilities in a model's safety, alignment, and robustness before deployment.

• Objective: Discover failure modes like jailbreaks, prompt injections, and harmful content generation.
• Methods: Can be manual (human experts crafting prompts) or automated red teaming using algorithms to generate and test thousands of variants.
• Outcome: Provides a risk assessment to guide the hardening of models and safety systems.

Safety filter bypass is the overarching objective of many adversarial prompting techniques. It refers to successfully circumventing the layered content moderation and refusal mechanisms—which can exist within the model's own training (alignment) or in an external application-layer filter.

• Targets: Includes output filters that scan generated text and input classifiers that screen user queries.
• Evasion Techniques: Obfuscation (token manipulation), context confusion, and indirect injection are common methods.
• Importance: Demonstrates the cat-and-mouse game between model defenders and attackers.

A universal adversarial prompt (UAP) is a single string of tokens that, when appended to many different and seemingly benign user queries, reliably causes a language model to generate harmful outputs. Unlike targeted jailbreaks, UAPs seek a general attack transferability.

• Characteristic: Often discovered through automated optimization techniques that search the input space.
• Impact: Highlights a systemic vulnerability rather than a niche exploit.
• Research Focus: Understanding why certain suffixes act as a universal trigger across diverse prompts.