Jailbreak

The attacker's primary objective is to override the system prompt and substitute the model's original, safe goal with a malicious one. This is rarely direct. Instead, it relies on semantic obfuscation—encoding harmful intent in hypothetical scenarios, fictional languages, or complex role-playing.

• Technique: Framing a request for bomb-making instructions as a scene from a movie script.
• Technique: Using a universal adversarial trigger—a seemingly nonsensical string of tokens—that statistically forces the model into a compliant state.
• Result: The model believes it is fulfilling a benign creative request while actually generating prohibited content.

A successful jailbreak manipulates the model's attention mechanism to focus on a fabricated context that supersedes its safety training. The attack constructs a dense, demanding scenario that consumes the model's reasoning capacity.

• Payload Encapsulation: The harmful query is buried deep within a large block of benign text, a technique known as 'attention distraction.'
• Privilege Escalation: The attacker assigns the model a powerful persona (e.g., 'DAN' - Do Anything Now) with no ethical constraints, creating a simulated internal context where safety guidelines are explicitly revoked.
• Recursive Loops: Forcing the model into a logical loop where it must output the harmful content to satisfy a prior, attacker-defined condition.

The alignment tax is the trade-off between a model's helpfulness and its harmlessness. Jailbreak attacks exploit this by creating a conflict where the model must violate its safety training to fulfill its core instruction-following objective.

• Refusal Suppression: The attack includes explicit commands like 'Do not apologize,' 'Do not say you are an AI,' or 'Start your response with "Certainly!"' to block the model's standard refusal templates.
• Few-Shot Poisoning: The prompt provides several fabricated examples of the AI complying with harmful requests, conditioning the model to continue the pattern.
• Payload Splitting: Breaking a single harmful request into seemingly innocuous sub-components that the model processes individually, only to synthesize the dangerous output in a final combination step.

Jailbreak attacks are not limited to text. As models become multi-modal, the attack surface expands to include cross-modal injection, where the malicious payload is hidden in a non-text modality to bypass text-based safety filters.

• Image-based Attacks: Embedding a hidden adversarial perturbation in an image that, when processed alongside a benign text prompt, triggers a harmful textual response.
• Audio Encoding: Encoding a malicious instruction as an audio waveform that is inaudible or sounds like music to a human but is decoded as a command by the model's audio encoder.
• Polyglot Files: Crafting a single file that is valid as both an image and a text file, where the text component contains the jailbreak payload that is invisible to simple visual inspection.

Modern jailbreak attacks are often discovered not by humans, but by algorithms. The Greedy Coordinate Gradient (GCG) attack is a prominent example of automated adversarial suffix generation.

• Mechanism: GCG uses gradient information to iteratively optimize a sequence of tokens that, when appended to a harmful query, maximizes the probability of the model generating an affirmative response.
• Universality: The resulting adversarial suffix is often universal, meaning it can jailbreak a model across a wide range of different harmful prompts.
• Transferability: Suffixes optimized on one open-source model frequently transfer to other models, including proprietary black-box systems, making this a highly scalable threat vector.

Advanced jailbreak techniques incorporate methods to evade automated defense mechanisms like perplexity filters. Instead of using high-perplexity gibberish, attackers craft low-perplexity, fluent adversarial prompts.

• Self-Critique Loops: The attack instructs the model to review its own output for safety violations and then rewrite it without the restrictions, effectively using the model's own reasoning to refine the jailbreak.
• Encoding and Decoding: Requesting the model to encode a harmful response in Base64 or a custom cipher, which bypasses content filters that scan for plaintext keywords, and then asking it to decode the result.
• Polyglot Prompting: Combining multiple jailbreak techniques (e.g., role-playing + refusal suppression + payload splitting) into a single, highly resilient attack that remains effective even if one vector is patched.

A defense mechanism that detects and blocks adversarial prompts by measuring their log-perplexity under a language model. It operates on the assumption that gibberish adversarial suffixes, like those generated by GCG, have high perplexity.

• A threshold is set; any input with a perplexity score above it is rejected before reaching the core model.
• Limitation: Advanced attacks can use perplexity-constrained optimization to generate fluent, low-perplexity jailbreaks that bypass this filter.

A structured adversarial testing process where a dedicated team simulates a malicious actor to probe an AI system for vulnerabilities, biases, and failure modes before deployment.

• Manual Red-teaming: Human experts use creativity and domain knowledge to find novel jailbreaks.
• Automated Red-teaming: Uses algorithms to generate and test thousands of adversarial prompts at scale.
• The goal is to discover edge cases and failure modes that automated metrics miss, providing a qualitative assessment of safety.

The resilience of a model trained with Reinforcement Learning from Human Feedback (RLHF) against adversarial attacks that attempt to subvert its learned human preferences and safety guardrails.

• RLHF trains a reward model on human preferences for helpful and harmless outputs, then fine-tunes the LLM to maximize this reward.
• A robust RLHF process must include adversarial data in the preference training set, teaching the reward model to correctly penalize jailbreak attempts that superficially appear helpful.