Jailbreaking

Attackers coerce the model into adopting a fictional persona that operates outside standard safety constraints. By framing the interaction as a creative writing exercise or a hypothetical scenario, the model's refusal mechanisms are bypassed.

• DAN (Do Anything Now): A classic prompt that instructs the model to act as an unconstrained alter-ego.
• Character Override: The model is told it is a character with no ethical guidelines.
• Nested Personas: Multiple layers of fictional roles are used to obscure the final objective from the content filter.

Malicious instructions are fragmented across multiple seemingly benign prompts or encoded in non-natural language formats. The model reassembles the fragments during inference, executing a prohibited command that no single prompt contained.

• Token Smuggling: Breaking keywords into sub-strings to evade keyword filters.
• Multi-Turn Decomposition: Splitting a dangerous request across a conversation history.
• Base64 Encoding: Instructing the model to decode and execute a base64-encoded malicious payload.

The model's limited attention window is flooded with complex, benign tasks or excessive constraints to exhaust its safety reasoning capacity. The prohibited request is buried within a sea of legitimate instructions.

• Long-Context Exploitation: Placing a harmful instruction in the middle of a 100k+ token document.
• Cognitive Overload: Demanding strict output formatting rules that consume the model's reasoning budget.
• Emotional Manipulation: Framing the request as a life-or-death emergency to override logical safety checks.

Researchers have discovered suffix strings—often nonsensical tokens—that, when appended to any harmful prompt, drastically increase the likelihood of a successful jailbreak across different model architectures.

• Greedy Coordinate Gradient (GCG): An algorithm that automatically discovers adversarial suffixes.
• Transferability: A suffix optimized on one open-source model often transfers to closed-source models.
• Automated Red Teaming: Using one model to generate jailbreaks for another target model.

Safety alignment is often bypassed by embedding prohibited instructions in non-text modalities that the vision-language model can read but the text-only safety classifier cannot parse.

• OCR Jailbreaks: Embedding hidden text in images that the model reads and executes.
• Audio Encoding: Using inaudible frequencies or encoded speech to issue commands.
• Typographic Attacks: Placing a malicious prompt as text within a complex visual scene.

Rather than tricking the model into compliance, these attacks directly suppress the model's ability to generate a refusal token. The model is forced to begin its response with an affirmative statement.

• Prefix Injection: Demanding the response start with 'Absolutely! Here is how...'
• Grammar Constraints: Forcing the output to follow a JSON schema that has no field for refusal.
• Few-Shot Poisoning: Providing examples in the prompt where all requests are answered without refusal.

A subtle, often imperceptible modification to input data specifically crafted to cause a machine learning model to make an incorrect prediction. In the context of language models, this translates to character-level manipulations—such as adding typos, using leetspeak, or appending nonsensical strings of tokens—that break safety classifiers while preserving semantic meaning for the human reader.

A fine-tuning technique that aligns a language model with human preferences by training a reward model on human-ranked outputs and using reinforcement learning to optimize the policy. The quality of the harmlessness reward model is the primary defense against jailbreaking; a poorly trained reward model can be exploited by adversarial prompts that appear benign to the annotator but trigger harmful latent knowledge.