Harmful Content Generation

A specific type of adversarial input crafted to bypass a language model's built-in safety filters and content moderation policies. The goal is to elicit responses—such as hate speech, illegal content, or detailed instructions for harm—that the model is explicitly designed to refuse. Techniques include role-playing scenarios, fictional frameworks, or encoding tricks that obscure the prompt's intent from automated scanners.

An adversarial attack technique where a malicious user provides input designed to override or subvert a language model's original system instructions. Unlike jailbreaks focused on content policies, prompt injection often aims to hijack the model's goal, such as making it ignore previous context, extract confidential data, or perform unauthorized actions. This is a critical vulnerability for applications that concatenate user input with trusted system prompts.

An attack where malicious instructions are embedded within data retrieved from an external source, such as a database, website, or document. When a Retrieval-Augmented Generation (RAG) system or other tool fetches this data, the poisoned content is injected into the model's context, subverting its function. This is particularly dangerous because the attack vector is not the direct user query but the trusted knowledge base itself.

A string of tokens appended to a user query, often discovered through automated optimization algorithms, that systematically induces a model to comply with harmful requests. Research has shown that relatively short, seemingly nonsensical suffixes can act as a universal key, causing models to ignore safety training. This demonstrates the fragility of alignment and the need for robust inference-time defenses.

The systematic practice of simulating adversarial attacks to proactively identify vulnerabilities in a model's safety and alignment. In AI security, this involves human experts or automated systems crafting jailbreak prompts, injection attacks, and other exploits in a controlled environment. The findings are used to patch vulnerabilities, improve safety training, and harden models before deployment.

The general objective of adversarial prompting techniques aimed at circumventing content moderation mechanisms. These filters can exist at multiple layers:

• Model-level refusals trained via Reinforcement Learning from Human Feedback (RLHF).
• Post-processing classifiers that scan output before it's shown to the user.
• Input sanitization that checks prompts for blocked terms. Bypassing these layers is the core challenge for attackers and the key defense metric for model developers.