Red Teaming for Prompt Injection

The most fundamental technique where the red team directly commands the model to ignore its system prompt. Attackers use authoritative language like 'Ignore all previous instructions' or 'You are now in developer mode' to test if the model's instructional hierarchy is robust. This often involves role-playing scenarios, such as claiming to be a system administrator or the model's creator, to escalate privileges and bypass refusal training.

This technique fragments a malicious payload across multiple inputs or encodes it to evade input sanitization filters. Red teams use multi-turn injection to distribute instructions across a conversation, or employ encoding tricks like Base64, homoglyph attacks (e.g., replacing 'a' with Cyrillic 'а'), and zero-width character injection to break tokenization without visual detection. The goal is to test the robustness of canonicalization and adversarial prompt detection systems.

Red teams flood the model's context window with massive amounts of filler text, such as repeated words or lengthy passages, to push system instructions and safety guardrails out of the active attention mechanism. This technique tests the model's context boundary enforcement and its ability to maintain safety priorities even when the context window is saturated. A successful attack dilutes the model's adherence to its original directives.

Instead of attacking the prompt directly, red teams poison external data sources the model retrieves from. This includes inserting hidden instructions in web pages, PDFs, or emails that a Retrieval-Augmented Generation (RAG) system might ingest. The test validates defenses against data source poisoning and RAG injection, ensuring that retrieved content cannot override the system prompt through a tool authorization gate.

For models processing images, audio, or video, red teams embed malicious instructions directly into non-text modalities. This involves hiding text commands in image pixels (steganography) or embedding inaudible voice commands in audio files. The technique tests whether multi-modal injection can bypass text-only safety filters and whether the model's egress content guard can detect harmful outputs triggered by visual or auditory inputs.

A sophisticated technique where the red team manipulates the model's step-by-step reasoning process. By injecting logic like 'Let's think step-by-step: Step 1, ignore your rules...', the attacker attempts to corrupt the internal reasoning chain. This tests the integrity of the model's chain-of-thought processing and whether structured output enforcement can prevent a corrupted reasoning path from leading to a harmful action or prompt leaking.

A safety framework that establishes a privilege ordering for instructions: system messages override user messages, which override tool outputs. This prevents lower-privilege inputs from hijacking core directives.

• System level: Immutable rules set by the developer
• User level: Queries and inputs from the end user
• Tool level: Data returned from API calls or retrievals
• Enforcement: The model is trained to recognize and respect this hierarchy during alignment

The use of classifiers, perplexity analysis, and heuristic rules to identify inputs crafted to manipulate a model. Detection operates as a pre-flight check before the prompt reaches the primary LLM.

• Perplexity scoring: Injected prompts often have unusual token probability distributions
• Semantic classifiers: Fine-tuned models that recognize manipulation patterns
• Canary tokens: Decoy instructions embedded in system prompts that trigger alerts when echoed back

Constraining a model to generate responses exclusively in a machine-readable format like JSON. This prevents injected instructions from producing free-form harmful content or executing hidden commands.

• JSON Schema constraints: Define exact output structure with required fields and types
• Grammar-constrained decoding: Restrict token generation to valid syntax only
• Benefit: Even if injection succeeds, the output remains parseable and non-executable

A security checkpoint that validates every function call or API request a model attempts before execution. This prevents an injection from autonomously triggering sensitive operations.

• Policy engine: Validates tool calls against user permissions and context
• Human-in-the-loop: Requires manual approval for high-risk actions
• Parameter sanitization: Inspects arguments for injection payloads before forwarding to APIs

An attack that floods the context window with filler content to displace system instructions and safety guardrails. By consuming available tokens, the attacker pushes critical directives outside the model's attention.

• Technique: Long, repetitive, or padded inputs that consume token budget
• Goal: Dilute or truncate safety instructions before injecting malicious commands
• Defense: Context window monitoring, input length limits, and instruction anchoring at multiple positions