System Prompt Hardening

A safety framework that establishes a strict privilege order for instructions: system-level directives override user-level inputs, which override tool-level outputs. This prevents lower-privilege sources from subverting core constraints.

• System > User > Tool: The canonical priority chain
• Prevents a malicious tool output from rewriting system rules
• Implemented via explicit delimiters and structured prompt formatting
• Anchors all other hardening techniques by defining who can say what

A mitigation technique that uses special character sequences to create unambiguous boundaries between trusted system instructions and untrusted user input. Common delimiters include XML tags, triple backticks, or custom tokens.

• Example: <system>You are a helpful assistant</system> <user>User query here</user>
• Prevents user input from being interpreted as system commands
• Must be paired with input sanitization to block delimiter injection
• Foundation for structured prompt engineering in production LLM applications

The process of cleaning and normalizing user-provided text to remove or neutralize potentially malicious control sequences before model processing. This includes stripping escape characters, normalizing Unicode, and detecting delimiter-like patterns.

• Removes zero-width characters that break tokenization
• Detects homoglyph attacks using visually similar characters
• Canonicalizes encoding to prevent filter bypass
• Acts as the first line of defense before prompt assembly

Constraining a model to generate responses in a specific, machine-readable format like JSON or XML to prevent the execution of injected free-form instructions. This technique limits the attack surface by removing the model's ability to output arbitrary text.

• Forces outputs through a schema validation layer
• Prevents injected content from becoming executable instructions
• Enables programmatic parsing and downstream safety checks
• Commonly implemented via function calling or grammar-constrained decoding

A safety technique that fine-tunes a model to explicitly reject requests that violate its usage policies, making it significantly harder to jailbreak via injection. The model learns to identify and refuse harmful instructions even when they are embedded within legitimate-appearing queries.

• Uses adversarial training examples during the RLHF or DPO phase
• Creates a robust 'refusal boundary' around core safety policies
• Resistant to social engineering and role-playing attacks
• Complements prompt-level defenses with model-level safeguards

A defensive technique that strictly segregates different information sources within a prompt to prevent cross-contamination and privilege escalation. Each context segment—system prompt, conversation history, retrieved documents—is wrapped in its own isolated boundary.

• Prevents a poisoned document from rewriting conversation history
• Uses distinct formatting or embedding spaces for each context type
• Critical for RAG systems where retrieved content is untrusted
• Enforces the principle of least privilege at the prompt assembly level

A safety framework that establishes a strict privilege ordering for instructions processed by a model. System-level directives are given the highest priority, followed by user messages, and finally tool outputs. This prevents a lower-privilege source, such as a retrieved document, from overriding core system rules.

• System: Immutable, highest authority
• User: Can be overridden by system
• Tool: Lowest authority, never overrides system or user constraints

A mitigation technique that uses special character sequences to create unambiguous boundaries between trusted system instructions and untrusted user input. By wrapping user content in markers like --- or XML tags, the model is explicitly told which sections are authoritative and which are potentially hostile.

• Prevents prompt injection by isolating untrusted content
• Common delimiters: ---, ###, <user_input>...</user_input>
• Requires the model to respect delimiter semantics in its training

The process of cleaning and normalizing user-provided text before it reaches the model. This includes stripping control characters, normalizing Unicode, and removing known injection patterns. Sanitization acts as a first-line filter to neutralize malicious payloads.

• Removes zero-width characters and homoglyphs
• Normalizes Unicode to prevent encoding-based bypasses
• Often implemented as a pre-processing middleware layer

A secondary, often smaller, model that operates as a policy enforcement layer around a primary LLM. Guard models screen both inputs and outputs for policy violations, injection attempts, and harmful content. They provide an independent security boundary that is harder to compromise than in-prompt defenses.

• Screens inputs for injection and jailbreak attempts
• Filters outputs for toxicity and data leakage
• Examples: Llama Guard, NVIDIA NeMo Guardrails

The use of classifiers, heuristics, and statistical analysis to identify user inputs that are likely crafted to manipulate a model. Detection systems analyze perplexity, semantic intent, and structural anomalies to flag suspicious prompts before they reach the core model.

• Perplexity analysis: Injected prompts often have unusual token distributions
• Intent classifiers: Detect goal-hijacking attempts
• Signature-based detection: Matches known attack patterns

A defensive technique that strictly segregates different information sources within a prompt to prevent cross-contamination. Each source—system instructions, conversation history, retrieved documents—is wrapped in its own isolated context block, preventing an attacker from leveraging one source to corrupt another.

• Prevents privilege escalation across context regions
• Uses structured formats like XML or JSON for explicit boundaries
• Complements instructional hierarchy with structural isolation