Instruction Hierarchy

Establishes a strict precedence order for instruction sources, typically structured as:

• System Message (Highest Priority): Core behavioral constraints and safety policies set by the model provider or enterprise administrator
• User Prompt (Medium Priority): Task instructions and queries from the end user
• Retrieved Data (Lowest Priority): Content fetched from external sources, documents, or tool outputs

This tiering prevents prompt injection attacks where malicious data embedded in a webpage or document attempts to override the agent's safety guardrails.

When instructions from different privilege levels conflict, the hierarchy enforces a deterministic resolution mechanism:

• Higher-tier directives always override lower-tier ones
• The agent is trained to recognize and ignore misaligned instructions from lower-privilege sources
• Example: If a user prompt says 'Ignore all previous instructions' but the system message forbids this, the system message wins

This eliminates ambiguity in adversarial scenarios where an attacker attempts to jailbreak the model through crafted inputs.

Models are fine-tuned to internalize the hierarchy using algorithmically generated training examples:

• Creates scenarios where lower-privilege instructions attempt to contradict higher-privilege ones
• Trains the model to selectively comply with aligned instructions while rejecting misaligned ones
• Includes adversarial examples where retrieved data mimics system-level language to test boundary recognition

This approach, pioneered by Anthropic and OpenAI, teaches models to generalize the hierarchy to novel attack patterns without requiring exhaustive real-world attack data.

Instruction hierarchy operates as one layer within a broader multi-layered safety architecture:

• Complements output sanitization and content filters that catch violations post-generation
• Works alongside action gates that require human approval for high-stakes operations
• Integrates with constrained decoding to enforce schema compliance at the token level

This layered approach ensures that even if one safety mechanism fails, others provide redundant protection against harmful agent behaviors.

The hierarchy extends beyond text generation to govern agentic tool execution:

• System-level policies can restrict which API endpoints an agent may call, regardless of user requests
• Retrieved tool outputs are treated as lowest-privilege data, preventing poisoned API responses from hijacking agent behavior
• Example: A system directive can forbid an agent from executing DELETE operations on a production database, even if a user prompt or retrieved document instructs otherwise

This is critical for least privilege execution in autonomous agent deployments.

Instruction hierarchy robustness is validated through systematic adversarial testing:

• Red-team exercises craft prompts that attempt to make lower-tier instructions masquerade as system-level directives
• Metrics track the override success rate—the percentage of attacks successfully blocked
• Benchmarks include scenarios with nested instructions, multi-turn dialogues, and code injection attempts

Continuous evaluation ensures the hierarchy remains effective against evolving attack vectors in production deployments.

A security principle that restricts an agent's access permissions and tool capabilities to the absolute minimum necessary to perform its designated task.

Implementation patterns:

• Role-based access control (RBAC) for tool authorization
• Capability-based security tokens scoped to specific operations
• Just-in-time privilege elevation with automatic revocation

This principle extends instruction hierarchy to the execution layer—even if a lower-privilege instruction manages to execute, its blast radius is contained by restricted tool access.

A control point in an agentic workflow that requires explicit validation or approval before a high-stakes tool call or state-changing operation is executed.

Gate types include:

• Human-in-the-loop gates: Require manual operator approval for critical actions
• Policy-based gates: Automatically validate against rules engines like OPA
• Confidence-based gates: Block actions when model certainty falls below a threshold

Action gates serve as the enforcement mechanism for instruction hierarchy by ensuring that system-level safety policies are checked before any privileged operation proceeds.

A secondary language model or classifier that evaluates the primary agent's outputs for correctness, safety, or alignment, providing a feedback signal for rejection or refinement.

Architecture patterns:

• LLM-as-judge: A separate model scores outputs against rubrics
• Specialized classifiers: Toxicity detectors, factuality verifiers, policy compliance checkers
• Ensemble critics: Multiple evaluators voting on output safety

Critic models operationalize instruction hierarchy by acting as an independent verification layer that can override the primary agent's decisions when they violate system-level constraints.

A security mechanism that runs an agent's generated code or tool calls inside an isolated, ephemeral environment—such as a Docker container or gVisor sandbox—to prevent access to the host system.

Key properties:

• Filesystem isolation: No access to host directories
• Network restrictions: Egress filtering and domain allowlisting
• Resource limits: CPU, memory, and time quotas enforced
• Ephemeral state: Environment destroyed after execution completes

Sandboxing provides defense-in-depth for instruction hierarchy by ensuring that even if lower-privilege instructions bypass other controls, they cannot cause persistent damage.