Prompt Injection Boundary

The boundary must enforce a hard syntactic separation between instructions and data. This is achieved through structured formats like Chat Markup Language (ChatML) or XML tags, where user input is strictly confined to a content block. The model is conditioned to treat the system prompt as immutable meta-instructions and the user block as a passive string, preventing the classic 'ignore previous instructions' attack vector.

Before user data reaches the model, it must pass through a pre-processing firewall. This layer strips or escapes special characters and known control sequences that mimic instruction syntax. Techniques include:

• Blacklisting known injection strings (e.g., '---END INSTRUCTION---').
• Whitelisting allowed characters for specific fields.
• Applying heuristic filters that detect the semantic intent of instruction-following within the user input, even if syntactically disguised.

A boundary extends to the output side. Output guardrails analyze the model's response to detect if the model was successfully jailbroken. This involves scanning for the presence of leaked system prompts or disallowed content categories. If a violation is detected, the system must quarantine the response and return a generic error message, preventing the exfiltration of proprietary logic or the execution of harmful downstream actions.

In agentic architectures, the boundary is enforced by disallowing direct model access to tools. The model generates a structured function call (e.g., JSON), which is parsed and validated by a deterministic runtime outside the model's context. This runtime checks the parameters against a strict schema before executing any API call. The model never sees raw API keys or executes code directly, creating an air-gap between the untrusted linguistic space and the trusted execution environment.

To detect boundary breaches, the system prompt can embed cryptographic canary tokens—unique, random strings that should never appear in normal output. If a canary token appears in a model's response, it serves as a high-fidelity signal that the system prompt has been exfiltrated. Furthermore, each user session should operate in a logically isolated context, preventing cross-session contamination where an injection in one conversation poisons the memory of another.

The boundary logic must differentiate between active user input and passive retrieved data. A common attack vector is indirect prompt injection, where malicious instructions are hidden in a document that a RAG system retrieves. A robust boundary treats all externally sourced data (web pages, PDFs, emails) as untrusted, applying the same sanitization and syntactic confinement rules to retrieved context as it does to direct user input, preventing the model from executing instructions hidden in search results.

Safety filters that evaluate the full conversational context to block policy-violating prompts or responses. Unlike simple keyword filters, contextual guardrails analyze semantic meaning across multiple turns to detect indirect attacks that exploit the injection boundary.

• Operate as a separate classification layer
• Detect jailbreak attempts embedded in benign-looking text
• Can be implemented via LLM-as-judge or dedicated classifiers

A structured formatting schema like ChatML that delineates roles and turns within a message array. Proper chat templating is the first line of defense against injection—it ensures the model correctly parses conversational structure and distinguishes developer instructions from user utterances.

• Uses special tokens: <|im_start|>user, <|im_end|>
• Prevents user input from escaping its designated role
• Critical for maintaining the injection boundary in multi-turn dialogue

A binary tensor applied during self-attention to prevent the model from attending to specific tokens. In the context of injection defense, attention masks can be used to architecturally enforce the boundary by restricting the model's ability to attend to user input when processing system instructions.

• 1 = attend, 0 = ignore
• Can isolate untrusted content from trusted prompt regions
• Experimental defense against indirect prompt injection

A failure state where the model loses distinction between different conversational threads or temporal states, flattening the dialogue into a single incoherent prompt. Context collapse often results from injection attacks that successfully blur the boundary between trusted instructions and untrusted input.

• Symptom: model follows user commands meant for system level
• Indicates failed input sanitization or delimiter parsing
• Recovery requires session reset and boundary reinforcement