Guardrail Violation Detection and Classification Prompt Template

This prompt is designed for trust and safety engineers who need a programmable, auditable detection surface for AI behavior that violates defined safety, policy, or operational guardrails. The core job-to-be-done is converting a human-readable policy document into a structured, machine-actionable violation record whenever an AI's input or output crosses a defined boundary. The ideal user is an engineer or compliance officer integrating this prompt into an AI pipeline—such as a chatbot, agent, or content generation system—who requires consistent classification, policy citation, and evidence extraction to feed downstream actions like blocking, logging, or human review queues. You must have a specific, well-defined policy document to use this prompt effectively; it is not a general-purpose content filter or a substitute for a comprehensive safety strategy.

Do not use this prompt when you lack a concrete policy, when the policy is too vague to produce consistent classifications, or when you need real-time blocking with sub-millisecond latency without a supporting application harness. This prompt is also inappropriate for detecting subtle context-dependent harms that require deep cultural or situational understanding beyond the provided policy text. Instead, use it as a programmable layer within a broader guardrail system: feed it the AI's input or output alongside your policy, and consume the structured violation record it produces. The prompt requires careful calibration with a golden dataset of labeled examples to set acceptable false positive and false negative rates, and it should always be paired with eval checks for policy alignment accuracy before production deployment.

Before integrating this prompt, ensure you have a version-controlled policy document, a set of representative violation and non-violation examples for testing, and a clear decision on how the output violation record will be consumed—whether by an automatic blocking rule, a human review queue, or an audit logging system. Start by running the prompt against your eval dataset to measure precision and recall, then adjust the policy text or prompt instructions to balance over-triggering against missed violations. Never deploy this prompt without a human-in-the-loop escalation path for high-severity or ambiguous cases, especially in regulated domains where false negatives carry legal or safety risk.

This prompt template is the core detection surface for your guardrail system. It instructs the model to act as a policy auditor, analyzing the provided interaction against a defined set of rules. The output is a structured JSON object that can be directly consumed by an escalation engine, a review queue, or a logging system. The template is designed to be policy-agnostic; you supply the specific rules through the [POLICY_DOCUMENT] placeholder, making it reusable across content safety, security, data governance, and compliance use cases.

textCopy
You are a Guardrail Policy Auditor. Your task is to analyze the provided [INPUT_TYPE] against the defined policy. You must detect, classify, and extract evidence for any policy violation.

INPUT:
[INPUT]

POLICY:
[POLICY_DOCUMENT]

INSTRUCTIONS:
1. Analyze the INPUT for any violation of the POLICY.
2. If no violation is found, set `violation_detected` to `false` and leave the `violations` array empty.
3. If a violation is detected, set `violation_detected` to `true` and populate the `violations` array with an object for each distinct violation found.
4. For each violation, you must provide:
   - `policy_id`: The specific rule or clause identifier from the POLICY that was violated.
   - `severity`: Your classification of the violation's severity. Must be one of [SEVERITY_LEVELS].
   - `evidence_excerpt`: A direct, verbatim quote from the INPUT that constitutes the violation. Do not paraphrase.
   - `reasoning`: A brief, step-by-step explanation connecting the evidence to the specific policy clause.

OUTPUT_SCHEMA:
{
  "violation_detected": boolean,
  "violations": [
    {
      "policy_id": "string",
      "severity": "string",
      "evidence_excerpt": "string",
      "reasoning": "string"
    }
  ]
}

CONSTRAINTS:
- Output ONLY the valid JSON object. No other text.
- The `evidence_excerpt` must be a substring of the INPUT.
- If the INPUT is ambiguous, apply the principle of "when in doubt, escalate" by flagging a low-severity violation and noting the ambiguity in the `reasoning`.

To adapt this template, replace the placeholders with your operational specifics. [INPUT] should be the raw text of the user query, model response, or full conversation transcript. [POLICY_DOCUMENT] is the most critical part: it should contain your enumerated, specific, and unambiguous policy rules with unique identifiers (e.g., 'SAFETY-01: No instructions for creating weapons'). [SEVERITY_LEVELS] must be a strict enum list like ["CRITICAL", "HIGH", "MEDIUM", "LOW"]. The [INPUT_TYPE] placeholder (e.g., 'user prompt', 'model output', 'retrieved document') helps the model orient itself. For high-stakes domains, always route any output where violation_detected is true to a human review queue before taking automated action like account suspension or content removal.

The guardrail violation detection prompt is not a standalone artifact—it is a decision node inside a larger safety pipeline. In production, this prompt typically sits between the model output generation step and the response delivery step. When a user-facing or agentic system produces an output, that output (along with surrounding context) is passed to this prompt for classification. The prompt returns a structured violation record. If violation_detected is true and severity exceeds a configured threshold, the application must block the output from reaching the user and instead route to an escalation path. If no violation is detected, the output proceeds. This gating pattern means the prompt's latency directly adds to the user's perceived response time, so model choice and caching strategy matter.

Wire the prompt into an async post-processing step rather than blocking the main generation loop where possible. Use a fast, cost-efficient model for the detection call—GPT-4o-mini, Claude Haiku, or a fine-tuned small classifier are common choices. The detection prompt should receive the full model output, the preceding user input, the active system prompt or policy document, and any tool call context. Validate the response against a strict JSON schema before trusting the violation_detected field. If the model returns malformed JSON, retry once with a repair prompt that includes the raw response and the expected schema. If the retry also fails, escalate to a human review queue with the raw output attached and log the schema failure as a separate observability event. For high-traffic systems, implement a sampling strategy: classify 100% of outputs for high-risk actions (financial, medical, child safety) and a configurable percentage for lower-risk content categories.

Log every classification result—including negative findings—to a structured audit store with the prompt version, model identifier, input hash, output hash, violation verdict, severity, policy references, and timestamp. This audit trail is essential for governance reviews, false positive analysis, and regulatory compliance. When a violation is detected, the application should assemble an evidence package from the classification output and route it to a human review queue. The queue item must include the violation summary, severity, the excerpt of the flagged content, the specific policy clause violated, and the decision options available to the reviewer. Do not rely solely on the prompt's severity classification to determine routing—apply a configurable rules engine in the application layer that maps severity levels and violation types to specific queues, SLAs, and escalation paths. This separation keeps policy logic auditable without requiring prompt changes for every routing adjustment.