Safety Policy Violation Escalation Prompt

This prompt is designed for trust and safety teams, content moderation engineers, and AI product developers who need a reliable, auditable method for classifying content policy violations in AI-generated outputs or user-submitted inputs. The core job is to take a piece of content and a defined safety policy, and produce a structured violation assessment that includes the specific policy clause violated, a severity rating, and a recommended escalation action. This is not a general-purpose content filter; it is a decision-support tool for high-stakes workflows where a human reviewer must be looped in, and where the rationale for escalation must be traceable to a specific policy.

Use this prompt when your application requires a consistent, policy-grounded escalation decision rather than a binary safe/unsafe flag. It is appropriate for post-generation review queues, asynchronous content auditing pipelines, and real-time moderation tooling where a human operator needs a clear summary of what was violated and why before taking action. The prompt expects a well-defined safety policy document as input, not vague principles. It works best when the policy is structured into discrete, referenceable clauses (e.g., 'Policy 3.2: Depictions of Violence') that the model can cite. Do not use this prompt for real-time blocking without human review, as it is designed to escalate, not to autonomously reject content.

Avoid this prompt when you need a simple toxicity score, when your policy is a single paragraph of aspirational guidelines, or when the cost of a false escalation is higher than the cost of a missed violation. It is also unsuitable for detecting novel, zero-day policy violations that have no precedent in the provided policy text. Before implementing, ensure you have a human review queue ready to receive the structured escalations this prompt produces. The next step is to wire this prompt into an application harness with validation, logging, and a human-in-the-loop review interface.

This prompt template is designed for trust and safety teams who need consistent, auditable violation classification from an AI system. It accepts an input to review, a defined safety policy, and optional context about the interaction. The model is instructed to act as a policy reviewer, not a content moderator with broad discretion. It must cite specific policy clauses, assign a severity level, and recommend a concrete action. The template uses square-bracket placeholders so you can swap in your own policy document, risk levels, and output schema without rewriting the core instruction structure.

textCopy
You are a safety policy reviewer. Your job is to examine the provided [INPUT_TYPE] for violations of the safety policy below. You must be precise, evidence-based, and conservative. When in doubt, escalate rather than dismiss.

## SAFETY POLICY
[SAFETY_POLICY]

## INPUT TO REVIEW
[INPUT]

## INTERACTION CONTEXT (if available)
[CONTEXT]

## INSTRUCTIONS
1. Compare the input against each clause in the safety policy.
2. If a violation is detected, cite the specific policy clause ID and the exact text from the input that triggered it.
3. Assign a severity level from the following list: [SEVERITY_LEVELS].
4. Recommend exactly one action from: [ACTION_OPTIONS].
5. If no violation is detected, explicitly state "No violation detected" and set severity to "None".
6. Do not speculate about intent beyond what is evident in the input text.
7. If the input is ambiguous, note the ambiguity and recommend escalation.

## OUTPUT FORMAT
Return a valid JSON object matching this schema:
{
  "violation_detected": boolean,
  "violations": [
    {
      "policy_clause_id": string,
      "policy_clause_text": string,
      "violating_text": string,
      "severity": string,
      "rationale": string
    }
  ],
  "recommended_action": string,
  "ambiguity_notes": string | null
}

To adapt this template, replace the placeholders with your operational definitions. [SAFETY_POLICY] should contain the full text of your policy, including clause IDs for traceability. [SEVERITY_LEVELS] might be a list like "Critical, High, Medium, Low, None" and [ACTION_OPTIONS] might include "Block, Quarantine, Escalate, Flag for Review, Allow". The [INPUT_TYPE] placeholder helps the model understand whether it is reviewing a user message, an AI-generated response, or retrieved content. If you need the output in a different schema, modify the JSON structure but keep the field descriptions explicit. For high-stakes production use, always pair this prompt with a validation step that checks the output JSON against your expected schema before taking any automated action.

This prompt is designed to sit inside a synchronous moderation pipeline, not a standalone chat interface. The typical call pattern is: user-generated content or an AI-generated output is intercepted, passed to this prompt with the relevant policy document injected as [POLICY_DOCUMENT], and the resulting classification determines the next system action. The prompt expects a structured JSON output, which means your application harness must parse the response, validate the schema, and branch on the recommended_action field. Do not treat the model's text response as the final decision—treat it as a structured recommendation that your application enforces.

Schema validation is mandatory. Before acting on the output, validate that violation_classification is one of the allowed enum values you defined in [OUTPUT_SCHEMA], that severity is on your numeric scale, and that violating_content_excerpt is a non-empty string when a violation is found. If the model returns malformed JSON, missing required fields, or an unrecognized severity level, do not silently default to 'no violation.' Instead, implement a retry with repair pattern: send the raw output and the validation error back to the model with a repair instruction, or escalate to a human reviewer if the repair fails after one attempt. For high-severity classifications (e.g., CSAM, violent threats, self-harm), bypass retries entirely and route directly to a human review queue with the full context payload attached.

Logging and audit trail design is critical here. Every invocation should produce an immutable log record containing: the input content hash, the policy document version, the raw model output, the validated classification, the recommended_action, the action taken by the system, and the reviewer identity if a human was involved. This log becomes your audit evidence for content moderation appeals, regulatory inquiries, and policy effectiveness measurement. Use structured logging (JSON to your observability stack) rather than plain text. If your platform supports it, attach the prompt template version and model identifier to each log entry so you can trace classification shifts when you update the policy document or switch model providers.

Model choice matters for boundary precision. This prompt relies on the model's ability to distinguish between policy violation categories and to avoid over-flagging edge cases. In testing, models with stronger instruction-following and reasoning capabilities (e.g., Claude 3.5 Sonnet, GPT-4o) produce more consistent severity calibration than smaller or older models. If you must use a smaller model for latency or cost reasons, invest in a calibration eval set: a golden dataset of 200+ examples spanning clear violations, clear non-violations, and boundary cases, with expected classifications and severities labeled by human reviewers. Run this eval on every prompt or model change and gate deployment on precision and recall thresholds you define for each severity level.

Human review integration should be designed as a first-class path, not an afterthought. When recommended_action is escalate or human_review, your harness should create a review ticket with: the full input content, the model's classification and reasoning, the specific policy clauses cited, and a pre-built decision UI (Approve / Override / Request More Info). Timeout your review SLA: if a human hasn't acted within your defined window (e.g., 15 minutes for high-severity, 4 hours for medium), have a fallback action defined—typically temporary content quarantine, not auto-approval. Never auto-resolve an escalated item as 'safe' without human confirmation.

What to avoid: Do not use this prompt as a real-time chat filter where latency above 500ms is unacceptable without first benchmarking your model provider's response time. Do not skip the policy document injection step—asking the model to classify violations against an unspecified policy produces inconsistent, un-auditable results. Do not treat the model's severity score as a continuous risk score for downstream ML models without calibration; the ordinal severity levels here are designed for human-readable decision routing, not statistical risk modeling. Finally, never log the full user content in plaintext to a system without access controls—the violating content itself may contain PII, illegal material, or sensitive data that requires strict handling.