Safety Policy Violation Classification Prompt

This prompt is designed for trust and safety engineering teams who need to automate the first pass of content moderation classification. Use it when you have a defined safety policy taxonomy and need every user request categorized into one or more violation categories with confidence scores, specific policy citations, and severity ratings. The primary job-to-be-done is turning an unstructured user input into a structured, machine-readable violation label that downstream routing logic, human review queues, or audit logging systems can consume without ambiguity.

This is not a refusal prompt. It does not decide what action to take. It classifies the violation so downstream systems can apply the correct policy response. Do not use this prompt when you need the model to generate a user-facing refusal message, when you lack a codified safety policy taxonomy, or when the decision requires real-time legal interpretation beyond your documented policies. The output is designed to be calibrated against human moderator decisions, so it is most effective when you have a labeled dataset of human decisions to measure precision and recall against.

Before deploying this prompt, ensure you have a version-controlled safety policy document, a mapping of policy categories to violation codes, and a severity rubric that defines what constitutes low, medium, high, or critical severity. The prompt expects these as input context. If your policy taxonomy changes frequently, treat this prompt as a living artifact that must be regression-tested against a golden set of labeled examples after every policy update. Start with a human-reviewed pilot on a sample of production traffic before removing the human-in-the-loop step.

This prompt template is the core classification engine for a content moderation pipeline. It takes a user request, an optional conversation context, and your organization's safety policy taxonomy, then returns a structured JSON object that identifies which policies were violated, how confident the model is in each classification, and the severity of the violation. The output is designed to be consumed by downstream routing logic, human review queues, or automated enforcement systems. Do not use this prompt for ambiguous gray-zone requests without a human-in-the-loop review step; it is optimized for clear policy boundaries, not edge-case adjudication.

textCopy
You are a safety policy classifier. Your job is to analyze a user request against a provided safety policy taxonomy and return a structured JSON classification.

## POLICY TAXONOMY
[POLICY_TAXONOMY]

## USER REQUEST
[USER_REQUEST]

## CONVERSATION CONTEXT (OPTIONAL)
[CONVERSATION_CONTEXT]

## INSTRUCTIONS
1. Analyze the user request against each policy category in the taxonomy.
2. For each policy category, determine if the request violates, potentially violates, or complies with the policy.
3. Assign a confidence score between 0.0 and 1.0 for each classification.
4. Assign a severity rating of "low", "medium", "high", or "critical" for any violation.
5. Cite the specific policy clause or rule that supports your classification.
6. If the request is ambiguous, flag it for human review and explain why.
7. Do not refuse the request in your response; only classify it.

## OUTPUT SCHEMA
Return ONLY valid JSON matching this schema:
{
  "classifications": [
    {
      "policy_id": "string",
      "policy_name": "string",
      "violation_detected": boolean,
      "confidence": number,
      "severity": "low" | "medium" | "high" | "critical" | null,
      "policy_citation": "string",
      "rationale": "string",
      "requires_human_review": boolean,
      "human_review_reason": "string or null"
    }
  ],
  "overall_severity": "low" | "medium" | "high" | "critical" | "none",
  "requires_immediate_action": boolean,
  "summary": "string"
}

## CONSTRAINTS
- Return ONLY the JSON object. No markdown fences, no explanatory text.
- If no policies are violated, return an empty classifications array with overall_severity "none".
- Confidence scores must reflect genuine uncertainty; do not default to 1.0.
- Severity must be null when violation_detected is false.
- Flag for human review when confidence is below [CONFIDENCE_THRESHOLD] or when the request falls in a policy gray zone.

To adapt this prompt for your production pipeline, replace [POLICY_TAXONOMY] with your organization's actual policy definitions, including category IDs, names, descriptions, and example violations. The taxonomy should be detailed enough that the model can distinguish between adjacent policy categories—for example, separating "harassment" from "hate speech" with clear boundary definitions. Replace [CONFIDENCE_THRESHOLD] with a numeric value appropriate for your risk tolerance; 0.85 is a common starting point for automated enforcement, while 0.70 might be acceptable for human-review triage. The [CONVERSATION_CONTEXT] placeholder should receive prior turns if this classification is part of a multi-turn conversation, formatted as a clear transcript with speaker labels. Before deploying, run this prompt against a golden dataset of at least 200 labeled examples that includes clear violations, compliant requests, and deliberate edge cases. Compare the model's classifications against human moderator labels to calibrate your confidence threshold and identify systematic over- or under-classification patterns.

This prompt is not a standalone chat interaction. It is a classification node inside a broader moderation pipeline. The implementation harness must treat the model's output as a structured signal that feeds downstream decisions—block, quarantine, flag for review, or allow. The harness is responsible for enforcing the output contract, handling model failures gracefully, and ensuring that high-severity or low-confidence classifications always reach a human moderator before action is taken against a user or content item.

Start by wrapping the prompt call in a function that accepts the user request text and an optional content payload. Before calling the model, validate that the input is not empty and does not exceed your maximum context window after prompt assembly. On the model response, parse the JSON output and validate it against a strict schema: violations must be an array of objects, each containing policy_id (string, required), policy_name (string, required), confidence (float between 0.0 and 1.0), severity (enum of low, medium, high, critical), evidence_quote (string, max 500 chars), and rationale (string). If parsing fails or schema validation fails, retry once with the error message appended to the prompt as a correction instruction. If the retry also fails, log the raw output and escalate the entire item to the human review queue with a CLASSIFICATION_PARSE_ERROR flag.

Confidence thresholds should be configurable per policy category. For example, a confidence below 0.7 on a critical severity classification should automatically route to human review rather than triggering an automated block. Similarly, any classification where the top two predicted policy categories have confidence scores within 0.15 of each other indicates ambiguity and should be escalated. Log every classification decision with the model version, prompt template hash, input hash, raw response, parsed output, validation status, and final routing decision. This audit trail is essential for calibrating thresholds, detecting model drift, and defending moderation decisions to users or regulators.

For high-throughput systems, consider batching multiple inputs into a single model call where the prompt template supports a JSON array of inputs and returns a corresponding array of classifications. This reduces latency and cost but requires careful token budgeting and increases the blast radius of a single malformed response. Always set a timeout on the model call (e.g., 10 seconds for a single item, 30 seconds for a batch of 10) and implement a circuit breaker that stops calling the model if the error rate exceeds 20% in a rolling 5-minute window. When the circuit is open, fall back to a rule-based keyword classifier and queue all items for human review.

Human review integration should expose the original content, the model's classification, the evidence quote, and the confidence score in a review UI. Reviewers should be able to confirm, override, or add policy categories. Their decisions should be logged and periodically used to recalibrate confidence thresholds and update few-shot examples in the prompt template. Do not use reviewer decisions to automatically fine-tune the model without a separate review and approval process. The goal is a tight feedback loop between automated classification and human judgment, not an unsupervised learning loop that can amplify errors.