Toxic Output and Harmful Content Escalation Prompt

This prompt is designed for content safety teams and trust and safety engineers who need a reliable, automated first pass at detecting toxic, hateful, harassing, or violent content in AI-generated outputs. The primary job-to-be-done is transforming an unstructured, potentially harmful message into a structured violation record that can be ingested by a human review queue or an automated enforcement system. It is not a general-purpose content moderator for user-generated content, nor is it a one-off ChatGPT query for checking a single sentence. The ideal user is integrating this prompt into a production guardrail system where consistency, structured output, and clear escalation paths are non-negotiable.

Before using this prompt, you must have defined safety policies with clear categories (e.g., hate speech, harassment, violent content) and calibrated severity levels (e.g., low, medium, high, critical) that map to your organization's enforcement playbook. The prompt assumes these definitions exist and can be referenced. You should not use this prompt when you need real-time streaming intervention with sub-millisecond latency, as the structured generation and reasoning steps add overhead. It is also inappropriate for jurisdictions where automated content screening is legally restricted without immediate human oversight; in those cases, this prompt should feed a human review queue, not make autonomous blocking decisions. For high-severity cases involving child safety, imminent threats, or regulated content, the output of this prompt is an escalation artifact, not a final adjudication.

To implement this effectively, wire the prompt's structured output into a logging and routing pipeline. The violation record it produces should be validated against a strict schema before being accepted. If validation fails, retry with a repair prompt or escalate the raw output for manual triage. Always log the full prompt, response, and validation result for auditability. The next step after reading this section is to review the prompt template and adapt the policy references and severity definitions to match your organization's specific trust and safety framework.

This template is designed to be dropped into a content safety pipeline where an AI system's output must be screened before it reaches a user or is logged. It instructs the model to act as a strict content safety classifier, analyzing the provided text against defined harm categories. The prompt is structured to produce a machine-readable JSON record, making it suitable for direct integration into an automated escalation workflow. The square-bracket placeholders allow you to inject the specific text for review, your organization's safety policy, the required output fields, and any edge-case examples that have caused false positives or negatives in the past.

textCopy
You are a strict content safety classifier. Your task is to analyze the provided AI-generated text for toxic, hateful, harassing, or violent content. You must apply the provided safety policy without exception.

INPUT TEXT TO ANALYZE:
"""
[AI_OUTPUT_TEXT]
"""

SAFETY POLICY:
"""
[SAFETY_POLICY_DEFINITION]
"""

CLASSIFICATION INSTRUCTIONS:
1. Analyze the input text against each category defined in the safety policy.
2. For any violation, extract the exact text spans that constitute the violation.
3. Assign a severity score from the provided rubric.
4. If no violation is found, the output must clearly indicate a clean result.
5. Pay special attention to the following edge cases which have been problematic in the past:
[EDGE_CASE_EXAMPLES]

OUTPUT SCHEMA:
You must produce a single JSON object conforming to this schema:
{
  "violation_detected": boolean,
  "violations": [
    {
      "category": string, // e.g., "hate_speech", "harassment", "violent_content", "toxic_language"
      "severity": string, // e.g., "low", "medium", "high", "critical"
      "evidence_excerpt": string, // The exact text that triggered the violation
      "policy_reference": string, // The specific clause from the safety policy that was violated
      "rationale": string // A brief explanation of why this text constitutes a violation
    }
  ],
  "overall_severity": string | null, // The highest severity among all violations, or null if clean
  "requires_immediate_escalation": boolean // True if any violation is 'high' or 'critical'
}

To adapt this template, start by replacing [SAFETY_POLICY_DEFINITION] with your organization's specific content policy, including clear definitions for each harm category. The [EDGE_CASE_EXAMPLES] placeholder is critical for reducing false positives and negatives; populate it with 2-3 few-shot examples of content that was previously misclassified, showing the correct label and rationale. The [AI_OUTPUT_TEXT] variable should be injected by your application harness. Before deploying, validate that the model reliably produces valid JSON matching the schema by running a test suite of at least 50 labeled examples, including clean content, borderline cases, and clear violations. If the model fails to produce valid JSON, implement a retry loop with the error message fed back into the prompt, or escalate the raw output for human review as a format-failure case.

This prompt is designed to operate as a post-generation safety classifier in a streaming or batch content pipeline. It should be invoked after the primary model produces a candidate output but before that output is surfaced to a user, stored, or passed to downstream systems. The prompt expects the candidate text as [INPUT] and a defined [POLICY_CONTEXT] that maps your organization's content categories (toxicity, hate speech, harassment, violent content) to severity definitions. The output is a structured JSON violation record that your application can parse to decide whether to block, flag, or escalate the content.

Integration pattern: Wrap this prompt in a dedicated safety service that sits between your generation endpoint and your delivery layer. On each generation response, call the safety service with the raw output text. The service should: (1) assemble the prompt with the candidate text and your policy definitions, (2) call a fast, cost-efficient model (GPT-4o-mini or Claude Haiku work well for classification latency), (3) parse the JSON response and validate it against your expected schema, (4) log the full violation record with a trace ID linking back to the originating request, and (5) route based on severity. For severity: "critical" or severity: "high", block the output immediately and push to a human review queue. For severity: "medium", flag for asynchronous review but consider allowing delivery with a warning depending on your risk tolerance. For severity: "low" or severity: "none", log for audit and allow delivery.

Validation and retry logic: The prompt returns JSON, but model outputs can still be malformed. Implement a schema validator that checks for required fields (violation_detected, severity, category, evidence_excerpt, policy_reference). If parsing fails, retry once with a repair prompt that includes the raw output and the schema. If the retry also fails, escalate the content to human review with a schema_failure flag and include the raw model output for diagnosis. Do not silently allow content through when the safety classifier itself fails. Log every schema failure as a separate incident for prompt debugging.

Human review queue integration: When content is escalated, the structured violation record becomes the review packet. Include the evidence_excerpt (the specific span of text that triggered the violation), the category and severity, the policy_reference (which policy clause was violated), and a link to the full conversation context. Reviewers should be able to adjudicate with a single click: uphold the block, override and release, or reclassify the severity. Capture every adjudication decision and feed it back into your evaluation dataset to measure classifier accuracy and reviewer agreement over time.

Model choice and latency budget: This prompt is a synchronous gate in your content delivery path, so latency matters. Use a small, fast model for the classification call. If your primary generation model is already a large frontier model, you can also explore having it self-classify its own output in the same request by appending the safety prompt to the generation call, but this couples generation and safety and makes independent auditing harder. A separate classifier call adds ~200-500ms but gives you a clean audit boundary and lets you swap safety models independently.

What to avoid: Do not use this prompt as your only safety layer. It should be one component in a defense-in-depth strategy that includes input filtering, keyword blocklists, and periodic red-team testing. Do not treat the severity score as ground truth without human calibration—run regular evaluations against a golden dataset of labeled examples and track precision, recall, and false positive rate by category. Finally, do not log the full violation record without considering data retention policies for sensitive content; hash or truncate evidence excerpts if storage of toxic content creates compliance risk.