Toxicity Detection Guardrail Prompt for User Inputs

This prompt is a production-grade guardrail for platform safety teams who need to classify user-submitted text before it reaches any downstream AI model, database, or human agent. The job-to-be-done is not just to detect profanity, but to produce a structured, actionable toxicity assessment—including severity, category labels, and a recommended action (BLOCK, FLAG, or ALLOW)—that can be consumed by application logic. The ideal user is a trust and safety engineer or an AI product manager integrating this into a content pipeline where a single missed toxic input can create brand risk or user harm.

You should use this prompt when you need a pre-processing gate that operates on raw, unverified user strings. It is designed for high-volume, low-latency classification where the cost of a false negative (allowing a threat or slur through) is significantly higher than the cost of a false positive (flagging a benign message for review). The prompt is calibrated to handle adversarial inputs, including slang, intentional misspellings ('leetspeak'), and obfuscation, but it is not a replacement for a full human review queue for borderline cases. It is most effective when wired into a synchronous API call that blocks or routes the user's message before any generative AI step consumes it.

Do not use this prompt as a post-generation output filter for an LLM. It is optimized for short, unstructured user inputs, not for evaluating long-form AI-generated text for subtle bias or toxicity. Do not use it as the sole defense in regulated environments where a single toxic exposure is a compliance violation; always pair it with a human-in-the-loop (HITL) review step for messages scored near the decision threshold. Finally, avoid using this prompt on inputs that have already been sanitized or redacted, as the removal of key terms can mask the true intent and cause the classifier to miss adversarial content.

This template is the core instruction set for a toxicity detection guardrail. It is designed to be placed in the system prompt of a stateless classifier model, or as a pre-processing step before user input reaches your main application logic. The prompt instructs the model to act as a content safety classifier, analyzing the raw user text for a range of harmful categories. It uses square-bracket placeholders for the dynamic parts you must supply: the user's text, your specific policy definitions, the required JSON output schema, and any few-shot examples to calibrate behavior for edge cases like slang or adversarial misspellings.

textCopy
You are a content safety classifier. Your sole task is to analyze the provided user input for toxicity, harassment, hate speech, self-harm, violence, and sexual content. Do not engage with the content as a conversation. Do not generate a response to the user. Only produce the classification output.

## Safety Policy
[POLICY_DEFINITIONS]

## Input
User Input: "[USER_INPUT]"

## Classification Task
1. Analyze the user input against each category defined in the Safety Policy.
2. Determine a severity level for any detected violation: `low`, `medium`, `high`, or `critical`.
3. Recommend a single action: `allow`, `flag_for_review`, `block`, or `escalate`.
4. If the input is safe across all categories, the severity must be `none` and the action must be `allow`.

## Output Format
You must respond with a single, valid JSON object conforming to this exact schema. Do not include any text outside the JSON object.
[OUTPUT_SCHEMA]

## Calibration Examples
[FEW_SHOT_EXAMPLES]

## Constraints
- Do not explain your reasoning in the output.
- Do not modify the user input.
- Adversarial misspellings and obfuscated terms (e.g., 'sh00t', 'h8') must be treated as their intended harmful words.

To adapt this template, start by replacing [POLICY_DEFINITIONS] with your organization's specific content policy, defining each category with clear, actionable descriptions. The [OUTPUT_SCHEMA] placeholder should be replaced with a strict JSON schema, for example: { "toxicity_score": <float 0-1>, "categories": [<string>], "severity": <string>, "action": <string> }. The [FEW_SHOT_EXAMPLES] are critical for handling edge cases; provide at least 3-5 examples that demonstrate the boundary between allow and flag_for_review, and show how to handle reclaimed slurs or coded language. For high-stakes production environments, the escalate action should trigger an immediate human review queue, and the raw prompt input and output must be logged immutably for audit trails. Avoid the temptation to make the policy too broad, as over-classification will break the user experience and create a backlog of false positives for your review team.

The toxicity detection prompt is not a standalone chatbot; it is a synchronous classification step in a larger application pipeline. Wire it as a pre-processing guard before any user input reaches your primary model, retrieval system, or agent loop. The harness should call the prompt, parse the JSON output, and branch based on the recommended_action field. A typical flow: receive user input → call toxicity detection prompt → if action: block, return a canned safety response and log the incident; if action: flag, route to a human review queue; if action: pass, continue to the main workflow. Do not pass the raw toxicity output to the user. The prompt's output is internal metadata for routing and audit, not a user-facing message.

Validation and retry logic is critical because the prompt returns structured JSON. Implement a JSON schema validator that checks for required fields (toxicity_score, categories, severity, recommended_action, rationale) and correct types. If parsing fails, retry once with the same input and a stricter format instruction appended to the prompt. If the retry also fails, treat the input as action: flag and escalate to human review rather than silently passing potentially toxic content. Log the raw model response and parse error for debugging. For high-throughput systems, set a latency budget: if the toxicity check exceeds 500ms, fall back to a simpler keyword-based filter or pass the input with a warning flag, depending on your risk tolerance. Never block the user indefinitely on a guardrail timeout.

Model choice matters for both accuracy and cost. Use a fast, instruction-tuned model (e.g., GPT-4o-mini, Claude 3.5 Haiku, or a fine-tuned small classifier) rather than a large reasoning model. The task is classification, not generation, and latency is the enemy of guardrails. If you observe false negatives on adversarial misspellings or slang, augment the prompt with a few-shot example bank of known bypass attempts, or pair the prompt with a separate pattern-matching layer that catches obvious keyword obfuscation before the LLM call. Logging and observability are non-negotiable: log every toxicity check with the input hash, model response, parsed score, action taken, and latency. This audit trail is essential for tuning thresholds, defending moderation decisions, and detecting prompt drift when you change models or system instructions. Wire these logs into your existing trust and safety dashboard, not a separate silo.

Human review integration should be designed for the flag action path. When the prompt returns action: flag, place the input and the full toxicity response into a review queue with context about the user session. Do not show the reviewer only the score; show the rationale field so they can make an informed decision quickly. Track reviewer decisions (override to block, override to pass) and use that data to calibrate your severity thresholds over time. If your flag rate exceeds 5% of total traffic, revisit your threshold definitions or add a second-pass classifier to reduce reviewer burden. Finally, never use the toxicity prompt's output to train a public model or share with third parties without scrubbing user-input text and reviewing data-sharing agreements. The prompt processes raw user content, which may contain PII or sensitive material even if classified as toxic.