Content Policy Rule Check Prompt Template

This prompt is designed for trust and safety engineers who need to programmatically validate LLM-generated content before it reaches end users. It acts as a second-pass filter, taking an AI's output and a set of configurable content policies, then returning a structured violation report. Use this when you have a defined content policy and need a consistent, automated gate that can be integrated into a CI/CD pipeline or a real-time moderation harness. This is not a replacement for human review in high-severity or legally ambiguous cases, but it provides a fast, auditable first line of defense.

Deploy this prompt when you have a clear, written content policy and need to enforce it at scale across thousands of AI-generated responses. It is ideal for applications where a primary model generates content—such as a customer-facing chatbot, an automated report writer, or a code documentation tool—and you need a secondary check to catch policy violations like prohibited topics, toxic language, or missing disclosures. The prompt's structured output, which includes a violation status, severity level, and specific rule citations, makes it directly consumable by downstream systems. For example, an API response with violation_found: true and severity: high can automatically trigger a block, a retry with a different prompt, or an escalation to a human review queue. Avoid using this prompt as the sole decision-maker for complex, context-dependent policies like harassment or self-harm, where human judgment is critical. It is also not a substitute for a real-time toxicity classifier like a fine-tuned BERT model when latency is measured in milliseconds; an LLM-based check adds latency better suited for asynchronous or near-real-time workflows.

Before integrating this prompt, you must define your policies as a set of discrete, machine-readable rules. A vague policy like 'be helpful and harmless' will produce inconsistent results. Instead, define rules such as 'The output must not contain instructions for creating weapons' or 'Financial advice must include a disclaimer.' Each rule should have a unique ID, a clear description, and a severity level (e.g., low, medium, high, critical). You will pass these rules into the prompt's [POLICY_RULES] placeholder as a structured list. To get started, take your existing content policy document and decompose it into individual, testable assertions. Then, run a batch of known violating and non-violating outputs through the prompt to calibrate its accuracy. The next step is to wire the prompt's structured output into your application's decision logic, which is covered in the implementation harness section.

This template is designed to be pasted directly into your system prompt or API call. It instructs the model to act as a content policy auditor, comparing a provided [OUTPUT] against a set of [POLICY_RULES]. The prompt forces structured reasoning by requiring the model to cite the specific rule violated, the offending text, and a severity classification. Replace every square-bracket placeholder with your specific inputs before use. For high-risk domains, always route CRITICAL severity findings to a human review queue rather than taking automated action.

textCopy
You are a content policy auditor. Your task is to evaluate the provided [OUTPUT] against a set of [POLICY_RULES]. You must produce a structured violation report in JSON format.

[POLICY_RULES]
{ "rules": [ { "id": "string", "description": "string", "category": "PROHIBITED_CONTENT | TONE_VIOLATION | DISCLOSURE_REQUIREMENT", "severity_threshold": "LOW | MEDIUM | HIGH | CRITICAL" } ] }

[OUTPUT]
"The full text of the LLM output to evaluate."

[OUTPUT_SCHEMA]
{
  "violations": [
    {
      "rule_id": "string (must match an id from POLICY_RULES)",
      "category": "PROHIBITED_CONTENT | TONE_VIOLATION | DISCLOSURE_REQUIREMENT",
      "severity": "LOW | MEDIUM | HIGH | CRITICAL",
      "offending_text": "Exact substring from OUTPUT that triggered the violation",
      "explanation": "Brief explanation of why this text violates the rule"
    }
  ],
  "summary": {
    "total_violations": "integer",
    "highest_severity": "LOW | MEDIUM | HIGH | CRITICAL | NONE",
    "pass": "boolean (true if no violations found)"
  }
}

[CONSTRAINTS]
1. Only flag text that clearly violates a specific rule. Do not flag borderline cases.
2. The `offending_text` field must contain an exact, copy-pasted substring from [OUTPUT].
3. If no violations are found, return an empty `violations` array, set `highest_severity` to "NONE", and set `pass` to true.
4. Do not include any text outside the JSON object.

To adapt this template, replace [POLICY_RULES] with your actual policy document serialized as a JSON object. The rules array should contain every policy you want enforced. For [OUTPUT], wire in the LLM-generated text you are auditing. The [OUTPUT_SCHEMA] and [CONSTRAINTS] sections are designed to be stable; modify them only if you need to add fields like a confidence score or a suggested remediation. After pasting, test the prompt against a golden dataset of known-clean and known-violating outputs to calibrate your severity thresholds before production use.

The Content Policy Rule Check prompt is designed to be a deterministic gate in a content moderation pipeline, not a conversational assistant. It should be deployed as a stateless function call within a broader moderation service. The typical integration pattern involves receiving a piece of content (a user message, a generated LLM output, or an uploaded document), injecting it into the prompt's [CONTENT] placeholder alongside your active [POLICY_RULES], and parsing the structured JSON violation report. Because this prompt makes high-stakes trust and safety decisions, the implementation harness must prioritize auditability, low latency, and a clear escalation path for ambiguous cases.

To harden this in production, wrap the prompt call in a validation layer that checks the structural integrity of the JSON output before acting on it. Use a JSON Schema validator to confirm the presence of required fields like violations, severity, and violation_type. If the model returns malformed JSON, implement a single retry with a repair-focused prompt that provides the raw output and the validation error message. If the retry also fails, the system should default to a safe state—flagging the content for human review rather than silently passing it. Log every decision, including the raw prompt, the full model response, the validation result, and the final action taken, to create an audit trail for policy tuning and incident review.

Model choice matters here. Smaller, faster models may miss subtle policy violations like veiled harassment or coded language, while the largest models may introduce unacceptable latency for real-time chat applications. A common pattern is to use a fast, cheap model (like GPT-4o-mini or Claude Haiku) for clear-cut cases and escalate to a more capable judge model (like GPT-4o or Claude Opus) when the primary model's confidence is low or when it flags a borderline severity score. The [RISK_LEVEL] placeholder in the prompt template should be dynamically set based on the user's context (e.g., high for a child-directed application, medium for a professional forum). Never hardcode this value; pull it from a feature flag or a user-segment configuration service to allow for real-time policy adjustments without a code deploy.

Finally, build a human-in-the-loop queue for all high severity violations and any content where the model's confidence score falls below a configurable threshold (e.g., 0.85). This queue should present the reviewer with the original content, the model's violation report, and a simple interface to uphold or overturn the decision. Use these human decisions to build a golden dataset for regression testing the prompt over time. Avoid the temptation to automatically delete or block content based solely on the model's output without a human review path for edge cases; a false positive that silences a legitimate user is a product failure, not just a model error.