Policy Violation Escalation Prompt for User-Generated Content

This prompt is designed for content moderation platform engineers who need to classify user-generated content against a configurable policy taxonomy and produce a structured escalation decision. It is not a general-purpose content filter. Use it when you need an explicit, auditable link between a piece of content, the specific policy clause it violates, a calibrated severity level, and a recommended action that a downstream review queue or automated system can consume. The prompt assumes you have a defined policy taxonomy with clause identifiers, severity definitions, and action mappings. If your policy is still evolving or you lack calibration data, start with a narrower scope and fewer severity tiers to avoid inconsistent outputs that undermine trust in the escalation pipeline.

This prompt belongs in a human-in-the-loop pipeline where false positives create unnecessary review work and false negatives create brand safety or regulatory exposure. It works best when paired with a review queue that can accept structured escalation payloads and when your moderation team has clear service-level agreements for each severity tier. Before deploying, you should calibrate the prompt against a labeled dataset of at least 200 examples spanning clear violations, borderline content, and benign edge cases. Measure both false-positive rate (content incorrectly escalated) and false-negative rate (violations missed). If either rate exceeds your operational tolerance, refine the policy descriptions, add few-shot examples, or narrow the taxonomy before expanding scope.

Do not use this prompt when you lack a stable policy taxonomy, when severity definitions are ambiguous or overlapping, or when the cost of human review is so low that a simple keyword filter would suffice. Avoid deploying it on content types or languages not represented in your calibration data—the model may apply policy concepts inconsistently across unseen formats. If your use case involves real-time streaming content at high throughput, pair this prompt with a lightweight pre-filter to reduce latency and cost before invoking the full classification. Finally, never treat the model's output as a final enforcement action without human review for high-severity classifications; the prompt is a triage tool, not a replacement for moderator judgment.

This prompt template is designed to be pasted directly into your orchestration layer, whether you're using LangChain, a custom API wrapper, or a direct model call. It expects two primary inputs: a policy document that defines your content rules and the user-generated content that needs evaluation. The prompt forces the model to cite specific policy clauses rather than making vague judgments, which makes escalation decisions auditable and defensible. Before deploying, you should calibrate this prompt against a golden dataset of borderline content examples to establish appropriate severity thresholds for your domain.

textCopy
You are a content policy enforcement classifier. Your task is to evaluate user-generated content against a provided policy document and produce a structured escalation decision.

## POLICY DOCUMENT
[POLICY_DOCUMENT]

## USER-GENERATED CONTENT
[CONTENT]

## INSTRUCTIONS
1. Read the policy document carefully. Each policy clause has an identifier, description, severity level, and examples of violations.
2. Analyze the user-generated content for any violations of the policy clauses.
3. If no violations are found, output a clean decision with no escalation.
4. If violations are found, identify the specific clause(s) violated and assess severity.
5. Consider the context and intent of the content. Do not flag content that merely discusses policy topics without violating them.
6. For borderline cases, prefer flagging with lower severity rather than missing a violation.

## OUTPUT FORMAT
Produce a JSON object with this exact structure:
{
  "escalation_required": boolean,
  "violations": [
    {
      "clause_id": "string (from policy document)",
      "clause_description": "string (exact text from policy)",
      "severity": "low|medium|high|critical",
      "evidence_excerpt": "string (the specific text that violates the policy)",
      "explanation": "string (why this excerpt violates the clause)"
    }
  ],
  "recommended_action": "approve|flag_for_review|block|escalate_immediately",
  "confidence": "number between 0 and 1",
  "reasoning_summary": "string (brief explanation of the overall decision)"
}

## CONSTRAINTS
- Only flag content that clearly violates a stated policy clause.
- Do not invent policy clauses that are not in the provided document.
- If multiple clauses are violated, list all of them.
- Set confidence below 0.7 if the violation is borderline or ambiguous.
- For content that is clearly safe, set confidence above 0.95.

To adapt this template for your platform, replace [POLICY_DOCUMENT] with your actual policy text, which should include clause identifiers, descriptions, severity levels, and examples. The [CONTENT] placeholder should receive the raw user-generated text from your ingestion pipeline. The output schema is designed to be machine-readable for downstream routing logic: use the recommended_action field to trigger automated workflows (approve, block) or human review queues (flag_for_review, escalate_immediately). The confidence field enables threshold-based routing where low-confidence decisions can be sent to a secondary reviewer or a more capable model. For high-risk domains, consider adding a [RISK_LEVEL] parameter that adjusts the sensitivity of the classification, and always log the full prompt and response for audit trails. Before production deployment, run this prompt against at least 50 hand-labeled examples spanning clear violations, clear safe content, and borderline cases to calibrate your severity thresholds and confidence cutoffs.

After integrating this prompt into your pipeline, the next step is to build validation logic around the output. Validate that the JSON parses correctly, that clause_id values match your policy document, and that severity levels are within your defined set. For any response where confidence falls below 0.7 or where recommended_action is escalate_immediately, route to a human review queue with the full prompt context attached. Avoid the common failure mode of deploying this prompt without calibration: a policy document that is too vague will produce inconsistent decisions, while one that is too rigid will generate false positives on benign content. Start with a focused policy covering your highest-risk categories and expand iteratively based on review outcomes.

This prompt is designed as a single decision point within a broader content moderation pipeline. It should not operate in isolation. The typical integration pattern places it after content ingestion and before any automated action (publish, block, quarantine). The prompt receives a pre-normalized content payload and a configurable policy taxonomy, and it returns a structured escalation decision. The application layer is responsible for enforcing the decision: if the output indicates escalation, the content is routed to a human review queue; if it indicates no violation, the content proceeds to the next pipeline stage. Never allow the model's output to directly trigger a destructive action (permanent ban, content deletion) without human confirmation for high-severity violations.

Validation and Retry Logic: The prompt's structured output must be validated before the decision is trusted. Implement a JSON schema validator that checks for the presence of required fields (violation_detected, escalation_required, policy_clause, severity, recommended_action). If validation fails, retry the prompt call once with the same input and an added instruction: 'Your previous output was invalid. You must return valid JSON matching the schema exactly.' If the retry also fails, log the failure and escalate the content to a human review queue with a validation_failure flag. Do not retry more than twice to avoid latency spikes. For high-throughput systems, consider a circuit breaker that temporarily quarantines all content if the validation failure rate exceeds a configurable threshold (e.g., 5% over a 1-minute window).

Model Choice and Tool Use: This prompt is optimized for frontier models with strong instruction-following and structured output capabilities (GPT-4o, Claude 3.5 Sonnet). Avoid using smaller or older models for this task, as they are more likely to miss subtle policy violations or produce malformed JSON. This prompt does not require external tools or retrieval-augmented generation (RAG) because the policy taxonomy is provided inline. However, if your policy taxonomy is large and changes frequently, consider storing it in a vector database and retrieving the most relevant policy clauses based on the content before assembling the prompt. This reduces token usage and keeps the model focused on the applicable rules. Log every prompt call with a unique content_id, the model version, the full prompt payload, the raw model output, the validation result, and the final escalation decision. This audit trail is essential for debugging false positives, tuning policy thresholds, and demonstrating moderation consistency to regulators or internal review boards.

Human Review Integration: When the prompt returns escalation_required: true, the application must create a review task in your moderation queue. The task payload should include the original content, the model's full output (policy clause violated, severity, reasoning), and a link to the prompt trace log. Design the review interface to allow moderators to confirm, override, or mark the decision as a false positive. Track override rates by policy clause and severity level. A rising override rate for a specific clause signals that the prompt's calibration has drifted and requires adjustment—either by updating the policy description in the taxonomy or by adding few-shot examples that clarify the boundary. For high-severity violations (e.g., severity: critical), always require a second human reviewer for confirmation before taking action. This dual-review gate prevents single-point errors from causing irreversible harm.