Policy Violation Triage Prompt for Streaming Content

This prompt is designed for content moderation engineers and trust and safety teams who need to classify a firehose of user-generated content against a defined policy framework. It is built for streaming ingestion pipelines where each piece of content must be evaluated for policy violations, assigned a severity level, and routed to the correct action queue (allow, block, quarantine, or escalate for human review). The core job-to-be-done is producing structured, auditable triage decisions that reference specific policy rules, not performing open-ended content analysis or generative rewriting. The ideal user is an engineer integrating this classification layer into a broader moderation system, where the prompt acts as the decision engine that feeds downstream enforcement, logging, and review workflows.

Use this prompt when you have already defined your policy categories, severity tiers, and action mappings. The prompt assumes the existence of a policy framework with explicit rules—such as 'hate speech,' 'harassment,' 'violent content,' or 'spam'—each with clear definitions and severity levels like 'low,' 'medium,' 'high,' and 'critical.' It also assumes you have predetermined action mappings, for example: low-severity spam is allowed with a flag, high-severity harassment is blocked, and critical violent content is escalated for human review. The prompt does not replace a full policy engine; it is the classification layer that feeds one. You should not use this prompt when you need to generate policy definitions, when you lack a clear action matrix, or when the content requires subjective, context-heavy judgment without defined rules. It is also unsuitable for low-volume, ad-hoc moderation where a human reviewer can make nuanced decisions without structured automation.

In production, wire this prompt into your streaming pipeline with validation, retries, and human review for high-risk decisions. The output must be validated against your expected schema before any action is taken—never trust an unvalidated classification to trigger a block or quarantine. For high-severity or low-confidence decisions, route to a human review queue with the full triage payload, including the policy rule reference and the content snippet. Monitor for common failure modes: policy conflict where content matches multiple categories, edge cases where severity is ambiguous, and adversarial content designed to evade classification. Before shipping, run this prompt against a golden dataset of known violations and benign content to measure precision, recall, and false positive rates. If your false positive rate on high-severity classifications exceeds your operational tolerance, adjust the prompt's severity criteria or add a confidence threshold that forces human review.

This prompt template is designed to be pasted directly into your system instructions or user message template for a policy violation triage workflow. It expects a content payload and a set of policy definitions, and it produces a structured triage decision that includes the violated policy, severity level, required action, and a reference to the specific policy rule. The template uses square-bracket placeholders that you replace with your actual policy taxonomy, content input, and output schema requirements before deployment.

textCopy
You are a content policy triage classifier for a streaming content moderation pipeline. Your job is to evaluate the provided content against the defined policy categories and produce a structured triage decision.

## INPUT
[CONTENT_PAYLOAD]

## POLICY DEFINITIONS
[POLICY_DEFINITIONS]

## INSTRUCTIONS
1. Analyze the content against each policy category defined above.
2. Identify all policy violations present in the content.
3. For each violation, determine:
   - The specific policy rule violated (use the exact rule identifier from the policy definitions)
   - The severity level: [SEVERITY_LEVELS]
   - The required action: [ACTION_OPTIONS]
   - A brief evidence excerpt from the content that supports the violation finding (max 200 characters)
4. If multiple policies are violated, list all violations ordered by severity (highest first).
5. If no policies are violated, return an empty violations array with a `no_violation` status.
6. If the content is ambiguous or requires human judgment, set the action to `ESCALATE` and include a `review_note` explaining what is unclear.

## OUTPUT SCHEMA
Return a valid JSON object matching this structure:
{
  "decision_id": "string (unique identifier for this triage decision)",
  "status": "violation_found | no_violation | uncertain",
  "violations": [
    {
      "policy_rule_id": "string (exact identifier from policy definitions)",
      "severity": "string (one of the defined severity levels)",
      "action": "string (one of the defined action options)",
      "evidence": "string (excerpt from content, max 200 chars)",
      "confidence": "number (0.0 to 1.0)"
    }
  ],
  "review_note": "string | null (required when status is uncertain or action is ESCALATE)",
  "processing_timestamp": "string (ISO 8601)"
}

## CONSTRAINTS
[CONSTRAINTS]

## EXAMPLES
[FEW_SHOT_EXAMPLES]

To adapt this template for your production environment, replace the square-bracket placeholders with your actual values. [POLICY_DEFINITIONS] should contain your complete policy taxonomy with rule identifiers, descriptions, and examples of violating content. [SEVERITY_LEVELS] should list your severity tiers such as critical, high, medium, low. [ACTION_OPTIONS] should define the actions your pipeline can take, such as BLOCK, QUARANTINE, FLAG_FOR_REVIEW, ESCALATE, or ALLOW. [CONSTRAINTS] should include any additional rules like latency budgets, batch processing requirements, or jurisdiction-specific handling. [FEW_SHOT_EXAMPLES] should provide 3-5 labeled examples covering clear violations, edge cases, and no-violation scenarios to calibrate the model's judgment. Always test this prompt with a golden dataset of known violations and non-violations before deploying to production, and implement a validation layer that checks the output JSON against the expected schema before routing decisions are executed.

This prompt is designed to be a single stage within a larger streaming content moderation pipeline. It should not be deployed as a standalone, synchronous HTTP endpoint where a client waits for a full response. Instead, integrate it into an event-driven architecture using a message broker like Kafka, Kinesis, or Pub/Sub. The prompt expects a pre-hydrated [INPUT] containing the content to be reviewed and a [POLICY_DOCUMENT] injected as context. The output is a structured JSON triage decision that downstream services—such as a quarantine service, a human review queue, or a content-blocking proxy—can act on without further parsing.

To wire this into an application, wrap the model call in a lightweight worker service. This service must perform several critical functions: Input Validation to ensure the [INPUT] and [POLICY_DOCUMENT] placeholders are not empty and do not exceed the model's context window; Structured Output Validation to parse the JSON response and confirm it matches the [OUTPUT_SCHEMA], specifically checking that violation_category is a valid enum and confidence_score is a float between 0.0 and 1.0; and Error Handling with Retries for transient model API failures. For high-risk decisions, the harness must implement a hard rule: if the model's confidence_score falls within a configurable 'uncertainty band' (e.g., 0.4 to 0.6), or if the required_action is ESCALATE, the record must be automatically routed to a human review queue, bypassing any automated enforcement. Log the raw prompt, the model's full response, and the final routing decision for every single record to create an audit trail for policy tuning and debugging.

Model choice is a critical performance lever in this streaming context. For high-throughput, cost-sensitive pipelines, use a fast, smaller model like Claude 3 Haiku or GPT-4o-mini for the initial triage. Reserve a more powerful model, such as GPT-4o or Claude 3.5 Sonnet, as a secondary judge only for records flagged as UNCERTAIN or HIGH_SEVERITY by the primary model. This two-tier routing pattern optimizes cost and latency. Avoid using the same prompt for batch processing without modification; streaming requires stateless workers that can scale horizontally. Ensure your worker's concurrency model does not hold up a partition while waiting for a long model inference. The next step is to build a monitoring dashboard that tracks the rate of each violation_category, the distribution of confidence_score, and the ratio of automated actions to human escalations to detect drift in either the content stream or model behavior.