Policy Violation Triage Prompt for API Gateways

This prompt is designed for API platform operators and trust and safety engineers who need to classify user prompts against a defined Acceptable Use Policy (AUP) before they reach a generative model. It acts as a pre-invocation guardrail, producing a structured verdict that includes a violation category, a confidence score, and a recommended action: block, quarantine, or flag for review. The primary job-to-be-done is to create a deterministic, auditable triage step that prevents policy-violating content from consuming compute, generating unsafe outputs, or creating regulatory exposure. The ideal user is an engineer integrating this prompt into an API gateway or middleware layer, where a decision must be made in milliseconds to either forward the prompt to a model or divert it to a rejection handler.

This prompt is not a safety classifier for extreme harm categories like CSAM or self-harm, which require specialized, often multimodal detectors with higher precision mandates. Instead, it is a configurable policy enforcement layer for commercial API products. You should use it when you have a clearly documented AUP with categories like 'hate speech,' 'harassment,' 'adult content,' or 'prompt injection,' and you need a consistent, loggable decision for each request. Do not use this prompt when the policy itself is ambiguous or when the cost of a false positive (blocking a legitimate request) is unacceptable without a human review path. The required context includes the full text of your AUP, a defined output schema for the verdict, and a clear mapping of violation categories to automated actions.

Before implementing, define your operational thresholds. A high-confidence 'block' decision can be automated, but a 'quarantine' or 'flag' action should always route to a review queue. Monitor the false-positive rate closely during initial deployment, as an overzealous guardrail can degrade the user experience more than the violations it prevents. The next step is to copy the prompt template, populate the [AUP_POLICIES] and [OUTPUT_SCHEMA] placeholders with your specific rules, and run a batch of known-clean and known-violating prompts to calibrate the confidence score thresholds before enabling it in production.

This prompt template is designed to be deployed as a pre-invocation guard in your API gateway. It accepts the raw user input and your defined acceptable use policy, then returns a structured JSON verdict that your gateway can act on programmatically. The template uses square-bracket placeholders for all variable components—replace each with your specific policy text, risk tolerance, and output requirements before integrating it into your request pipeline.

textCopy
You are a policy enforcement classifier for an API gateway. Your job is to evaluate the user's input against the provided Acceptable Use Policy and return a structured verdict. Do not generate a response to the user. Do not answer the user's request. Only classify it.

## ACCEPTABLE USE POLICY
[POLICY_TEXT]

## USER INPUT
[USER_INPUT]

## CLASSIFICATION TASK
1. Determine if the user input violates any clause in the Acceptable Use Policy.
2. If a violation is found, identify the specific policy clause violated and the category of violation.
3. Assign a confidence score from 0.0 to 1.0 reflecting how certain you are in the classification.
4. Recommend an action: BLOCK, QUARANTINE, or FLAG.
   - BLOCK: Clear, high-confidence violation. Reject the request immediately.
   - QUARANTINE: Suspicious but not certain. Isolate for human review.
   - FLAG: Low-confidence or borderline. Allow but log for monitoring.
5. Provide a brief, factual reason for the decision. Do not speculate beyond the evidence in the input.

## OUTPUT SCHEMA
You must return ONLY a valid JSON object with this exact structure:
{
  "violation_detected": boolean,
  "violation_category": string | null,
  "policy_clause_reference": string | null,
  "confidence_score": number,
  "recommended_action": "BLOCK" | "QUARANTINE" | "FLAG",
  "reason": string
}

## CONSTRAINTS
- Do not include any text outside the JSON object.
- If no violation is detected, set violation_category and policy_clause_reference to null, confidence_score to 1.0, and recommended_action to "FLAG".
- The reason field must cite specific evidence from the user input when a violation is found.
- If the input is ambiguous, err on the side of lower confidence and recommend QUARANTINE or FLAG rather than BLOCK.
- Do not classify based on assumptions about the user's intent beyond what is explicitly stated.

To adapt this template, replace [POLICY_TEXT] with your organization's exact acceptable use policy clauses—include definitions, examples, and edge-case guidance. Replace [USER_INPUT] with the raw user prompt from your API request. The output schema is designed for direct parsing by your gateway middleware; map the recommended_action field to your request lifecycle logic (e.g., return HTTP 403 for BLOCK, enqueue for QUARANTINE, attach a warning header for FLAG). Before deploying, run this prompt against a golden dataset of known violations and benign inputs to calibrate your confidence thresholds and monitor false-positive rates.

This prompt is designed to sit at the ingress layer of your API gateway, acting as a pre-invocation classifier before any user prompt reaches a generative model. The implementation harness must treat this prompt as a synchronous decision point within the request lifecycle. The model call for triage should be made with a strict latency budget—typically under 500ms—using a fast, cost-optimized model (e.g., GPT-4o-mini, Claude 3 Haiku, or a fine-tuned small classifier). If the triage model times out or returns an unparseable response, the harness must default to a safe action: either block or quarantine, depending on your risk posture. Never fail open.

The harness must validate the structured output against a strict schema before acting on the verdict. Expect the model to return a JSON object with violation_category, confidence_score, and recommended_action fields. Implement a post-processing validator that checks: (1) violation_category is one of your predefined enums (e.g., hate_speech, self_harm, sexual_content, violence, none); (2) confidence_score is a float between 0.0 and 1.0; (3) recommended_action is exactly block, quarantine, or flag. If validation fails, log the raw output, increment a triage_parse_failure metric, and treat the request as block until a human reviews the failure pattern. For quarantine actions, route the request to an asynchronous review queue and return an immediate 202 Accepted to the client with a review ticket ID—do not block the client indefinitely.

Monitoring is the most critical part of this harness. Track three primary metrics: false-positive rate (legitimate requests incorrectly blocked or quarantined), false-negative rate (policy-violating requests that pass through), and triage latency p99. False positives require a human review pipeline where quarantined requests can be manually approved and added to an allowlist or used to refine the prompt's few-shot examples. False negatives require sampling passed requests and running them through a more capable (and more expensive) judge model offline to detect missed violations. Set up a dashboard that breaks down verdicts by violation_category and confidence_score band so you can spot drift—for example, a sudden spike in high-confidence hate_speech blocks might indicate a real attack or a prompt regression. Log every triage decision with the input hash, verdict, confidence, and model version for auditability.

When wiring this into your gateway code, keep the prompt template and its few-shot examples in a configuration store (not hardcoded) so you can update policy definitions without a full deployment. Use a circuit breaker: if the triage model's error rate exceeds 5% over a 60-second window, switch to a static rule-based fallback (e.g., keyword blocklist) and alert the on-call engineer. For high-throughput systems, consider batching triage requests where latency allows, but be aware that batching introduces head-of-line blocking risks. Finally, implement a feedback loop: when a human reviewer overturns a triage decision, store the corrected verdict as a new few-shot example for the next prompt revision. This closes the gap between policy intent and model behavior over time.