Prohibited Use-Case Detection Prompt

This prompt is designed for platform operators and trust and safety engineers who need to enforce a configurable Acceptable Use Policy (AUP) or Terms of Service (ToS) at the very edge of an AI system. Its core job is to act as a programmable gatekeeper, classifying user inputs against a defined list of prohibited use cases—such as surveillance, disinformation, or automated decisions without human review—before they can consume compute, invoke tools, or reach a generative model. The ideal user is someone responsible for platform integrity who needs an auditable, deterministic, and non-negotiable enforcement layer, not a general-purpose safety classifier. You should use this prompt when you have a specific, documented policy document to inject as context and need a structured verdict that cites evidence directly from the user's input, making every block or flag decision defensible to auditors and internal stakeholders.

This prompt is not a replacement for a dedicated safety boundary classifier that detects content harm like violence or self-harm. It is purpose-built for use-case prohibition, meaning it looks at the intended application of a request, not its toxic content. For example, a user asking for help drafting a marketing email would pass a safety check but should be flagged by this prompt if your policy prohibits 'automated generation of deceptive commercial content.' Deploy this prompt at the ingress layer of an AI platform, inside an API gateway, or as a pre-processing step in any agent or assistant pipeline. It assumes you have a well-defined policy document to inject as the [POLICY_CONTEXT]. Without a clear, specific policy, the prompt will produce inconsistent and ungrounded results. The output is a structured JSON verdict containing a violation boolean, a category, a confidence score, and direct quotes from the user's input as evidence, which you can then use to programmatically block, quarantine, or flag the request.

Before implementing, understand the primary failure modes. The most common is a false negative, where a subtly phrased prohibited request evades detection. This often happens when the policy context is too vague or when the user's prompt is adversarial. To mitigate this, you must pair this prompt with a robust evaluation set that includes paraphrased and obfuscated versions of prohibited requests. A secondary failure mode is a false positive, where a legitimate request is incorrectly blocked, frustrating users. This typically occurs when policy categories are overly broad. To address this, you should implement a human review queue for high-confidence blocks or for requests from high-value accounts, allowing you to refine the policy and the prompt's instructions over time. The next step is to integrate this prompt into your request pipeline, ensuring the structured output is parsed and acted upon before any model inference begins.

This template is designed to be placed directly into your system instructions or as the core of a classification microservice. It forces the model to act as a strict policy enforcement layer, comparing the user's request against a defined list of prohibited use cases. The prompt is structured to require evidence from the input itself, reducing the risk of subjective or over-generalized blocking. Before copying, you must define your specific policy categories, provide few-shot examples that illustrate edge cases, and decide on the exact output schema your downstream application expects.

textCopy
You are a policy enforcement classifier. Your only job is to analyze the user's request against the provided [POLICY_DOCUMENT] and determine if it constitutes a prohibited use case.

## Classification Rules
1.  Compare the user's request to each prohibited category in the [POLICY_DOCUMENT].
2.  A request is a violation only if it explicitly describes or instructs a prohibited activity. Do not infer intent.
3.  For every violation you identify, you MUST cite the exact phrase from the user's request that matches the policy definition. If you cannot cite evidence, it is not a violation.
4.  If the request is ambiguous, classify it as `AMBIGUOUS` and explain what clarification is needed.

## Input Data
<policy_document>
[POLICY_DOCUMENT]
</policy_document>

<user_request>
[USER_INPUT]
</user_request>

## Output Schema
Respond exclusively in JSON format matching this schema:
{
  "verdict": "VIOLATION" | "NO_VIOLATION" | "AMBIGUOUS",
  "violations": [
    {
      "category": "string, from [POLICY_CATEGORIES]",
      "confidence": "HIGH" | "MEDIUM" | "LOW",
      "evidence": "verbatim quote from user_request",
      "reasoning": "brief explanation linking the evidence to the policy category"
    }
  ],
  "clarification_required": "string, only if verdict is AMBIGUOUS"
}

## Examples
[FEW_SHOT_EXAMPLES]

## Constraints
- Do not answer the user's request.
- Do not add commentary outside the JSON object.
- If no policy categories match, the verdict must be `NO_VIOLATION`.

To adapt this template, start by replacing [POLICY_DOCUMENT] with your actual terms of service or acceptable use policy, condensed into clear, machine-readable categories. The [POLICY_CATEGORIES] placeholder in the schema should be a list of strings representing the exact names of your prohibited categories, such as ["surveillance", "disinformation", "automated_decision_without_review"]. The [FEW_SHOT_EXAMPLES] section is critical for handling adversarial inputs; include examples where a user uses coded language, hypotheticals, or role-playing to mask a prohibited request, and show the model correctly extracting the evidence and classifying it as a VIOLATION. Also include a near-miss example that is classified as NO_VIOLATION to prevent over-blocking.

After integrating the prompt, you must build a validation layer in your application code. Do not trust the raw string from the model. Parse the JSON and validate it against the schema. If the model fails to produce valid JSON after a configured number of retries, your application should default to a safe action, such as routing the request for human review. For high-stakes enforcement, log every classification decision, including the evidence field, to create an audit trail that allows policy teams to review and calibrate the classifier's performance without needing to replay raw user data.

Integrating the Prohibited Use-Case Detection Prompt into a live system requires treating it as a deterministic classification step within a broader policy enforcement pipeline. The prompt should be invoked before any generative model processes the user's full request, ideally at the API gateway or ingress layer. This minimizes wasted compute on violating requests and prevents the downstream model from generating content that could be logged, cached, or returned before a violation is caught. The harness must accept the raw user input and a structured representation of your platform's Acceptable Use Policy (AUP) or Terms of Service as the [POLICY_DOCUMENT] variable. The output must be a machine-readable JSON object containing the violation verdict, the specific policy clause cited, and the exact substring of the user input that triggered the detection.

The core implementation loop should follow a validate-parse-escalate pattern. First, validate that the model returned syntactically correct JSON matching your expected [OUTPUT_SCHEMA]. If parsing fails, implement a single retry with a stricter prompt variant that includes the raw model output and a clear error message. If the retry also fails, log the raw response and escalate the input to a human review queue rather than silently passing it. For latency-sensitive applications, deploy a smaller, fine-tuned classifier model for the initial detection pass and use the general-purpose LLM prompt only for ambiguous cases where the confidence score falls within a configurable 'gray zone' (e.g., 0.4 to 0.7). All decisions—block, quarantine, or allow—must be logged with the model version, prompt template hash, input hash, violation category, confidence score, and the cited evidence for auditability.

Adversarial robustness is the primary failure mode to instrument against. Log the rate of inputs that receive a low confidence score but are submitted repeatedly with minor character substitutions or rephrasing, as this is a common evasion technique. Implement a circuit breaker that temporarily blocks a user or session if the violation rate or the rate of 'gray zone' inputs exceeds a threshold within a short time window. Crucially, never expose the raw violation detection prompt or its specific policy clauses in client-side code, as this provides attackers with a blueprint for crafting evasions. The next step after building this harness is to run a red-team exercise using the evaluation criteria defined in the Prompt QA section, specifically testing for obfuscation, role-playing, and multi-turn grooming attacks, before deploying the prompt to production traffic.