Disallowed Use Case Detection Prompt

This prompt is designed for onboarding and monitoring systems that must detect when users attempt to repurpose an AI product for prohibited use cases. Its job is to classify a user's conversation or request against a defined disallowed-use policy, producing a structured verdict with supporting evidence. The ideal user is a trust and safety engineer, a platform security lead, or a product manager responsible for preventing abuse of an AI application. You need a clear, written disallowed-use policy and access to user conversation history or a representative sample of user requests before this prompt can do useful work.

Use this prompt when you need an automated, auditable signal to flag accounts, trigger human review, or block access before harm occurs. It is not a replacement for a full safety classifier on every message; it is a focused detector for use-case-level abuse patterns. For example, if your policy prohibits using your AI for automated trading advice, this prompt should fire when a user's conversation reveals they are building a trading bot, not when they ask a single question about stock prices. Wire the output into an account flagging queue, a human review dashboard, or an automated access restriction system. Always log the verdict, evidence, and model confidence for auditability.

Do not use this prompt as a real-time message-by-message filter—it is too heavy and too slow for per-message safety classification. Do not use it without a written disallowed-use policy; the model cannot enforce rules that do not exist. Do not use it to make fully automated account termination decisions without human review, especially in regulated domains. If your product operates in finance, healthcare, or legal contexts, ensure that a qualified human reviews any enforcement action triggered by this prompt. The next section provides the copy-ready prompt template you can adapt to your specific policy.

This prompt template is designed to be dropped into an onboarding or monitoring system where you need to detect when a user is attempting to repurpose the AI for a prohibited use case. It forces the model to act as a strict policy classifier, not a conversational assistant. The output is a structured classification with direct quotes from the user's request as evidence, making the decision auditable and defensible. Before using this prompt, you must have a clearly defined disallowed-use policy document. The prompt will fail silently or produce inconsistent results if the policy is vague, contradictory, or missing explicit categories.

textCopy
You are a use-case compliance classifier. Your only job is to determine whether a user's request attempts to use the AI system for a disallowed purpose, as defined in the policy below.

## DISALLOWED-USE POLICY
[POLICY_DOCUMENT]

## USER REQUEST
[USER_INPUT]

## CLASSIFICATION TASK
Analyze the user request against every category in the disallowed-use policy. For each relevant policy category, determine if the request violates it. You must extract the exact text from the user request that serves as evidence for your determination.

## OUTPUT SCHEMA
Return a valid JSON object with this exact structure:
{
  "classification": "VIOLATION" | "NO_VIOLATION" | "AMBIGUOUS",
  "violations": [
    {
      "policy_category": "string (exact category name from policy)",
      "confidence": "HIGH" | "MEDIUM" | "LOW",
      "evidence": "string (direct quote from user request)",
      "rationale": "string (brief explanation linking evidence to policy)"
    }
  ],
  "adjacent_legitimate_use_cases": ["string (plausible legitimate uses that might look similar)"],
  "recommended_action": "BLOCK" | "FLAG_FOR_REVIEW" | "ALLOW"
}

## CONSTRAINTS
- If the request is clearly legitimate, return an empty violations array and classification "NO_VIOLATION".
- If the request is ambiguous but could be interpreted as a violation, set classification to "AMBIGUOUS" and include the potential violation with LOW confidence.
- Do not refuse to classify. Do not provide disclaimers. Do not engage with the user request as a conversation.
- Only quote the user's own words as evidence. Do not paraphrase or infer intent beyond what is explicitly stated.
- If multiple policy categories are violated, list each one separately.

To adapt this prompt for your system, replace [POLICY_DOCUMENT] with the full text of your disallowed-use policy, including category names, definitions, and examples. The policy must be explicit enough that a reasonable person could apply it consistently. Replace [USER_INPUT] with the raw user message or conversation transcript. If you are monitoring multi-turn conversations, include the full history in [USER_INPUT] to avoid missing context-dependent violations. The output schema is intentionally strict—validate the JSON structure in your application layer before acting on the classification. For high-risk domains, route all AMBIGUOUS classifications and LOW-confidence violations to human review. Never automate a BLOCK action without a human-in-the-loop fallback for edge cases.

The Disallowed Use Case Detection Prompt is designed to sit at the boundary of your AI product—either during user onboarding or as a continuous monitoring layer on conversation logs. It takes a user's stated intent or conversation transcript and classifies it against your published disallowed-use policy. The prompt returns a structured classification, a confidence score, and evidence from the input. This output should never be the final enforcement action on its own; it is a signal that feeds into a larger decision pipeline. Wire this prompt into an application by treating it as a classifier that triggers downstream workflows: allow, block, flag for review, or request clarification.

Integration pattern: Deploy this prompt behind an API endpoint that accepts [USER_INPUT] and [POLICY_DOCUMENT] as variables. The [POLICY_DOCUMENT] should be injected from a configuration store, not hardcoded, so policy updates don't require prompt redeployment. On the response side, parse the JSON output and validate the classification field against your allowed enum values (ALLOWED, DISALLOWED, UNCLEAR). If the model returns malformed JSON, use a retry with repair prompt that includes the raw output and a stricter format instruction. After three failed parse attempts, escalate to a human review queue with the raw model response attached. For high-traffic systems, add a pre-filter: use a lightweight embedding similarity check against known disallowed use cases to skip the LLM call for obviously benign inputs, reducing latency and cost.

Model choice and latency: For real-time onboarding flows, use a fast model (e.g., GPT-4o-mini, Claude Haiku) with a strict timeout of 2–3 seconds. If the prompt times out, default to a conservative UNCLEAR classification and route to manual review rather than blocking the user. For async batch monitoring of conversation logs, you can afford a more capable model and a longer timeout. Logging and audit trail: Log every classification with the evidence field, the confidence score, the model version, and the policy document hash. This creates an auditable record for compliance teams and enables offline analysis of classification drift when policies change. Human review integration: Any classification with confidence below 0.85 or classification equal to UNCLEAR should create a review task. The review UI should display the original user input, the model's classification, the cited evidence, and the relevant policy section side by side. Human reviewers can then confirm, overturn, or refine the classification, and their decisions become ground-truth data for future fine-tuning or few-shot example curation.

Testing and eval before launch: Before deploying, build a test harness that runs this prompt against three datasets: (1) a golden disallowed set with known policy violations where you expect DISALLOWED with high confidence; (2) a legitimate adjacent set with edge cases that look close to disallowed but should return ALLOWED (e.g., a security researcher asking about prompt injection for a bug bounty program); and (3) an adversarial set with obfuscated or indirect attempts to disguise disallowed use. Measure false-acceptance rate, false-refusal rate, and the proportion of UNCLEAR classifications. If the UNCLEAR rate exceeds 10%, refine the policy document or add few-shot examples to the prompt. What to avoid: Do not use this prompt as the sole gate for account suspension or legal action. Always keep a human in the loop for enforcement decisions. Do not log raw user input indefinitely without a data retention policy—the evidence field should be sufficient for audit purposes.