Multi-Modal Abuse Content Classification Prompt

This prompt is designed for trust and safety engineers at API platform operators who need to classify multimodal inputs for abusive, harassing, or violent content before those inputs reach downstream models or tools. It operates at the ingress layer, producing a structured classification that your application can use to block, quarantine, or route content for human review. The prompt handles text, images, audio, and video inputs, assigns severity tiers, and maps violations to modality-specific policy categories. Use this when you need a single classification gate that works across modalities and produces auditable, structured decisions.

Do not use this prompt as a standalone content moderation UI or as a replacement for human review in high-severity cases. It is a routing and triage component, not a final safety determination. The prompt is most effective when wired into an application harness that enforces validation on the output schema, logs every classification decision with the input hash and model version, and escalates severity: critical or confidence: low results to a human review queue. For regulated domains or user-facing moderation appeals, always pair this prompt with evidence grounding requirements and a human-in-the-loop approval step before taking irreversible action such as account suspension or content removal.

Before deploying, test against your specific abuse taxonomy and edge cases that are common in your domain. News imagery, medical documentation, and artistic content are frequent sources of false positives that can break legitimate workflows. Run eval suites that measure over-blocking rates on these categories alongside recall on known abuse patterns. If your platform serves user-generated content at scale, combine this classification prompt with hash-matching and reputation signals rather than relying on the model alone. Start with a shadow mode deployment where classifications are logged but not enforced, then gradually enable blocking as your eval results and human review feedback confirm the prompt's decisions align with your safety policies.

This prompt template is designed to be deployed as a system-level instruction for a multimodal model acting as a trust and safety classifier. It accepts a combined [INPUT] payload that may include text, image, audio, or video references, and produces a structured [OUTPUT_SCHEMA] with severity tiers and modality-specific policy violation categories. The template uses square-bracket placeholders for all configurable elements—replace each with your specific policy definitions, content taxonomy, and output format requirements before use. This prompt is intended for pre-invocation screening, not for moderating model outputs after generation.

textCopy
You are a trust and safety content classifier for an API platform. Your task is to analyze the provided multimodal input and determine whether it contains abusive, harassing, or violent content according to the defined policies. You must classify the content across all present modalities and assign an overall severity tier.

## POLICY DEFINITIONS
[POLICY_DEFINITIONS]

## INPUT
[INPUT]

## CLASSIFICATION RULES
1. Analyze each modality present in the input independently (text, image, audio, video).
2. For each modality, check against all relevant policy categories defined in [POLICY_CATEGORIES].
3. If a modality is not present or cannot be analyzed, mark it as "not_applicable" with a null confidence score.
4. When text and visual/audio modalities conflict, flag the conflict explicitly in the output.
5. Apply the exception rules defined in [EXCEPTION_RULES] for edge cases such as news imagery, medical documentation, or educational content.
6. Assign an overall severity tier based on the highest-severity violation found across all modalities.

## SEVERITY TIERS
[SEVERITY_TIERS]

## OUTPUT FORMAT
Return a valid JSON object matching this schema exactly:
[OUTPUT_SCHEMA]

## CONSTRAINTS
- Do not generate content that violates the policies you are checking for.
- Do not describe or repeat abusive content in your reasoning; reference policy category codes instead.
- If the input is empty or all modalities are unreadable, return an error object with reason "unprocessable_input".
- Confidence scores must be floats between 0.0 and 1.0.
- If confidence for any violation flag is below [CONFIDENCE_THRESHOLD], mark that flag as "uncertain" and include a note recommending human review.

## EXAMPLES
[FEW_SHOT_EXAMPLES]

To adapt this template for production, start by defining your [POLICY_DEFINITIONS] with precise, legal-reviewed language for each prohibited content category. Populate [POLICY_CATEGORIES] with modality-specific violation types—for example, "hate_symbols_imagery" for images, "threats_of_violence_text" for text, and "harassing_audio" for audio. Define [SEVERITY_TIERS] with clear escalation thresholds (e.g., "critical" for immediate blocking, "high" for quarantine, "low" for logging only). The [OUTPUT_SCHEMA] should include fields for per-modality flags, overall severity, confidence scores, conflict notes, and a human-review recommendation boolean. Replace [EXCEPTION_RULES] with your organization's specific carve-outs for legitimate content such as news reporting, medical documentation, or law enforcement evidence. The [FEW_SHOT_EXAMPLES] placeholder should be populated with 3–5 representative cases covering clear violations, edge cases, and non-violations to calibrate the model's behavior. Always test this prompt against a golden dataset of known abuse cases and benign edge cases before deployment, and ensure a human review queue is configured for any output flagged as "uncertain" or above a defined risk threshold.

This prompt is designed as the ingress guard in a multi-modal API platform. It must execute before any generative model invocation, tool use, or user-facing response. The harness should call this prompt synchronously in the request path, blocking downstream processing until a classification decision is returned. A common integration point is a reverse proxy, API gateway middleware, or a dedicated safety service that receives the raw user request payload—including text, base64-encoded images, audio file URLs, or video frame samples—and returns a structured verdict. The output of this prompt determines whether the request proceeds, is quarantined for review, or is rejected with a policy-specific error code.

Validation and retry logic is critical because a malformed classification output is a safety bypass. The harness must validate the JSON response against a strict schema: violation_detected (boolean), severity (enum of none, low, medium, high, critical), categories (array of strings from a closed taxonomy like CSAM, graphic_violence, hate_speech, self_harm, sexual_content, harassment), confidence (float 0-1), flagged_modalities (array of strings), and rationale (string). If validation fails, retry once with a repair prompt that includes the schema and the raw malformed output. If the retry also fails, escalate to a human review queue and block the request. For critical severity or CSAM category detections, bypass retries entirely and immediately quarantine the content with an immutable audit log entry.

Model choice and latency are primary constraints. This prompt should run on a fast, cost-efficient model (e.g., GPT-4o-mini, Claude Haiku, or Gemini Flash) because it gates every request. For text-only inputs, consider a lightweight text classification model as a pre-filter before invoking the multi-modal prompt. The harness should enforce a strict latency budget—typically 500-1000ms—and circuit-break to a conservative 'block' decision if the safety model times out. Logging must capture the full prompt input, raw model output, validated classification, and routing decision in a structured, queryable format. Do not log the raw user content if it contains CSAM or is otherwise illegal to store; instead, log a cryptographic hash and the classification metadata. Human review is mandatory for medium and high severity detections where confidence is below 0.85, and for all critical detections regardless of confidence. The harness should enqueue these cases with the rationale, flagged modalities, and a secure reference to the quarantined content.

Avoid over-blocking by implementing a feedback loop. When human reviewers overturn a classification, log the case as a false positive and use it to refine your evaluation dataset. Run weekly regression tests against a golden set of edge cases: news imagery depicting violence, medical documentation, artistic nudity, and screenshots of hate speech being reported by users. The harness should expose a dry-run mode that logs classifications without blocking, enabling you to measure the false-positive rate before enforcing policy in production. Never deploy this prompt without a kill switch that can disable blocking and revert to log-only mode if the false-positive rate spikes.