Terms of Service Violation Detection Prompt

This prompt is designed for trust and safety engineering teams who need to screen AI-generated content, user-facing outputs, or proposed automated actions against a platform's Terms of Service before those items are published, executed, or delivered to end users. It functions as a compliance gate: the model receives the full ToS text as grounding context along with the content to evaluate, and it returns a structured violation assessment that your application can act on programmatically. Use this prompt when you need clause-level traceability, severity classification, and a clear enforcement recommendation rather than a simple pass/fail flag.

The ideal user is an engineer or operator integrating this prompt into a content pipeline, agent workflow, or moderation queue. Required context includes the complete and current ToS text, the content or proposed action to evaluate, and a defined output schema that your downstream systems expect. The prompt works best when the ToS contains explicit, rule-like clauses that can be matched against specific content attributes. It is less reliable for vague policy language like 'objectionable content' or 'spirit of the rules' unless those terms are operationally defined in the provided context. Always include the full relevant ToS sections rather than summaries, as paraphrased policies introduce interpretation drift.

This prompt does not replace legal review for novel or high-severity cases, and it should not be used as the sole decision-maker for account-level enforcement actions such as permanent bans. It is a screening and triage tool, not a final adjudicator. When the model flags a high-severity violation or encounters ambiguous policy language, route the case to a human review queue with the full violation assessment attached. For low-severity, clear-cut violations with high model confidence, you may configure automated enforcement actions, but always log the assessment for audit. Never use this prompt to evaluate content where the ToS itself is disputed, incomplete, or subject to active legal interpretation without human oversight.

This prompt template is designed to be embedded directly into your trust and safety pipeline. It accepts the content or action to be screened, the relevant Terms of Service text as grounding context, and a defined output schema. The model's job is to act as a compliance gate: it must identify violations, cite the specific clause breached, classify severity, and recommend an enforcement action. The template uses square-bracket placeholders for all dynamic inputs—replace each before sending the prompt to the model. No placeholder should remain unresolved in production traffic.

textCopy
You are a trust and safety compliance reviewer. Your task is to evaluate the provided [CONTENT_OR_ACTION] against the [TERMS_OF_SERVICE_TEXT] and determine whether a violation exists.

# INPUT
[CONTENT_OR_ACTION]

# POLICY CONTEXT
[TERMS_OF_SERVICE_TEXT]

# OUTPUT SCHEMA
Return a single JSON object with the following structure:
{
  "violation_detected": boolean,
  "violations": [
    {
      "clause_reference": "string (exact section or clause identifier from the ToS)",
      "violation_summary": "string (one-sentence description of the violation)",
      "severity": "critical | high | medium | low",
      "recommended_action": "block | quarantine | flag_for_review | warn | allow",
      "rationale": "string (explanation linking the content to the specific clause)"
    }
  ],
  "overall_severity": "critical | high | medium | low | none",
  "overall_recommendation": "block | quarantine | flag_for_review | warn | allow",
  "uncertainty_notes": "string or null (describe any ambiguity, missing context, or borderline cases that warrant human review)"
}

# CONSTRAINTS
- Only flag violations where the [CONTENT_OR_ACTION] clearly matches a clause in the [TERMS_OF_SERVICE_TEXT].
- If no violation is detected, return an empty violations array, overall_severity of "none", and overall_recommendation of "allow".
- Do not invent clauses or policies not present in the provided ToS text.
- If the ToS text is ambiguous on a point, note this in uncertainty_notes and recommend "flag_for_review" rather than "block".
- For any severity of "critical" or "high", overall_recommendation must be "block" or "quarantine".
- Cite clause references exactly as they appear in the provided ToS text.

# RISK LEVEL
[RISK_LEVEL]

Adaptation guidance: Replace [CONTENT_OR_ACTION] with the raw text, generated output, or action description under review. Replace [TERMS_OF_SERVICE_TEXT] with the full, unredacted ToS document—do not summarize or truncate it, as clause-level citation accuracy depends on complete context. The [RISK_LEVEL] placeholder accepts a value like "high", "medium", or "low" to adjust the model's sensitivity; at high risk, the model should default to flagging borderline cases for human review. After receiving the output, validate the JSON structure programmatically before acting on the recommendation. For any overall_recommendation of "block" or "quarantine", route the decision to a human reviewer with the full violation payload attached. Never automate blocking actions without a human-in-the-loop checkpoint for critical or high-severity violations.

This prompt is not a standalone classifier; it is a gate in a larger enforcement pipeline. The implementation harness must treat the model's output as a recommendation, not a final enforcement action. The harness is responsible for providing the correct ToS text as grounding context, validating the output schema, enforcing severity thresholds, and routing violations to the appropriate enforcement queue or human review. A well-built harness prevents the model from hallucinating policy clauses, misclassifying severity, or taking automated action on high-severity violations without human approval.

Input assembly is the first critical step. The harness must retrieve the correct version of the platform's Terms of Service and inject it into the [TOS_TEXT] placeholder. This retrieval should be version-pinned and auditable; never rely on the model's parametric knowledge of a ToS. The [CONTENT_TO_CHECK] should be the raw text or structured representation of the user-generated content, proposed action, or AI-generated output under review. If the content is multimodal (images, video), extract a text description or transcript before passing it to this prompt. The harness should also inject a unique review_id and timestamp into the context for traceability. For high-throughput systems, consider batching multiple content items into a single request with a clear array structure in [CONTENT_TO_CHECK] and an explicit instruction to return an array of assessments, but monitor for attention dilution across long batches.

Output validation and enforcement routing form the core of the harness. The model returns a JSON object with violation_detected (boolean), violated_clauses (array of clause references), severity (enum: low, medium, high, critical), reasoning (string), and recommended_action (enum: allow, flag_for_review, block, escalate). The harness must validate that every violated_clause reference exists in the provided ToS text—hallucinated clause numbers are a common failure mode. Implement a post-processing check that searches for each cited clause reference in the source document; if a reference is not found, downgrade the confidence and flag the item for human review. Severity routing logic should be configurable: for example, low severity violations might be logged and allowed with a warning, medium routed to a review queue, and high or critical violations blocked pending human adjudication. Never automate enforcement on critical severity findings without human approval.

Retry and fallback logic must account for model failures and edge cases. If the model returns malformed JSON, use a repair prompt or a lighter-weight model to restructure the output before failing. If the model returns a severity value outside the defined enum, map it to flag_for_review and log the anomaly. If the model's reasoning field is empty or nonsensical, treat the assessment as unreliable and escalate. Implement a circuit breaker: if the violation rate suddenly spikes beyond a configurable threshold (e.g., >30% of items flagged in a 5-minute window), halt automated processing and alert the trust and safety team—this may indicate a prompt regression, a coordinated attack, or a ToS update that the model is misinterpreting.

Logging and audit trail requirements are non-negotiable for this workflow. Every assessment must be logged with the review_id, the full prompt (including the ToS version hash), the raw model output, the post-validation result, the enforcement decision, and the human reviewer's identity if manual review occurred. This audit trail serves three purposes: defending enforcement decisions during user appeals, detecting prompt drift over time, and providing evidence for regulatory inquiries. Store these logs immutably and retain them in line with your platform's data retention policy. For high-stakes platforms, consider running a second model (or the same model with a different temperature) as a shadow evaluator and comparing disagreement rates as a quality signal.

Model choice and latency considerations directly impact this harness. For real-time content screening (e.g., before a post is published), latency must be under 2 seconds, favoring smaller, faster models like Claude Haiku or GPT-4o-mini. For asynchronous batch review of existing content, larger models like Claude Sonnet or GPT-4o provide better clause-level reasoning at higher cost and latency. If your ToS document is long (>50 pages), use a RAG retrieval step to pull only the most relevant sections before passing them as [TOS_TEXT]; do not stuff the entire document into the context window and hope the model attends to the right clauses. Test your harness with adversarial content designed to probe the boundaries of your policies—content that is technically compliant but violates the spirit of the rules, or content that uses obfuscation to hide violations. These edge cases will reveal where your prompt and harness need reinforcement.