Abuse Report Summarization for Trust and Safety Teams Prompt

This prompt is designed for trust and safety operations leads and engineers who need to synthesize multiple incoming signals—automated detection flags, user reports, and prior moderation history—into a single, structured case summary. The job-to-be-done is reducing the cognitive load on human reviewers by providing a consistent, evidence-backed brief that accelerates decision-making without sacrificing accuracy. It is not a replacement for human judgment in high-severity cases; it is a force multiplier for triage and review queues.

Use this prompt when you have a high volume of abuse reports that require a standardized intake format before a human reviewer or an automated policy engine takes action. The required inputs include raw report text, a list of automated classifier outputs with confidence scores, and the relevant policy document excerpts. The ideal output is a JSON object containing a severity classification, a timeline of events, extracted evidence snippets with source grounding, and a recommended action. Do not use this prompt for real-time blocking decisions where latency is critical and a simple classifier would suffice, or for generating final user-facing communications.

The primary constraint is evidentiary integrity. The prompt harness must be designed to prevent the model from hallucinating details not present in the source reports. This means every claim in the summary must be traceable to an input field. Before deploying, you must implement a validation step that checks for unattributed statements and a human-review escalation path for cases with conflicting or low-confidence evidence. If your workflow involves legally mandated reporting, the summary must be treated as an aid, not the official record, and a human must always confirm the final determination.

This template is the core instruction set for an LLM acting as a trust and safety case analyst. It is designed to receive a batch of disparate signals—automated detection flags, user reports, and moderator notes—and produce a single, structured case summary. The prompt enforces strict evidentiary discipline: every claim in the summary must be grounded in a specific source signal, and the model is instructed to explicitly state when information is missing rather than inferring it. Use this template as the foundation for your summarization harness, adapting the placeholders to match your platform's specific policy taxonomy, severity matrix, and output schema.

codeCopy
SYSTEM:
You are a Trust and Safety Case Analyst. Your task is to synthesize multiple abuse signals into a single, structured case summary. Your summary must be evidence-grounded, policy-aware, and actionable. You must never introduce details not present in the input signals. If information is missing or conflicting, state that explicitly.

INPUT SIGNALS:
[INPUT_SIGNALS]

POLICY TAXONOMY:
[POLICY_TAXONOMY]

SEVERITY MATRIX:
[SEVERITY_MATRIX]

OUTPUT SCHEMA:
{
  "case_id": "string, the identifier for this case",
  "summary": "string, a concise 2-3 sentence executive summary of the case",
  "signals_summary": [
    {
      "source_id": "string, the identifier of the input signal",
      "source_type": "string, e.g., 'automated_detection', 'user_report', 'moderator_note'",
      "key_finding": "string, the most relevant extracted fact or claim from this signal"
    }
  ],
  "policy_violations": [
    {
      "policy_id": "string, reference to the specific policy rule",
      "confidence": "string, HIGH | MEDIUM | LOW",
      "evidence_source_ids": ["string, list of source_ids supporting this violation"]
    }
  ],
  "severity_assessment": {
    "level": "string, e.g., 'CRITICAL', 'HIGH', 'MEDIUM', 'LOW'",
    "rationale": "string, explanation referencing the severity matrix and evidence"
  },
  "recommended_actions": [
    {
      "action": "string, e.g., 'SUSPEND_ACCOUNT', 'REMOVE_CONTENT', 'FLAG_FOR_REVIEW', 'NO_ACTION'",
      "rationale": "string, why this action is recommended based on policy and severity"
    }
  ],
  "missing_information": ["string, list any critical facts needed to make a determination that are absent from the input signals"],
  "conflicting_signals": ["string, describe any contradictions between input signals and how they were resolved"]
}

CONSTRAINTS:
[CONSTRAINTS]

USER:
Please generate the case summary for the provided input signals.

To adapt this template, start by replacing [POLICY_TAXONOMY] with your platform's actual policy rules, including definitions and examples for each violation category. The [SEVERITY_MATRIX] should map violation types and aggravating factors (e.g., volume, targeting of protected groups) to severity levels. The [CONSTRAINTS] placeholder is critical for high-risk deployments: use it to enforce rules like 'never output the raw content of CSAM reports in the summary' or 'always recommend human review for CRITICAL severity cases.' The output schema is a starting point; extend it with fields your review tooling requires, such as actor IDs, content permalinks, or legal hold flags. Before deploying, run this prompt against a golden dataset of known cases to calibrate confidence levels and ensure the model is not hallucinating policy references or evidence links.

This prompt is designed to operate as a synthesis step within a larger case management pipeline, not as a standalone chat interface. It receives pre-validated, structured inputs from upstream detection systems and user-reporting tools. The harness must enforce a strict boundary: the model is allowed to summarize and correlate evidence it receives, but it must never invent new evidence, infer unreported user identities, or speculate on intent beyond what the source flags explicitly state. The implementation should treat the model as an evidence organizer, not an investigator.

Wire the prompt behind an API endpoint that accepts a case_id and a pre-assembled [EVIDENCE_BLOCK] containing serialized abuse flags, user reports, and automated detection outputs. Before calling the model, validate that every evidence item has a non-null source_id, timestamp, and raw_excerpt. Reject the request if the evidence block is empty or exceeds a configurable token budget (recommended: 8K tokens). On the model response, run a strict post-processing validator that checks for hallucinated source IDs by comparing every cited reference in the output against the input evidence set. If the model cites a source not present in the input, flag the summary for human review and log the mismatch.

For model choice, use a model with strong instruction-following and low hallucination rates on structured summarization tasks. GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro are suitable starting points. Set temperature to 0 or near-zero to maximize factual consistency. Implement a retry loop with a maximum of 2 attempts: if the output fails JSON schema validation or the source-grounding check, retry with the validation error message appended to the prompt as a correction hint. If the second attempt also fails, route the case to a human reviewer with the partial output and validation errors attached. Log every attempt, validation result, and final routing decision to your case audit trail.

The output JSON must be parsed and stored in your case management system, not displayed raw to reviewers. Map the severity field to your internal severity taxonomy before writing to the database. The recommended_actions array should be compared against a whitelist of allowed automated actions in your system; any action not on the whitelist must be converted to a human-review task. Never execute automated enforcement actions (suspension, content removal) directly from model output without human approval unless your system has a separate, tested policy engine that validates the action against current policy rules and confidence thresholds.