Safety Refusal Event Summary Prompt Template

This prompt is designed for trust and safety engineers who need to brief compliance stakeholders, legal teams, or product leadership on refusal events detected in production AI traces. It transforms raw safety filter activations, policy boundary decisions, and refusal logs into a structured summary that preserves diagnostic signal while being readable by non-engineering audiences. Use this prompt when you have a batch of refusal events from a monitoring period and need to communicate trends, false-positive/false-negative rates, and policy categorization accuracy.

The ideal input is a structured dataset containing refusal event records, each with a timestamp, the user prompt or a sanitized version of it, the specific safety policy that was triggered, the model's refusal response, and any human review outcome or override flag. The prompt works best when the data has already been deduplicated and enriched with policy categories. Do not feed raw, unsampled trace streams directly into this prompt; pre-aggregate the data to avoid overwhelming the context window and to ensure the summary reflects a representative sample rather than noise. The output is a prose summary with embedded statistics, not a raw data table, so the input must contain enough volume for trend analysis but be compact enough for a single model call.

Do not use this prompt for real-time intervention; it is a retrospective reporting tool, not an operational alerting system. It is also unsuitable for generating individual refusal justifications for end-users or for making automated policy changes. The summary is intended to inform human decision-makers, so every output must be reviewed by a qualified engineer before distribution. Avoid using this prompt when the underlying data contains unresolved false-positive disputes, as the summary will inherit that ambiguity and may misrepresent the system's actual safety performance.

This prompt template is designed to ingest a batch of production trace events related to safety refusals, policy boundary decisions, and filter activations. It instructs the model to produce a structured summary suitable for trust and safety engineers briefing compliance stakeholders. The output must prioritize correct policy categorization, identification of false-positive and false-negative patterns, and trend context over raw event volume.

textCopy
You are a trust and safety analysis assistant. Your task is to review a set of safety refusal events extracted from production AI traces and produce a structured compliance summary.

## INPUT DATA
[REFUSAL_EVENT_LOG]

## TASK
Analyze the provided refusal events and generate a summary report. The report must be factual, grounded only in the provided events, and structured for a compliance stakeholder audience.

## OUTPUT SCHEMA
Your response must be a valid JSON object with the following structure:
{
  "summary": {
    "total_events": <integer>,
    "analysis_period": "[START_DATE] to [END_DATE]",
    "overall_trend": "<increasing|decreasing|stable>"
  },
  "by_policy_category": [
    {
      "policy_name": "<string>",
      "event_count": <integer>,
      "trend_direction": "<up|down|flat>",
      "representative_example_id": "<string or null>"
    }
  ],
  "false_positive_analysis": {
    "identified_count": <integer>,
    "common_pattern": "<string summarizing the pattern or 'None identified'>",
    "risk_assessment": "<string>"
  },
  "false_negative_analysis": {
    "identified_count": <integer>,
    "common_pattern": "<string summarizing the pattern or 'None identified'>",
    "risk_assessment": "<string>"
  },
  "key_observations": ["<string>", ...],
  "recommended_actions": ["<string>", ...]
}

## CONSTRAINTS
- Do not invent events, dates, or patterns not present in the input log.
- If a false positive or false negative pattern cannot be reliably identified, set the count to 0 and state 'None identified'.
- The 'risk_assessment' fields must use one of the following values: 'Low', 'Medium', 'High', 'Critical'.
- 'key_observations' should contain 3-5 concise, evidence-backed findings.
- 'recommended_actions' should contain 1-3 actionable suggestions for the trust and safety team.

## EXAMPLES
[FEW_SHOT_EXAMPLES]

To adapt this template, replace the [REFUSAL_EVENT_LOG] placeholder with a structured JSON array of refusal events from your trace store. Each event object should contain at least an event_id, timestamp, policy_triggered, user_input_snippet, and a review_label field indicating if the refusal was a true positive, false positive, or false negative. The [FEW_SHOT_EXAMPLES] placeholder should be replaced with one or two correctly formatted output examples that demonstrate the desired tone and categorization precision for your specific policy taxonomy. Before deploying this prompt in a production pipeline, validate that the output JSON strictly conforms to the schema using a programmatic validator, and route any output where risk_assessment is 'High' or 'Critical' for human review before distribution.

This prompt is designed to run as a scheduled batch job or on-demand report generator, not as a real-time inline classifier. It consumes structured refusal event records extracted from production traces—typically from a logging pipeline or observability platform—and produces a stakeholder-ready summary. The implementation harness must enforce strict input schema validation before the prompt runs, because malformed or incomplete event records will produce misleading summaries that undermine trust with compliance stakeholders. Each input record must include at minimum: a unique event ID, timestamp, refusal category (from a controlled taxonomy), the user prompt or a sanitized version of it, the safety filter or policy rule that triggered, and a disposition flag (true positive, false positive, false negative, or borderline). If any required field is missing or outside the expected enum, the harness should reject the batch and log a schema violation before the model ever sees the data.

The harness should follow a validate → assemble → invoke → validate → escalate pattern. First, validate all input records against the required schema and taxonomy. Second, assemble the prompt by injecting the validated records into the [EVENT_RECORDS] placeholder, populating [REPORTING_PERIOD], [POLICY_TAXONOMY], [TREND_CONTEXT], and [STAKEHOLDER_AUDIENCE] from the job configuration. Third, invoke the model with a moderate temperature (0.1–0.3) to preserve factual consistency while allowing natural summary language. Fourth, validate the output: parse the JSON response, confirm all required sections are present, verify that every cited event ID exists in the input batch, and check that refusal category counts match the input distribution within a small tolerance. If the output fails validation, retry once with an explicit error message injected into the retry prompt. If the retry also fails, escalate the raw input and partial output to a human reviewer queue rather than silently publishing a broken summary. For high-risk compliance workflows, always require human approval before the summary reaches external stakeholders—the prompt's [RISK_LEVEL] parameter should gate whether the pipeline auto-publishes or queues for review.

Model choice matters here. Use a model with strong instruction-following and structured output capabilities (GPT-4o, Claude 3.5 Sonnet, or equivalent). Avoid smaller or older models that may miscount events, hallucinate trend percentages, or misclassify refusal categories in the summary text. If you are processing large volumes of refusal events (thousands per reporting period), pre-aggregate the data into statistical summaries before passing it to the prompt—do not dump raw event lists into the context window. The prompt's [TREND_CONTEXT] field should contain pre-computed comparison data (prior period counts, change percentages, top categories) so the model's job is synthesis and narrative construction, not arithmetic. Log every invocation with the prompt version, model identifier, input record count, output validation status, and reviewer decision. This audit trail is essential for compliance stakeholders who may later ask how a specific summary was produced. Finally, treat the refusal category taxonomy as a configuration artifact that evolves with your safety policies—when you update the taxonomy, update the [POLICY_TAXONOMY] placeholder and run a regression test against a golden set of historical events to confirm the new taxonomy doesn't break the summary structure or miscategorize legacy events.