Refusal Event Stakeholder Summary Prompt

This prompt is designed for trust and safety leads, AI program managers, and compliance officers who need to prepare a weekly summary of refusal events for non-technical stakeholders. The core job-to-be-done is converting raw, aggregated production trace data into a structured brief that communicates refusal volume, top categories, notable false positive cases, and trend direction. The output must be decision-ready, allowing a reader to understand the safety posture without needing to interpret trace-level details or raw logs.

Use this prompt when you have already aggregated refusal event data from your observability pipeline and need a consistent, executive-friendly format for weekly reporting. The ideal input is a structured dataset containing categorized refusal counts, a list of reviewed false positive cases, and a comparison to the previous period's volume. The prompt is not designed for real-time incident response or for diagnosing individual refusal traces; those tasks require the sibling trace-review prompts in this content group, which are built for granular, event-level analysis. Do not use this prompt if you lack aggregated data, as it will hallucinate trends and categories to fill the gap.

Before executing this prompt, ensure your input data has been validated for completeness. The prompt expects concrete numbers and pre-classified categories. If your observability pipeline cannot yet produce this aggregated data, implement a batch categorization step first using a prompt like the 'Refusal Event Categorization Prompt for Production Logs' from this content group. The summary prompt's value is entirely dependent on the quality of the upstream aggregation. After generating the summary, a human must review the 'Notable False Positive Cases' section for accuracy and sensitivity before distribution, as this section can influence policy and perception.

This prompt template is the core of the Refusal Event Stakeholder Summary workflow. It is designed to be populated with data from your refusal event observability pipeline—such as log aggregators, trace databases, or safety monitoring dashboards—and then sent to a model to produce a structured, non-technical brief. The output is intended for trust and safety leads, product managers, or executive stakeholders who need to understand refusal trends without inspecting raw traces.

textCopy
You are a trust and safety reporting assistant. Your task is to generate a weekly stakeholder summary of AI refusal events from the provided production data. The summary must be clear, concise, and decision-ready for a non-technical audience.

## INPUT DATA
- Refusal Event Logs (JSON array): [REFUSAL_EVENT_LOGS]
- Previous Period Summary (for trend comparison): [PREVIOUS_PERIOD_SUMMARY]
- Reporting Period: [REPORTING_PERIOD]
- Taxonomy of Refusal Categories: [REFUSAL_CATEGORY_TAXONOMY]

## OUTPUT SCHEMA
Produce a JSON object with the following structure:
{
  "executive_summary": "A 2-3 sentence overview of the period's refusal activity and key takeaway.",
  "total_refusal_volume": <integer>,
  "volume_change_percent": <number or null>,
  "top_refusal_categories": [
    {
      "category": "string",
      "count": <integer>,
      "trend_direction": "up" | "down" | "stable"
    }
  ],
  "notable_false_positive_cases": [
    {
      "trace_id": "string",
      "user_intent_summary": "A brief, de-identified description of what the user was trying to do.",
      "refusal_reason": "string",
      "impact_assessment": "low" | "medium" | "high"
    }
  ],
  "trend_analysis": "A short paragraph describing the overall trend, citing the volume change and any category shifts.",
  "recommended_actions": [
    "A list of 1-3 actionable recommendations for the stakeholder, such as policy review, allowlist update, or user communication."
  ],
  "requires_human_review": <boolean>
}

## CONSTRAINTS
- Base every claim strictly on the provided [REFUSAL_EVENT_LOGS]. Do not infer or invent data.
- De-identify all user-specific information in summaries. Use general descriptions only.
- If [PREVIOUS_PERIOD_SUMMARY] is null or empty, set `volume_change_percent` to null and note the lack of baseline in `trend_analysis`.
- Classify each refusal event into exactly one category from [REFUSAL_CATEGORY_TAXONOMY]. If an event does not fit, categorize it as "other" and flag it for review.
- A "false positive" is defined as a refusal where the user's intent, after review, was deemed safe and policy-compliant. Only include cases explicitly marked as confirmed false positives in the input logs.
- Set `requires_human_review` to `true` if any false positive has a "high" impact assessment or if the volume change exceeds 20%.

To adapt this template, replace each square-bracket placeholder with data from your observability pipeline. The [REFUSAL_EVENT_LOGS] should be a pre-processed JSON array of refusal events, each containing a trace_id, timestamp, refusal_reason, category, user_intent_summary, and a false_positive boolean flag. The [REFUSAL_CATEGORY_TAXONOMY] should be your organization's canonical list of refusal categories, provided as a JSON array of strings. Before sending this prompt to a model, validate that the input logs are complete and that any false positive cases have been confirmed by a human reviewer. The output JSON must be validated against the schema before it is shared with stakeholders; any schema violations should trigger a retry or a human-in-the-loop review.

This prompt is designed to run as a scheduled batch job, not as a real-time interactive query. The primary integration point is a weekly or daily cron trigger that queries a production trace database or log store for refusal events within the reporting window, aggregates the raw events into the [RAW_REFUSAL_EVENTS] input structure, and invokes the LLM to produce the stakeholder summary. The output should be written to a shared document store, wiki, or notification channel where non-technical stakeholders consume it. Because the summary will be read by people who cannot inspect raw traces, the implementation must validate the output before distribution.

The harness should enforce a strict input schema before the prompt runs. The [RAW_REFUSAL_EVENTS] placeholder expects a JSON array of refusal event objects, each containing at minimum: timestamp, refusal_category, user_visible_refusal_message, safety_policy_triggered, and trace_id. If the upstream query returns zero events, the harness should short-circuit and generate a 'no refusal events this period' placeholder summary rather than invoking the LLM with an empty input. After the LLM returns the summary, run a structural validator that checks for the required sections: refusal volume, top categories, notable false positive cases, and trend direction. If any section is missing or the output is malformed, retry once with a repair prompt that includes the validation errors. If the retry also fails, escalate the raw trace data and the partial output to a human reviewer via a ticketing system or review queue.

Model choice matters for this workflow. Use a model with strong instruction-following and structured output capabilities, such as Claude 3.5 Sonnet or GPT-4o, because the prompt requires precise section adherence and numeric accuracy. Set temperature to 0 or a very low value (0.1) to minimize variance in the summary language and category counts. The prompt does not require tool use or retrieval-augmented generation because all necessary data is provided in the input. However, the harness should log the full prompt, the raw input, the model response, and the validation result for every run. These logs serve as an audit trail for compliance reviewers who may later need to trace a summary claim back to a specific refusal event. Finally, add a human approval gate before the summary is distributed: if the validator flags any anomaly, or if the refusal volume exceeds a configurable threshold, route the summary to a trust and safety lead for review rather than publishing automatically.