Refusal Event Categorization Prompt for Production Logs

This prompt is designed for data analysts and trust and safety engineers who need to categorize a batch of refusal events from production logs into a predefined taxonomy for trend reporting. The core job-to-be-done is turning a raw, unstructured refusal trace or log entry into a structured, machine-readable JSON object with a category, sub-category, and confidence score. This structured output is essential for ingestion into dashboards, generating weekly safety reports, and identifying emerging patterns in model refusals without manually reading every trace. The ideal user has access to production log exports and a defined safety taxonomy, and they need consistent, repeatable labels at scale.

Use this prompt when you have a backlog of refusal events that require classification for aggregate analysis. It is a batch processing prompt, meaning it is optimized for running over a static dataset, not for real-time decision-making. The prompt expects a single raw trace as the [INPUT] and returns a single JSON object. You should integrate it into a script or pipeline that iterates over log files, sending one trace per API call. The prompt is not designed to act as a real-time safety filter, a single-trace diagnostic tool for root-cause analysis, or a replacement for human judgment on ambiguous policy violations. For those tasks, use sibling prompts like the Production Refusal Event Trace Review Prompt or the Safety Filter Activation Root-Cause Analysis Prompt.

Before using this prompt, ensure you have a well-defined, stable taxonomy of refusal categories and sub-categories. The prompt's reliability depends on this taxonomy being provided clearly within the [CONTEXT] or [CONSTRAINTS] placeholder. Do not use this prompt if your goal is to discover new categories or if your taxonomy is in flux; the model will force-fit events into the provided schema, which can mask novel failure modes. After classification, always review the confidence scores. A low-confidence score is a signal to route that trace to a human reviewer or a more detailed diagnostic prompt. The next step is to wire this prompt into a validation harness that checks the output schema and flags low-confidence results for escalation.

This prompt template is designed to be pasted directly into your batch processing script. It instructs the model to analyze a raw refusal event trace and classify it according to your predefined taxonomy. The output is a structured JSON object suitable for direct ingestion into monitoring dashboards and trend analysis tools. Before using this template, ensure you have defined your refusal taxonomy and have extracted the relevant fields from your production traces.

textCopy
You are a production safety log classifier. Your task is to analyze a raw refusal event trace and categorize it into a predefined taxonomy.

## TAXONOMY
Use the following taxonomy to classify the refusal event. If no category fits, use "other".
[TAXONOMY]

## INPUT DATA
The following is a raw refusal event from our production logs. It includes the user's prompt, the model's refusal response, and any relevant system messages or safety filter metadata.
[RAW_REFUSAL_EVENT]

## INSTRUCTIONS
1. Analyze the user's prompt and the model's response to understand the reason for the refusal.
2. Identify the single most specific category and sub-category from the provided taxonomy that matches the refusal reason.
3. Assign a confidence score between 0.0 and 1.0 for your classification.
4. If the refusal reason is ambiguous or doesn't clearly fit any category, set the category to "other" and the confidence score to a low value.
5. Provide a brief, one-sentence justification for your choice.

## OUTPUT_SCHEMA
You must output a single, valid JSON object with the following structure. Do not include any other text.
{
  "category": "string",
  "sub_category": "string | null",
  "confidence": "float",
  "justification": "string"
}

To adapt this template, replace the [TAXONOMY] placeholder with your organization's specific refusal categories and sub-categories, formatted as a clear list or hierarchy. The [RAW_REFUSAL_EVENT] placeholder should be replaced with the full text of the event you are classifying, which might include the user prompt, the final refusal message, and any relevant system instructions or safety filter metadata. For high-stakes compliance workflows, the justification field is critical for downstream human review. Always validate the output JSON against the schema before sending it to your dashboard to catch malformed responses.

This prompt is designed to be called once per refusal event in a batch processing workflow. The typical integration pattern involves a script or lightweight worker that reads raw refusal events from your log store (e.g., Elasticsearch, BigQuery, S3, or a Kafka topic), constructs the prompt payload for each event, calls the model endpoint, and writes the structured JSON result to your analytics database. Because this prompt produces a machine-readable JSON classification, it is intended for downstream dashboard ingestion and trend reporting, not for real-time user-facing decisions. The batch size, rate limiting, and concurrency settings will depend on your model provider's throughput limits and your cost tolerance. For high-volume pipelines processing millions of events per day, consider using a cheaper, faster model (such as GPT-4o-mini, Claude Haiku, or Gemini Flash) for initial categorization and escalating only low-confidence cases (confidence < 0.7) to a more capable model for re-evaluation.

Implement a strict validation layer between the model response and your database write. The prompt instructs the model to return a JSON object with category, sub_category, and confidence fields, but production models will occasionally return malformed JSON, wrap the output in markdown fences, or omit required fields. Your harness must parse the response, validate the schema, check that category matches your predefined taxonomy, and confirm that confidence is a float between 0 and 1. If validation fails, implement a single retry with a repair prompt that includes the original input, the malformed response, and a clear instruction to return only valid JSON matching the expected schema. If the retry also fails, log the raw event and the failed response to a dead-letter queue for manual review rather than silently dropping the data. Log every successful categorization alongside the raw refusal event ID and the model version used, so that any downstream trend analysis can be audited back to the source trace.

Before sending any user input to an external model endpoint, strip personally identifiable information (PII) from the refusal event context. Refusal traces often contain the user's original request, which may include names, email addresses, phone numbers, or account identifiers. Your preprocessing step should redact or hash these fields before they leave your infrastructure. If you are using a self-hosted or private deployment, this constraint may be relaxed, but the audit trail should still record whether PII was present in the original event and whether it was redacted before model inference. Finally, treat the model's categorization as a recommendation, not a ground-truth label. For high-risk categories (such as those involving legal or safety policy boundaries), route a sample of categorizations to a human reviewer for calibration, and use their corrections to refine your taxonomy definitions and few-shot examples over time.