Refusal Reason Extraction Prompt from Production Logs

Operations teams running AI products at scale need to categorize why the system refused a user request. Manual review of every refusal trace does not scale. This prompt extracts a structured refusal reason from a raw production trace and normalizes it to a predefined taxonomy. It is designed for batch processing, dashboard ingestion, and trend reporting. The prompt returns a null output for ambiguous cases and flags items for human review, so your team focuses only on the traces that need judgment. Use this when you have a queue of refusal traces and need consistent, auditable categories without building a custom classifier.

This prompt is not a replacement for a full safety policy engine or a real-time intervention system. It is a post-hoc analysis tool. Do not use it to make blocking decisions in the request path. It works best when you already have a defined taxonomy of refusal categories—such as 'illegal content', 'self-harm', 'copyright', 'harassment', or 'regulated advice'—and you need to map raw trace data into those buckets. The prompt expects a trace that includes the user's input, the model's refusal response, and any safety filter metadata. If your traces are missing these fields, the extraction quality will degrade. For ambiguous cases where the refusal reason spans multiple categories or the trace evidence is thin, the prompt is instructed to return a null category and set a human_review_required flag to true. This keeps your review queue focused on the traces that actually need human judgment rather than flooding it with every refusal event.

Before wiring this into a production pipeline, run a calibration pass on 50–100 traces with a human reviewer. Compare the prompt's category assignments against the reviewer's judgments and measure agreement on the top 5 categories. If agreement falls below 90% for high-severity categories, refine the taxonomy descriptions in the prompt or add few-shot examples for the categories that drift. After calibration, you can batch-process refusal traces through this prompt and feed the structured output into your observability dashboard. Set up a separate alert for any batch where the human_review_required rate exceeds 20%, as this may indicate a taxonomy gap or a shift in the types of refusals your system is generating. The next section provides the copy-ready prompt template you can adapt to your taxonomy and trace format.

This prompt template is designed to be pasted directly into your trace analysis pipeline. It instructs the model to act as a compliance classifier, extracting the single most probable refusal reason from a raw production trace and normalizing it to your predefined taxonomy. The prompt is structured to handle ambiguity by returning a null reason and flagging the trace for human review, preventing forced misclassification. Replace the square-bracket placeholders with your specific taxonomy, trace data format, and output schema requirements before deployment.

textCopy
You are a compliance classifier. Your task is to extract the single most probable refusal reason from the provided production trace and normalize it to the predefined taxonomy.

## TAXONOMY
[TAXONOMY]

## INPUT TRACE
[TRACE_DATA]

## INSTRUCTIONS
1. Analyze the user's request and the model's refusal response in the trace.
2. Identify the primary safety or policy reason for the refusal.
3. Map this reason to the single best-matching category in the provided TAXONOMY.
4. If the reason is ambiguous or does not clearly match any category, set the reason to `null` and set the `human_review_flag` to `true`.
5. Do not invent a reason if the trace evidence is insufficient.

## OUTPUT FORMAT
Return a single, valid JSON object with the following schema:
{
  "refusal_reason": "<matching taxonomy category or null>",
  "human_review_flag": <boolean>,
  "trace_id": "<extracted from trace>"
}

To adapt this prompt, start by defining your [TAXONOMY] as a strict list of categories, such as ['illegal_content', 'self_harm', 'copyright', 'harassment', 'prompt_injection']. The [TRACE_DATA] placeholder should be replaced with the full JSON or text log of the user-model interaction, including system prompts and tool calls if available. For high-stakes compliance workflows, the human_review_flag is a critical safety valve; ensure your downstream processing logic respects this flag by routing flagged items to a review queue instead of auto-resolving them. Always validate the model's output against the defined JSON schema before ingestion.

This prompt is designed to run as a batch processing step against production refusal traces, not as a real-time user-facing interaction. The implementation harness should pull raw trace data from your observability store, format it into the [TRACE_LOG] and [TAXONOMY] placeholders, call the model, validate the structured output, and route ambiguous or low-confidence results to a human review queue. Because this workflow touches safety policy enforcement, treat the extraction output as advisory until it passes validation checks and, where flagged, human confirmation.

Wire the prompt into an asynchronous job runner (e.g., a scheduled Lambda, a Celery task, or a batch inference pipeline). For each trace, construct the prompt by injecting the raw refusal event context into [TRACE_LOG] and your organization's predefined refusal taxonomy into [TAXONOMY]. Call a model that supports strict structured output (GPT-4o with response_format, Claude with tool use, or any model with JSON mode enabled). Parse the returned JSON and validate: confirm the refusal_reason field matches an allowed enum value from your taxonomy, check that confidence is a float between 0.0 and 1.0, and verify that requires_human_review is a boolean. Any output that fails schema validation should trigger a single retry with the validation error appended to the prompt as additional [CONSTRAINTS]. If the retry also fails, route the trace to a dead-letter queue for manual inspection.

Set a confidence threshold for automatic acceptance—typically 0.85 or higher for this safety-critical task. Traces with confidence below the threshold or with requires_human_review set to true must be routed to a review interface where a trust and safety engineer can confirm or override the extracted reason. Log every extraction result, including the raw model output, validation status, retry count, and final reviewer decision, back to your observability platform. This audit trail is essential for demonstrating policy enforcement consistency to compliance reviewers. Avoid running this prompt on traces that contain PII in the refusal context unless you have redaction in place before the model call; the prompt itself does not perform redaction and should not receive unmasked sensitive data.