Refusal Reason Extraction and Classification Prompt

This prompt is designed for observability pipelines that need to convert unstructured model refusal responses into structured, queryable data. The primary job-to-be-done is post-hoc analysis: you have a log of raw text where a model refused a user request, and you need to classify why it refused, identify the specific policy invoked, assess the model's confidence in the refusal, and determine if the refusal is potentially recoverable. The ideal user is a safety engineer, policy auditor, or product manager who needs to understand refusal trends at scale without manually reading thousands of responses. Required context includes the raw refusal text and the original user request; optional context includes the system prompt, conversation history, and applicable policy documents.

This is not a real-time intervention tool. Do not use this prompt to make live safety decisions or to gate model responses in a production request path. It is an offline ingestion step for analytics, trend analysis, and policy auditing. The prompt assumes the refusal has already occurred and that you are now extracting structured metadata from it. It is also not a substitute for human policy review when dealing with genuinely ambiguous safety boundaries—the recoverability flag is a signal for downstream triage, not an automated override. For high-risk domains, always route flagged cases to human review before taking action on the classification output.

Before implementing this prompt, ensure your observability pipeline can capture the raw model response and the original user request together. The prompt's value depends on having both sides of the interaction. If your logging only captures the refusal text without the triggering request, the classification will be incomplete and the recoverability assessment unreliable. Once you have the structured output, you can ingest it into your analytics database for trend detection, policy gap analysis, and false-positive rate monitoring. The next step is to pair this extraction prompt with a validation schema that rejects malformed classifications before they enter your analytics store.

The prompt template below is designed to be dropped directly into your observability or safety logging pipeline. It takes an unstructured model refusal response and a reference safety policy, then outputs a structured JSON object suitable for analytics ingestion. The template uses square-bracket placeholders for all dynamic inputs, making it easy to parameterize in code before sending to the model.

textCopy
You are a safety observability classifier. Your task is to analyze a model's refusal response against a provided safety policy and extract structured refusal data.

## INPUT
Model Refusal Response:
"""
[REFUSAL_RESPONSE_TEXT]
"""

Safety Policy Reference:
"""
[SAFETY_POLICY_TEXT]
"""

## OUTPUT SCHEMA
Return ONLY a valid JSON object with the following fields. Do not include any text outside the JSON.

{
  "refusal_detected": boolean,
  "refusal_reason": string (direct quote or tight paraphrase from the response explaining why the request was refused),
  "reason_code": string (classify into one of: "POLICY_VIOLATION", "CAPABILITY_GAP", "LEGAL_RESTRICTION", "SAFETY_BOUNDARY", "AMBIGUOUS_REQUEST", "OTHER"),
  "policy_reference": string (the specific policy clause, section, or principle cited, or null if none cited),
  "confidence": number (0.0 to 1.0 indicating how clearly the refusal maps to a policy reason),
  "recoverable": boolean (true if the request could likely be fulfilled with rephrasing or additional context, false if the refusal appears definitive),
  "recovery_suggestion": string (if recoverable, a brief suggestion for how the user could rephrase; otherwise null),
  "analysis_notes": string (brief analyst note on any ambiguity, policy interpretation edge cases, or notable patterns)
}

## CONSTRAINTS
- If the response does not contain a refusal, set refusal_detected to false and leave other fields null.
- Do not invent policy citations. Only reference policy text actually provided.
- For ambiguous cases, set confidence below 0.7 and explain the ambiguity in analysis_notes.
- Never classify a legitimate safety refusal as recoverable if recovery would circumvent the policy.

To adapt this template, replace [REFUSAL_RESPONSE_TEXT] with the raw model output you want to classify, and [SAFETY_POLICY_TEXT] with the relevant policy document or excerpt. For production use, validate the output JSON against the schema before ingestion. If the model returns malformed JSON, implement a retry with stricter format instructions or fall back to a repair prompt. For high-stakes safety observability, always sample and manually review classifications to calibrate confidence thresholds and catch systematic misclassifications.

This prompt is designed to be a post-processing step in an observability pipeline, not a real-time user-facing component. After a model generates a refusal response, that raw text is routed to this prompt for structured extraction. The output is a machine-readable classification record that downstream analytics systems can ingest. The primary integration point is a logging middleware or a message queue consumer that receives model responses, detects refusals (either via a lightweight classifier or a keyword heuristic), and then invokes this prompt to produce the structured record. Do not use this prompt on non-refusal responses; it will hallucinate refusal reasons where none exist.

The implementation harness should enforce a strict schema contract before the classification record enters your analytics store. Wrap the LLM call in a validation layer that checks: (1) the reason_code matches your internal taxonomy enum, (2) the confidence field is a float between 0.0 and 1.0, (3) the recoverable field is a boolean, and (4) the policy_reference field is not null when the refusal is policy-based. If validation fails, implement a single retry with the validation error message appended to the prompt as additional context. If the retry also fails, log the raw refusal text and the failed parse attempt to a dead-letter queue for manual review. For high-volume production systems, consider using a cheaper, faster model (like a fine-tuned small classifier) for the initial refusal detection step, reserving this prompt for the detailed extraction on detected refusals only.

Logging is critical for this workflow. Every invocation should produce an audit record containing: the original model response, the extracted classification, the model used for extraction, the latency, and the validation status. This audit trail enables offline analysis of refusal patterns, false-positive rate monitoring, and policy gap detection. When deploying this prompt, start with a shadow mode that logs classifications without blocking or altering the user experience. After validating accuracy against a human-labeled sample of at least 200 refusal responses, promote the pipeline to active use. Avoid wiring this prompt directly into a synchronous user-facing code path; the extraction latency will degrade the user experience, and a failed extraction should never block a response from being delivered.