Refusal Trace Root-Cause Classification Prompt

This prompt is built for AI SREs and trust and safety engineers who are facing a sudden increase in production refusal events and need to understand the dominant failure mode before diving into individual traces. Instead of manually inspecting hundreds of traces, you feed each refusal trace into this classifier to get a root-cause category: system prompt instruction, safety filter keyword, model safety training, retrieval context contamination, or user adversarial input. The prompt also produces a distribution summary so you can direct the investigation toward the most frequent cause—whether that means rolling back a prompt change, adjusting a keyword filter, cleaning a contaminated retrieval index, or escalating an attack pattern to security.

Use this prompt when you have a batch of refusal traces and need to answer the question: 'Is this a prompt problem, a filter problem, a retrieval problem, or an attack?' It is designed for triage at scale, not for deep-dive forensics on a single trace. The classification output is structured for ingestion into dashboards, incident reports, or automated alerting pipelines. Pair this with the Production Refusal Event Trace Review Prompt Template when you need to zoom into a specific trace after the distribution summary has identified the dominant category. Do not use this prompt for single-trace deep-dive forensics; it will produce a classification but won't give you the full decision path, policy clause mapping, or remediation steps that a dedicated trace review prompt provides.

Before running this prompt, ensure each trace contains the system prompt version, the user input, the model's refusal output, any safety filter metadata, and the retrieved context if RAG was involved. Incomplete traces will produce low-confidence classifications or null outputs. The prompt includes a confidence score for each classification and a flag for traces that require human review—typically those with conflicting signals or ambiguous refusal triggers. For high-severity incidents where a false negative or false positive could cause user harm, always route low-confidence classifications to a human reviewer before taking action on the distribution summary.

This prompt template is designed to be the core classification engine for your refusal trace triage workflow. It accepts a structured trace object and a predefined taxonomy of root-cause categories, then returns a single classification with supporting evidence. The template is model-agnostic but assumes the calling application has already extracted the relevant trace segments—system prompt, safety filter metadata, retrieved context, and user input—into a structured format before invoking the prompt.

textCopy
You are a refusal trace root-cause classifier. Your task is to analyze a production trace where the model refused to respond and classify the primary root cause into exactly one of the following categories:

- SYSTEM_PROMPT_INSTRUCTION: The refusal was triggered by an explicit instruction in the system prompt (e.g., "Do not answer questions about X").
- SAFETY_FILTER_KEYWORD: The refusal was triggered by a keyword or pattern match in a safety filter before the main model processed the request.
- MODEL_SAFETY_TRAINING: The refusal came from the model's internal safety training, not from an external filter or system prompt rule.
- RETRIEVAL_CONTEXT_CONTAMINATION: The refusal was caused by retrieved documents that contained disallowed content, triggering a downstream block.
- USER_ADVERSARIAL_INPUT: The user input contained an obvious prompt injection, jailbreak attempt, or adversarial pattern designed to bypass safeguards.
- AMBIGUOUS: The trace does not contain enough signal to confidently assign a single root cause.

Input trace data:
[TRACE_JSON]

Instructions:
1. Examine the system prompt section for any refusal-triggering instructions.
2. Examine safety filter metadata for keyword matches or pattern-based blocks.
3. Examine the model's refusal message for signs of internal safety training activation.
4. Examine retrieved context for content that could have triggered a refusal.
5. Examine the user input for adversarial patterns.
6. Weigh all evidence and select the single most likely root cause.

Output a JSON object with the following schema:
{
  "root_cause": "<CATEGORY>",
  "confidence": <0.0 to 1.0>,
  "evidence_summary": "<One-sentence summary of the key evidence supporting this classification>",
  "alternative_causes": ["<Other possible categories ruled out>"],
  "trace_signal_location": "<Where in the trace the decisive signal was found>"
}

To adapt this template for your environment, replace [TRACE_JSON] with your actual structured trace data before each call. The trace object should include at minimum: the system prompt text, any safety filter metadata (filter name, matched pattern, activation timestamp), the model's refusal response, retrieved context chunks with source identifiers, and the raw user input. If your observability platform captures additional fields—such as model version, latency breakdown, or tool-call history—include those as well, as they can help disambiguate between MODEL_SAFETY_TRAINING and SYSTEM_PROMPT_INSTRUCTION when both are present. For batch processing, wrap this prompt in a loop that feeds one trace at a time and aggregates the output classifications into a distribution summary for your investigation dashboard.

Before deploying this prompt into a production triage pipeline, validate it against a labeled set of at least 20 refusal traces where the root cause is known. Measure precision and recall per category, paying special attention to the AMBIGUOUS rate—if it exceeds 15%, your trace extraction step may be dropping critical signal. Wire the output into your incident response runbook so that SYSTEM_PROMPT_INSTRUCTION classifications route to the prompt engineering team, SAFETY_FILTER_KEYWORD classifications route to the safety filter configuration team, and USER_ADVERSARIAL_INPUT classifications route to the security team. For high-severity refusal spikes, require human review of all AMBIGUOUS classifications before closing the incident.

This prompt is designed to run as a batch classification step inside an AI SRE triage pipeline, not as a one-off chat interaction. When a spike in refusal events triggers an alert, the pipeline should pull the affected traces from your observability store, format each trace into the [TRACE_JSON] placeholder, and invoke the classification prompt. The output is a structured JSON object containing the root-cause category, supporting evidence, and a confidence score. This structured output is essential for downstream aggregation—you need to count categories across hundreds of traces, not read prose explanations one at a time.

Validation and retry logic is critical here. The prompt requests a specific JSON schema with an enum of root-cause categories (system_prompt_instruction, safety_filter_keyword, model_safety_training, retrieval_context_contamination, user_adversarial_input). Your harness must validate that the model's output parses as valid JSON, that the root_cause field contains exactly one of these enum values, and that the confidence field is a float between 0.0 and 1.0. If validation fails, implement a single retry with the error message appended to the prompt: "Your previous output failed validation: [VALIDATION_ERROR]. Please correct the JSON and ensure root_cause is one of the allowed enum values." After two failures, flag the trace for human review rather than silently defaulting to "unclassified". For model choice, prefer a model with strong instruction-following and JSON mode support (e.g., GPT-4o, Claude 3.5 Sonnet) at low temperature (0.0–0.2) to maximize classification consistency. Avoid models that tend to add explanatory prose outside the JSON block.

Downstream routing depends on the distribution summary this prompt produces. After classifying a batch of refusal traces, aggregate the results into a histogram of root-cause categories. If system_prompt_instruction dominates, route the investigation to the prompt engineering team to review the system prompt for overly broad refusal instructions. If safety_filter_keyword leads, route to the safety infrastructure team to audit keyword lists for false positives. If retrieval_context_contamination is the top category, the RAG pipeline team needs to investigate whether retrieved documents are introducing toxic or policy-violating content into the context window. Build this routing logic into your triage runbook so the classification output directly triggers the correct investigative workflow. Log every classification result—including the trace ID, root cause, confidence, and evidence excerpt—to your observability platform so you can track refusal root-cause trends over time and measure whether remediation efforts are reducing specific categories.