Refusal Root Cause Analysis Prompt

This prompt is designed for safety engineers, policy teams, and AI reliability engineers who need to perform a structured root cause analysis on a specific model refusal. The job-to-be-done is not real-time recovery; it is post-incident diagnosis. Use it when a refusal pattern appears systemic, during a policy refinement cycle, or when an incident review requires a traceable explanation of why a legitimate or borderline request was blocked. The prompt assumes you have already collected the original user request, the model's exact refusal response, the active system prompt and safety policies, and any relevant conversation context. Without these artifacts, the analysis will be speculative rather than evidentiary.

Do not use this prompt as a live recovery mechanism in a user-facing session. It is an offline diagnostic tool. It is also not a replacement for aggregate refusal analytics or an A/B testing framework. If you need to correct a false refusal in the same session, use the Legitimate Request Recovery After False Refusal prompt. If you need to review refusal patterns at scale, use the Refusal Audit and Log Review prompt. This prompt is most effective when you have a single, concrete refusal event that represents a larger concern—such as a new policy interaction, a model version upgrade, or a high-visibility user complaint. The analysis will be only as good as the completeness of the input artifacts you provide.

The output is a structured root cause analysis that traces the refusal from the surface response backward through three layers: the model's interpretation of the active safety policies, the influence of the system prompt and any conflicting instructions, and potential training data or alignment effects that may have biased the decision. Each causal factor is rated for confidence and linked to specific evidence in the provided artifacts. The analysis concludes with corrective action recommendations, each carrying an explicit risk assessment so your team can decide whether to adjust policy language, modify system instructions, change escalation thresholds, or accept the refusal as correct. Always pair the automated analysis with human review before acting on recommendations that could weaken safety boundaries.

This is the core diagnostic prompt for investigating a single refusal event. It is designed to be pasted into your investigation workflow, such as an internal safety analysis tool, a manual review queue, or an automated post-mortem pipeline. The prompt instructs the model to act as a safety engineer and trace the refusal from the surface-level response back to its originating policy, instruction, or training data influence. Before using it, you must gather the exact user request, the full model response, and any relevant system prompt or policy documents that were active at the time of the refusal.

textCopy
You are a senior safety engineer conducting a root cause analysis of a specific AI model refusal. Your goal is to trace the refusal back to its origin and recommend corrective action, not to justify the refusal or critique the user.

## INPUT

**User Request:**
[USER_REQUEST]

**Model Response (Refusal):**
[MODEL_RESPONSE]

**Active System Prompt / Policy Instructions:**
[SYSTEM_PROMPT]

**Relevant Policy Documents (if any):**
[POLICY_DOCUMENTS]

**Additional Context (user history, session data, etc.):**
[ADDITIONAL_CONTEXT]

## ANALYSIS INSTRUCTIONS

1.  **Refusal Classification:** Categorize the refusal type (e.g., Over-Refusal, Correct Refusal, Policy Misinterpretation, Capability Gap, Ambiguous Boundary).
2.  **Causal Chain:** Trace the most likely causal chain from the model's response back through policy interpretation, system instructions, and any few-shot examples or training data influence you can infer. Be explicit about each link in the chain.
3.  **Policy Citation:** If a specific policy clause or instruction was the primary trigger, quote it directly. If the refusal stems from an absence of policy, state that clearly.
4.  **Corrective Action Recommendation:** Recommend a specific, actionable change. This could be a policy clarification, a system prompt edit, a few-shot example addition, or a note that the refusal was correct and no action is needed. For each recommendation, assess the risk of introducing new safety gaps.
5.  **Confidence Assessment:** Provide a confidence level (High/Medium/Low) for your root cause analysis and explain the primary source of uncertainty.

## OUTPUT FORMAT

Return your analysis as a structured JSON object with the following schema:
{
  "refusal_classification": "string",
  "causal_chain": [
    {
      "step": "string describing a link in the chain",
      "evidence": "string quoting the source of this step (response, policy, etc.)"
    }
  ],
  "policy_trigger": "string | null",
  "corrective_actions": [
    {
      "action": "string describing the recommended change",
      "target": "string indicating what to change (e.g., system_prompt, policy_doc, few_shot_examples)",
      "risk_assessment": "string describing potential safety regression risk"
    }
  ],
  "root_cause_confidence": "High | Medium | Low",
  "uncertainty_source": "string"
}

To adapt this template, replace the square-bracket placeholders with data from your production logs or review queue. The [SYSTEM_PROMPT] and [POLICY_DOCUMENTS] fields are critical for an accurate analysis; without them, the model can only guess at the policy interpretation step. For high-risk safety systems, the output of this prompt should be treated as an initial diagnostic, not a final verdict. Always have a human safety engineer review the corrective_actions before they are implemented to prevent accidental weakening of genuine safety constraints. The structured JSON output is designed to be logged directly into an incident tracking system for trend analysis over time.

The Refusal Root Cause Analysis prompt is designed to be a diagnostic tool, not a real-time interceptor. It should be wired into an offline investigation workflow or a post-hoc observability pipeline. The primary integration point is a log sink that captures refused requests, the raw model response, the active system prompt and policy configuration at the time of refusal, and any preceding conversation turns. The harness must assemble these artifacts into the [INPUT_CONTEXT] placeholder before invoking the analysis prompt.

For a production-grade implementation, wrap the prompt call in a structured logging harness. On invocation, log a unique analysis_id, the request_id of the refused interaction, and the model version used for analysis. Validate the output against the [OUTPUT_SCHEMA] strictly: the root_cause_classification must be one of the predefined enums (e.g., policy_misinterpretation, overly_conservative_instruction, training_data_influence, ambiguous_input), and the corrective_action object must contain a non-null risk_assessment field. If validation fails, retry once with a stricter schema reminder. If it fails again, escalate the raw artifacts to a human safety reviewer via a ticketing queue, as an unparseable analysis is itself a signal of a complex or novel failure mode.

Model choice matters here. Use a highly capable, reasoning-forward model (such as GPT-4o, Claude 3.5 Sonnet, or Gemini 2.5 Pro) because the task requires nuanced policy interpretation and counterfactual reasoning. Do not run this prompt on a lightweight model optimized for latency. The analysis is inherently asynchronous and high-stakes; correctness is more important than speed. Store the validated output alongside the original refusal log to build a searchable knowledge base of refusal patterns. This allows you to track the frequency of specific root causes over time and measure the impact of policy or instruction changes. Avoid wiring this prompt directly into a user-facing retry loop; its purpose is to inform systemic fixes, not to generate an immediate user-facing recovery message.