Production Refusal Event Trace Review Prompt Template

This prompt is built for a trust and safety engineer or AI reliability engineer who needs to perform a structured post-mortem on a single production refusal event. The job-to-be-done is to reconstruct the full decision path that led to a refusal, not just to confirm that a refusal occurred. The reader is expected to have access to a raw trace containing the user input, system prompt, any intermediate reasoning or tool calls, the safety filter activation signal, and the final refusal message. Without this trace data, the prompt cannot produce a reliable analysis.

Use this prompt when you need to answer specific diagnostic questions: What exact input triggered the refusal? Which safety policy or rule was activated? Was the refusal consistent with the policy definition, or does it look like a false positive? Did the model generate any unsafe content before the refusal? The prompt is designed to produce a structured output that can be logged, compared across events, or used as evidence in a compliance review. It is not a batch classification tool—use the Refusal Event Categorization Prompt for high-volume labeling—and it is not a replacement for a full incident report. It is a single-event diagnostic instrument.

Do not use this prompt when the trace is incomplete, when the refusal reason is already obvious from a simple keyword match, or when you are investigating a spike in aggregate refusal rates. For aggregate analysis, start with the Refusal Trace Root-Cause Classification Prompt to triage the spike, then drill into individual traces with this template. Also avoid this prompt if you are reviewing a policy boundary decision where the model allowed a request that feels borderline—use the Policy Boundary Decision Audit Prompt instead. This prompt assumes a refusal occurred; it is not designed to judge whether a refusal should have occurred in ambiguous cases.

This prompt template is designed to be copied directly into your AI harness, observability pipeline, or manual review tool. It accepts a raw production trace and a set of constraints, and it produces a structured analysis identifying the triggering input, the safety policy activated, the refusal message generated, and the full decision path. The template uses square-bracket placeholders for all dynamic inputs, making it safe to embed in code, configuration files, or prompt management systems without accidental token substitution.

textCopy
You are a trust and safety trace analyst. Review the following production trace for a single refusal event. Your task is to produce a structured analysis that identifies what triggered the refusal, which safety policy was activated, the refusal message generated, and the full decision path.

## INPUT TRACE
[TRACE_JSON]

## SAFETY POLICIES IN EFFECT
[POLICY_DEFINITIONS]

## REFUSAL TAXONOMY
[REFUSAL_CATEGORIES]

## OUTPUT SCHEMA
Return a valid JSON object with the following structure:
{
  "trace_id": "string",
  "refusal_detected": true,
  "triggering_input": {
    "turn_index": "integer",
    "user_message": "string",
    "flagged_content": "string"
  },
  "policy_activation": {
    "policy_name": "string",
    "policy_clause": "string",
    "activation_reason": "string",
    "activation_confidence": "number (0-1)"
  },
  "refusal_message": {
    "text": "string",
    "style": "enum: [standard, terse, verbose, helpful, deflective]",
    "safe_alternative_offered": "boolean"
  },
  "decision_path": [
    {
      "step": "integer",
      "component": "string (e.g., system_prompt, safety_filter, model_internal, tool_output)",
      "observation": "string",
      "decision": "string"
    }
  ],
  "context_contributing_factors": ["string"],
  "false_positive_risk": "enum: [low, medium, high]",
  "false_positive_evidence": "string (null if risk is low)",
  "requires_human_review": "boolean",
  "human_review_reason": "string (null if false)"
}

## CONSTRAINTS
- If the trace does not contain a refusal event, set refusal_detected to false and populate only trace_id.
- If the refusal reason is ambiguous, set requires_human_review to true and explain why.
- If the refusal appears to be a false positive, set false_positive_risk to high and provide specific evidence.
- Do not invent policy clauses. Reference only the policies provided in [POLICY_DEFINITIONS].
- If the trace contains PII, do not reproduce it in the output. Replace with [REDACTED].
- If the refusal message deviates from the approved template, flag it in the style field.

## EVALUATION CRITERIA
After producing the analysis, self-evaluate against these checks:
1. Is the triggering input correctly identified with the exact turn index?
2. Is the policy activation mapped to a specific clause from [POLICY_DEFINITIONS]?
3. Is the decision path complete and traceable to trace events?
4. Is the false_positive_risk assessment supported by evidence?
5. Is the human_review flag set correctly for ambiguous or high-risk cases?

Return ONLY the JSON object. No markdown fences, no commentary.

To adapt this template for your environment, replace each square-bracket placeholder with your actual data or configuration. [TRACE_JSON] should contain the full production trace in your observability format. [POLICY_DEFINITIONS] should include the exact policy text your safety filters enforce. [REFUSAL_CATEGORIES] should list your organization's refusal taxonomy. If your trace format differs from the assumed structure, add a preprocessing step that normalizes it before passing it to this prompt. For high-volume production use, wrap this prompt in a validation layer that checks the output JSON against the schema before accepting it, and route any output with requires_human_review: true to a review queue.

This prompt is designed to be invoked programmatically as part of a trust and safety review queue, not as a one-off chat interaction. The application layer is responsible for fetching the raw trace data from your observability platform, assembling the required input fields, and calling the LLM with this prompt template. The model's structured output must then be validated, logged immutably, and routed for human review when confidence is low or the event is high-severity. Treat this prompt as a deterministic analysis step in a larger pipeline—its value depends on consistent input formatting and strict output validation.

Input Assembly: The application must gather the full trace for a single refusal event, including the user's original request, the system prompt in use, any retrieved context or tool calls, the safety filter's activation metadata, and the final refusal message. These should be serialized into the [FULL_TRACE_JSON] placeholder. The [POLICY_DOCUMENT] placeholder expects the relevant policy text (e.g., acceptable use policy, safety guidelines) against which the refusal will be evaluated. The [REFUSAL_TAXONOMY] placeholder should contain your organization's predefined list of refusal categories. Model Choice: Use a model with strong instruction-following and structured output capabilities (e.g., GPT-4o, Claude 3.5 Sonnet). Set temperature=0 to maximize determinism for audit scenarios. Validation: Parse the model's JSON output and validate it against a strict schema. Check that triggering_input is a non-empty string, policy_activated matches an entry in your taxonomy, and decision_path is a non-empty array. Reject malformed responses and retry once with a repair prompt before escalating to a human.

Logging and Human Review: Every analysis result must be written to an append-only audit log with a timestamp, the trace ID, the model version used, and the full prompt and response. Implement a human review gate for any analysis where confidence_score < 0.85, severity is 'high' or 'critical', or the policy_activated category is on a pre-defined high-risk list (e.g., child safety, self-harm). The review interface should display the original trace side-by-side with the model's structured analysis, allowing a human to confirm, override, or add notes. Next Steps: After validation, route the structured output to your incident management system if the severity warrants it, or to a periodic compliance report if it's a routine audit. Avoid using this prompt for batch processing of thousands of traces without a human-in-the-loop sampling strategy—its primary value is in deep-dive review of individual events, not aggregate trend analysis.