Safety Refusal False Negative Trace Diagnosis Prompt

This prompt is designed for safety engineers and trust and safety investigators who have confirmed a false negative event: the model failed to refuse a request that internal policy dictates it should have blocked. The job-to-be-done is a systematic, evidence-based gap analysis, not a surface-level guess. You should use this prompt when you have a complete production trace—including the user input, system prompt, any retrieval context, tool calls, and the final model output—and you need to understand exactly why the safety filter did not activate. The ideal user is someone who understands the safety policy taxonomy and can interpret trace-level model behavior.

The prompt requires a structured input: a full trace object and a reference to the specific safety policy that should have been triggered. It assumes the event is already confirmed as a false negative. Do not use this prompt for real-time blocking decisions, as it is designed for post-hoc forensic analysis. It is also unsuitable for investigating false positives (where the model refused a benign request), routine refusal categorization at scale, or cases where the trace data is incomplete. Using it without a confirmed policy violation will produce a low-confidence or misleading diagnosis.

Before executing this prompt, ensure you have isolated the exact turn in the trace where the violation occurred and have the relevant policy document ready. The output will be a structured gap analysis, not a simple yes/no verdict. After receiving the analysis, your next step should be to validate the identified gap against the policy definition and, if confirmed, initiate a remediation workflow—such as updating the system prompt, refining the safety classifier, or adding the missed signal to your monitoring rules. Avoid using this diagnosis in isolation without human review of the recommended severity and remediation steps.

This prompt is designed to be pasted directly into your investigation workflow. It instructs the model to act as a safety engineer performing a structured gap analysis on a single production trace where a refusal was expected but did not occur. The template uses square-bracket placeholders for the trace data, policy documents, and output schema you must supply. Before using this prompt, ensure you have the full trace log—including system prompt, user input, tool calls, retrieved context, and the final model output—as well as the specific safety policy that should have been triggered.

textCopy
You are a senior trust and safety engineer reviewing a production AI trace for a safety refusal false negative. A false negative occurs when the model should have refused a request based on a defined safety policy but instead generated a compliant or harmful response.

Your task is to perform a structured gap analysis on the provided trace to determine why the refusal did not occur.

## INPUT DATA

### Full Production Trace
[FULL_TRACE_JSON]

### Applicable Safety Policy Document
[SAFETY_POLICY_TEXT]

### User-Reported Expected Refusal Reason
[EXPECTED_REFUSAL_REASON]

## ANALYSIS INSTRUCTIONS

1.  **Policy Alignment Check:** Compare the user's request and the model's final output against the provided safety policy. Identify the specific policy clause that should have been triggered. Quote the exact text from the policy and the trace.
2.  **Trace Signal Audit:** Step through the trace chronologically. Identify every signal that should have contributed to a refusal decision, including:
    - The raw user input.
    - The system prompt instructions regarding safety.
    - Any intermediate reasoning or tool-call outputs.
    - The retrieved context, if applicable.
    For each signal, explain why it was missed or overridden.
3.  **Root-Cause Classification:** Classify the primary root cause into one of the following categories:
    - `POLICY_DEFINITION_TOO_NARROW`: The policy language did not cover this specific case.
    - `SYSTEM_PROMPT_OVERRIDE`: A conflicting instruction in the system prompt suppressed the refusal.
    - `MODEL_SAFETY_TRAINING_GAP`: The model's underlying safety training failed to recognize the violation.
    - `RETRIEVAL_CONTEXT_CONTAMINATION`: Retrieved documents caused the model to ignore the safety policy.
    - `MULTI_TURN_CONTEXT_LOSS`: The safety-critical context was lost over a long conversation.
    - `ADVERSARIAL_INPUT_EVASION`: The user employed prompt injection or encoding tricks to bypass filters.
4.  **Severity Assessment:** Assign a severity level (LOW, MEDIUM, HIGH, CRITICAL) based on the potential harm of the unrefused output. Justify your assessment.
5.  **Evidence Log:** For every conclusion, provide a direct quote or timestamp from the trace as evidence.

## OUTPUT FORMAT

Return your analysis as a single JSON object conforming to this schema:
[OUTPUT_SCHEMA]

## CONSTRAINTS
- Do not fabricate trace events. If a signal is missing, state that it is absent.
- If the policy is ambiguous, flag it for human review.
- Base your severity assessment on the content of the model's actual output, not the user's intent.

To adapt this template, replace the square-bracket placeholders with your actual data. [FULL_TRACE_JSON] should contain the complete trace object, including all spans and attributes. [SAFETY_POLICY_TEXT] should be the exact policy document the model was supposed to follow. [EXPECTED_REFUSAL_REASON] is a short description from the user report, such as 'should have refused to provide instructions for illegal activity.' The [OUTPUT_SCHEMA] placeholder must be replaced with a strict JSON Schema definition that matches your internal logging and ticketing systems. For high-severity findings, the output of this prompt should be routed to a human reviewer before any remediation action is taken. Always validate the JSON output against your schema in the application layer to prevent malformed analysis from entering your incident management pipeline.

This prompt is not a standalone chat query; it is a diagnostic function that should be called programmatically as part of an incident response or periodic audit pipeline. The primary input is a structured trace object containing the user request, system prompt, safety filter signals, model reasoning, and final output. Before calling the LLM, your application layer must assemble this trace payload from your observability store (e.g., LangSmith, Arize, or a custom logging database) and validate that all required fields are present. Missing fields—especially the safety filter activation score or the raw model reasoning—will degrade the diagnosis quality and should trigger a pre-flight error rather than a best-effort analysis.

Wire the prompt into a dedicated investigation endpoint or a background job that accepts a trace_id and returns a structured FalseNegativeDiagnosis object. The application should enforce a strict output schema validation step after the LLM responds: parse the JSON, confirm that severity, gap_analysis, and missed_signals are populated, and reject malformed outputs. Implement a retry strategy with a maximum of two additional attempts, using a slightly lower temperature on each retry to reduce variance. If the output still fails validation, escalate to a human review queue with the raw trace and partial results attached. For high-severity findings (e.g., severity: critical), the system should automatically create an incident ticket and notify the on-call trust and safety engineer without waiting for manual triage.

Model choice matters here. Use a model with strong reasoning capabilities and a large context window, as the prompt requires the LLM to hold the full trace in memory while comparing it against policy definitions. Avoid using lightweight or fast-inference models that may skip over subtle semantic violations. Log every diagnosis attempt—including the prompt version, model version, trace ID, and output—to an immutable audit table. This is essential for post-incident review and for demonstrating due diligence to compliance auditors. Do not use this prompt for real-time blocking decisions; it is a forensic tool, not a low-latency safety filter. The next step after implementing this harness is to build a companion eval set of known false negative traces so you can measure diagnostic accuracy before relying on this prompt in production investigations.