Safety Filter Activation Review Prompt

This prompt is designed for trust and safety engineers, compliance reviewers, and AI operations staff who need to investigate a single production trace where a safety filter blocked a request or response. The job-to-be-done is not aggregate trend analysis or dashboard monitoring; it is the precise, forensic reconstruction of a single decision. Use it when you have a specific trace ID and need to understand exactly what the user asked, what the filter classified, what output was blocked, and which policy boundary triggered the action. The ideal user is someone who can read raw trace spans, understands the system's safety taxonomy, and is authorized to review potentially sensitive content.

The required context is a complete production trace containing the user input, the safety filter's classification metadata (scores, labels, thresholds), the blocked or flagged output, and the active policy configuration at the time of the request. Without this full trace, the prompt will produce an incomplete or misleading justification. Do not use this prompt for real-time intervention during an active incident—it is designed for post-hoc review. Do not use it to analyze aggregate false-positive rates across thousands of traces; that is a batch analysis task that requires a different prompt and a statistical harness. This prompt is also unsuitable for cases where the safety filter's internal logic is a black box with no exposed classification metadata, as the reconstruction will lack the necessary evidence.

Before running this prompt, confirm that you have the trace data, the policy version, and the authority to review the content. The output is a structured filter activation justification report that can be used for audit, policy tuning, or false-positive/false-negative analysis. If the trace is incomplete, the prompt's harness should flag missing spans and refuse to produce a definitive justification. After generating the report, the next step is typically a human review of the justification, followed by either a policy threshold adjustment, a filter rule update, or an escalation to the model provider if the classification appears incorrect. Avoid using this report as the sole evidence for a policy change without corroborating it with a batch analysis of similar traces.

This prompt template is designed to be pasted directly into your review tool or AI harness. It instructs the model to act as a trust and safety auditor, reconstructing the events of a single production trace where a safety filter was triggered. The goal is to produce a structured justification report that explains why the filter activated, what policy boundary was crossed, and whether the activation appears to be a true positive or a potential false positive. This report is the first step in calibrating filter thresholds and documenting review decisions for compliance.

textCopy
You are a trust and safety audit assistant. Your task is to review a single production trace where a safety filter activated. You will reconstruct the sequence of events, analyze the filter's classification, and produce a structured justification report.

## INPUT DATA
- User Input: [USER_INPUT]
- System Prompt (excerpt): [SYSTEM_PROMPT_EXCERPT]
- Activated Filter Name: [FILTER_NAME]
- Filter Classification: [FILTER_CLASSIFICATION]
- Filter Confidence Score: [CONFIDENCE_SCORE]
- Blocked Model Output (if any): [BLOCKED_OUTPUT]
- Relevant Policy Text: [POLICY_TEXT]

## TASK
1.  **Reconstruct the Event:** Summarize the user input and the model's intended response before the filter intervened.
2.  **Analyze the Classification:** Explain how the user input or model output maps to the activated filter's policy definition. Quote the specific policy text that applies.
3.  **Determine Justification:** Based on the policy and the evidence in the trace, classify the activation as one of the following:
    - **True Positive:** The content clearly violates the stated policy.
    - **Likely True Positive:** The content is in a grey area but leans toward a violation.
    - **Likely False Positive:** The content is in a grey area but leans toward being benign.
    - **False Positive:** The content clearly does not violate the stated policy.
4.  **Provide a Justification Rationale:** Write a concise paragraph explaining your classification, referencing specific evidence from the input, output, and policy.

## OUTPUT FORMAT
You must respond with a single JSON object conforming to this schema:
{
  "trace_summary": "A one-sentence summary of the user's intent.",
  "filter_name": "[FILTER_NAME]",
  "policy_trigger": "The specific policy clause that was violated.",
  "activation_classification": "True Positive | Likely True Positive | Likely False Positive | False Positive",
  "justification_rationale": "A detailed paragraph explaining the classification with evidence.",
  "recommended_action": "A clear next step: 'Escalate for human review', 'Log as false positive for filter tuning', or 'No action, filter worked correctly'."
}

## CONSTRAINTS
- Base your analysis ONLY on the provided trace data and policy text. Do not infer external context.
- If the evidence is ambiguous, default to 'Likely True Positive' and recommend human review.
- Do not generate the blocked content in your response. Refer to it abstractly.

To adapt this template for your environment, replace the square-bracket placeholders with data extracted from your observability platform or log files. The [POLICY_TEXT] placeholder is critical; without the exact policy definition, the model cannot make a justified decision. For high-stakes domains, this prompt's output should be treated as a draft recommendation, not a final ruling. Always log the generated JSON alongside the original trace for a complete audit trail.

The Safety Filter Activation Review Prompt is designed to be integrated into a trust and safety review queue, not run as a one-off chat interaction. When a safety filter activates in production, the system should capture the full trace—including user input, system prompt, filter classification, blocked output, and any tool calls or retrieved context—and enqueue it for automated analysis. This prompt consumes that trace and produces a structured justification report, which becomes the first pass in a human review workflow. The goal is to accelerate reviewer decision-making by pre-analyzing the filter activation, not to replace human judgment on policy boundary cases.

To wire this into an application, build a review pipeline with the following stages: Trace Capture — log the complete request context when any safety filter fires, including the filter name, confidence score, and the raw input that triggered it. Prompt Assembly — inject the captured trace fields into the prompt's [TRACE_INPUT], [FILTER_CLASSIFICATION], [BLOCKED_OUTPUT], and [POLICY_DEFINITIONS] placeholders. Model Invocation — send the assembled prompt to a capable model (GPT-4o, Claude 3.5 Sonnet, or equivalent) with temperature=0 to maximize consistency in justification structure. Output Validation — parse the JSON response and validate that required fields (filter_activation_justification, policy_boundary_assessment, false_positive_risk, recommended_action) are present and conform to the expected schema. Reject malformed outputs and retry once with a repair prompt that includes the validation error. Human Review Routing — if false_positive_risk is high or recommended_action is escalate, route the case to a human reviewer with the full justification report pre-filled. If false_positive_risk is low and the action is uphold, log the decision and close the case automatically, with a sampling rate for periodic audit.

For testing and evaluation, maintain a labeled dataset of known filter activations with ground-truth classifications: true positives (correct blocks), false positives (legitimate requests blocked), and boundary cases (ambiguous policy fits). Run the prompt against this dataset and measure false positive detection recall (did the prompt flag cases that were actually false positives?) and escalation precision (did cases routed to humans actually need review?). Track these metrics over time and set alert thresholds—for example, if false positive detection recall drops below 0.85, investigate whether policy definitions in [POLICY_DEFINITIONS] have drifted or whether the model is becoming overly deferential to filter decisions. Log every automated decision with the full prompt, model response, and validation result for auditability. Never allow automated overturning of a safety filter block without human approval; the prompt's recommended_action field should inform routing, not execute the final decision.