Safety Classification with Uncertainty Output Prompt Template

This prompt is for safety systems that must do more than output a binary safe/unsafe label. It produces a classification alongside explicit epistemic uncertainty (what the model does not know because of insufficient or ambiguous evidence) and aleatoric uncertainty (what is inherently ambiguous in the input itself). Use this when your safety architecture routes high-uncertainty cases to human review, logs uncertainty for audit trails, or applies different response policies based on confidence levels. This prompt belongs in the classification layer of a safety platform, upstream of any response generation or tool access decision. It assumes you have defined safety policies and harm categories, and that you need machine-readable uncertainty estimates rather than a single point prediction.

The ideal user is an ML engineer or safety platform builder who already has a defined taxonomy of harm categories and needs to integrate classification into a larger decision pipeline. You should have a clear understanding of where this classifier sits in your architecture—typically after input ingestion but before any response generation, tool invocation, or user-facing output. The prompt requires you to supply a [SAFETY_POLICY] document, a [HARM_CATEGORIES] schema, and the [INPUT] to classify. The output is a structured JSON object containing the classification label, confidence score, and separate uncertainty components. This is not a prompt for generating user-facing refusal messages or safe alternatives; it is a pure classification step that feeds downstream routing logic.

Do not use this prompt when you need a simple binary allow/block decision with no audit trail, when your latency budget cannot accommodate the additional tokens required for uncertainty decomposition, or when you lack a defined safety policy to ground the classification. This prompt also assumes you have a mechanism for handling high-uncertainty outputs—if you have no human review queue, no logging infrastructure, and no differentiated response policies based on confidence, the uncertainty estimates add complexity without value. Before deploying, ensure you have built the eval harness described in the companion Confidence Score Calibration Set Generation prompt to validate that your model's uncertainty estimates are well-calibrated against human judgments.

This template instructs the model to classify a user request across defined safety categories and produce explicit uncertainty estimates for each classification. The output distinguishes between epistemic uncertainty (the model knows it lacks knowledge) and aleatoric uncertainty (inherent ambiguity in the request itself). This separation is critical for safety systems that must route high-uncertainty cases to human review rather than making overconfident decisions on ambiguous inputs. Copy the template below and replace the square-bracket placeholders with your specific safety categories, policy definitions, and output schema requirements.

textCopy
You are a safety classifier. Your task is to evaluate the user request against defined safety policies and produce a classification with explicit uncertainty estimates.

## SAFETY POLICIES
[POLICY_DEFINITIONS]

## INPUT
User Request: [USER_REQUEST]
Context (if available): [CONTEXT]

## INSTRUCTIONS
1. For each safety category in [SAFETY_CATEGORIES], determine whether the request violates the policy.
2. For each classification, provide:
   - A binary label: VIOLATION or NO_VIOLATION
   - An epistemic uncertainty score (0.0 to 1.0): how uncertain you are due to missing knowledge, ambiguous policy wording, or insufficient context. 0.0 means completely certain, 1.0 means completely uncertain.
   - An aleatoric uncertainty score (0.0 to 1.0): how inherently ambiguous or edge-case the request itself is, regardless of your knowledge. 0.0 means the request is clear-cut, 1.0 means the request is fundamentally ambiguous.
   - A brief evidence statement citing the specific part of the request or policy that supports your classification.
3. If epistemic uncertainty exceeds [EPISTEMIC_THRESHOLD], explicitly state what additional information would reduce your uncertainty.
4. If aleatoric uncertainty exceeds [ALEATORIC_THRESHOLD], flag the request for human review and explain why the request is inherently ambiguous.
5. Provide an overall risk level: LOW, MEDIUM, HIGH, or CRITICAL based on [RISK_LEVEL_CRITERIA].
6. If the overall risk level is HIGH or CRITICAL, or if combined uncertainty exceeds [COMBINED_THRESHOLD], recommend escalation to human review.

## OUTPUT SCHEMA
Return a valid JSON object matching this schema:
[OUTPUT_SCHEMA]

## CONSTRAINTS
[CONSTRAINTS]

## EXAMPLES
[EXAMPLES]

Adaptation guidance: Replace [POLICY_DEFINITIONS] with your organization's specific safety policies, including examples of violations and edge cases. Define [SAFETY_CATEGORIES] as an explicit list such as ["hate_speech", "violence", "self_harm", "sexual_content", "illegal_activity"]. Set [EPISTEMIC_THRESHOLD] and [ALEATORIC_THRESHOLD] based on your risk tolerance—start with 0.3 for epistemic and 0.5 for aleatoric, then tune based on production data. The [OUTPUT_SCHEMA] should be a concrete JSON schema with field names, types, and constraints. For [EXAMPLES], include at least three few-shot demonstrations: one clear violation, one clear benign request, and one ambiguous edge case with appropriate uncertainty scores. The [CONTEXT] placeholder is optional but valuable when you have conversation history, user metadata, or session state that informs the safety decision.

Validation and testing: Before deploying, validate that the model produces parseable JSON matching your schema on every call. Build a golden test set of at least 50 cases spanning clear violations, benign requests, and deliberate edge cases. Measure whether epistemic uncertainty scores are higher on cases where the policy is genuinely ambiguous and lower on clear-cut cases. If the model produces overconfident classifications on edge cases, increase the [EPISTEMIC_THRESHOLD] or add more ambiguous examples to [EXAMPLES]. For high-risk deployments, route all cases where combined uncertainty exceeds [COMBINED_THRESHOLD] to human review and log the model's evidence statements for auditability.

This prompt is designed to be called as a synchronous safety gate before a user-facing model generates a response. The harness should treat the prompt output as a structured decision record, not as conversational text. The primary integration point is an API call that wraps the model request, validates the returned JSON against the expected schema, and routes the original user request based on the classification, confidence scores, and uncertainty estimates. The harness must handle three output paths: allow (high confidence, low risk), block (high confidence, high risk), and escalate (low confidence or high uncertainty). Each path requires different downstream handling, and the harness is responsible for enforcing that routing logic consistently.

Validation and retry logic must be strict. The harness should validate that the model output contains all required fields: classification, epistemic_uncertainty, aleatoric_uncertainty, risk_score, and rationale. If the JSON is malformed or missing required fields, implement a single retry with the error message injected into the prompt context. If the retry also fails, log the failure and route the request to a human review queue with a VALIDATION_FAILURE flag. Do not allow malformed outputs to default to 'allow' or 'block'. For numeric fields, validate that uncertainty scores are between 0 and 1, and that the risk score falls within the defined range. Any value outside these bounds should trigger a retry or escalation.

Logging and observability are critical because uncertainty outputs are used to calibrate thresholds over time. Log every classification decision with: the full prompt input, the raw model output, the validated JSON, the routing decision, the model ID and version, and a timestamp. If the decision is escalate, also log the specific uncertainty scores that triggered the escalation. This data feeds into calibration dashboards, A/B test analysis for threshold tuning, and audit trails for compliance review. Use structured logging (JSON) rather than plain text to enable querying by classification, risk_score range, or uncertainty bucket. For high-risk domains, ensure logs are immutable and retained according to your organization's data governance policies.

Model choice and latency directly impact this prompt's effectiveness. Use a model with strong instruction-following and JSON output capabilities. If your platform supports it, use structured output mode (e.g., JSON mode or function calling with a defined schema) rather than relying solely on the prompt to enforce format. This reduces malformed output rates and eliminates an entire class of retry failures. Latency is a concern because this prompt sits on the critical path before a user response. Set a timeout appropriate for your SLA—typically 2-5 seconds. If the model does not respond within the timeout, implement a degraded mode decision: either escalate to human review or apply a conservative default block policy, depending on your risk tolerance. Never silently allow a request when the safety classifier times out.

Threshold configuration should live in the application layer, not in the prompt. The prompt produces scores; the harness compares those scores against configurable thresholds to make routing decisions. Store thresholds in a configuration service or feature flag system so they can be adjusted without changing the prompt. This enables A/B testing of different threshold values, gradual rollout of stricter policies, and emergency tightening of safety gates during an incident. The harness should also track threshold boundary crowding—if a large percentage of requests fall within a narrow band around the decision boundary, that signals that the threshold may need adjustment or that the model's confidence calibration has drifted.

Human review integration is the final piece. When the harness routes a request to escalate, it should attach the full classification output, the original user input, and any session context to a review ticket. The review queue should be prioritized by risk_score and the magnitude of uncertainty. Reviewers should be able to see both the epistemic uncertainty (model doesn't know the policy well) and aleatoric uncertainty (the request is genuinely ambiguous) to inform their judgment. The harness should also support a feedback loop: when a human reviewer overrides a classification, log the override and use it as a labeled example for future calibration evaluation. This closes the loop between automated classification and human judgment, which is essential for maintaining safety system quality over time.