Safety Classification Confidence Scoring Prompt Template

This prompt is for ML engineers and safety platform builders who need a safety classifier that outputs more than a binary label. It produces a calibrated confidence score alongside each classification, with explicit uncertainty quantification and evidence citations. Use this when you need to distinguish between clear violations and ambiguous edge cases, when you need scores you can threshold for A/B testing, and when downstream systems require audit trails explaining why a classification was made. This prompt belongs in the safety classification layer of your AI pipeline, before any refusal or routing decision. It assumes you have defined safety policies and harm categories. It does not make the final refusal decision; it provides the scored classification that a downstream gating system consumes.

Before adopting this prompt, verify that your pipeline actually needs a scored output. If you are building a simple content filter that only requires a binary allow/block decision, a lighter-weight classification prompt will be cheaper and faster. This prompt is the right choice when you need to tune refusal behavior without redeploying prompts—by adjusting a numeric threshold in your application logic instead. It is also the right choice when your safety team needs to audit decisions, because the evidence citations and confidence scores create a reviewable record. Do not use this prompt as the final refusal mechanism; always pair it with a separate gating system that consumes the structured output and applies your organization's risk thresholds.

The prompt expects you to supply a defined taxonomy of harm categories, a set of safety policies, and the user request to classify. It returns a structured object containing the predicted class, a confidence score between 0 and 1, an uncertainty breakdown, and citations to the specific policy clauses and input excerpts that support the classification. Wire this into your pipeline by placing it after input ingestion but before any model response generation. Route high-confidence violations to refusal, low-confidence cases to human review, and benign requests through to your normal generation flow. Start by running this prompt against a golden evaluation set to calibrate your thresholds before exposing it to production traffic.

This prompt template is designed to be pasted directly into your model's system instructions or a classification microservice. It instructs the model to evaluate a user request against your defined safety policies, assign a risk classification, and output a calibrated confidence score. The prompt enforces structured output, requires evidence citations, and separates epistemic uncertainty (what the model doesn't know) from aleatoric uncertainty (inherent ambiguity in the request). Before deploying, you must replace the square-bracket placeholders with your organization's specific policy text, harm categories, and output format requirements.

textCopy
You are a safety classification system. Your task is to analyze the following user request and determine whether it violates any defined safety policies. You must produce a risk classification, a calibrated confidence score, and specific evidence from the request that supports your assessment.

## SAFETY POLICIES
[POLICY_DEFINITIONS]

## HARM CATEGORIES
[HARM_CATEGORIES]

## INPUT
User Request: [USER_REQUEST]
Conversation Context: [CONVERSATION_CONTEXT]

## INSTRUCTIONS
1. Analyze the user request against each policy defined above.
2. Determine the primary harm category that applies, or classify as [BENIGN_CATEGORY_LABEL] if no policy is violated.
3. Assign a confidence score between 0.0 and 1.0 representing your certainty in this classification. A score of 1.0 indicates absolute certainty; 0.5 indicates maximum uncertainty. Do not use intermediate values as a hedge—be decisive.
4. Identify the specific spans of text in the user request that serve as evidence for your classification. If the request is benign, cite the absence of policy-violating content as evidence.
5. If your confidence is below [LOW_CONFIDENCE_THRESHOLD], explicitly state whether the uncertainty is due to missing context (epistemic) or inherent ambiguity in the request (aleatoric).

## OUTPUT FORMAT
Respond with a single JSON object conforming to this schema:
{
  "classification": {
    "primary_harm_category": "string",
    "violates_policy": boolean,
    "policy_clauses_triggered": ["string"]
  },
  "confidence_score": number,
  "uncertainty_type": "epistemic" | "aleatoric" | "none",
  "evidence": [
    {
      "text_span": "string",
      "rationale": "string"
    }
  ],
  "reasoning": "string"
}

## CONSTRAINTS
- Do not refuse to classify. Your output is for internal routing, not a user-facing response.
- If the request is ambiguous, assign your best-estimate classification and set the confidence score accordingly. Do not default to a benign classification when uncertain.
- Cite exact text from the user request in the evidence array. Do not paraphrase.

To adapt this template for production, start by replacing [POLICY_DEFINITIONS] with your organization's actual policy text, formatted as a numbered or bulleted list. Replace [HARM_CATEGORIES] with your taxonomy—such as hate_speech, self_harm, violence, sexual_content, illegal_activity, or custom categories relevant to your domain. Set [BENIGN_CATEGORY_LABEL] to the label your downstream routing logic expects for safe requests, such as none or benign. The [LOW_CONFIDENCE_THRESHOLD] should be tuned based on your risk tolerance; a starting value of 0.7 is common, but you should adjust this after running calibration evaluations against a golden dataset. The [CONVERSATION_CONTEXT] placeholder should be populated with prior turns if you are scoring multi-turn interactions; for single-turn classification, pass an empty string or omit the field. After adapting the template, run it against a held-out test set and measure Expected Calibration Error (ECE) to verify that confidence scores align with actual accuracy. If the model is consistently overconfident on misclassifications, add few-shot examples of ambiguous cases with appropriately low confidence scores to the prompt.

This prompt is not a standalone safety filter; it is a scoring engine that must be embedded in a broader application harness. The harness is responsible for managing the model request lifecycle, enforcing the output schema, validating the confidence score, and making the final routing decision. A typical production pipeline calls this prompt, parses the JSON response, checks that the confidence_score is a float between 0.0 and 1.0, and then compares the score against a configurable threshold to decide whether to allow, block, flag for review, or redirect the user input. The prompt itself should remain stateless; the harness owns session context, threshold configuration, and audit logging.

The implementation should include a validation layer that rejects malformed responses before they reach decision logic. At minimum, validate that: (1) the response is valid JSON, (2) the classification field matches an allowed enum of harm categories plus benign, (3) the confidence_score is a number in [0.0, 1.0], and (4) the evidence array contains non-empty strings that reference specific parts of the input. If validation fails, the harness should retry the prompt once with a stronger format reminder. If the retry also fails, the harness must escalate to a human review queue rather than silently defaulting to allow or block. For high-throughput systems, consider a circuit breaker that stops calling the model if the validation failure rate exceeds a threshold (e.g., 5% over a rolling window) and falls back to a conservative block-unknown policy.

Model choice and latency are critical harness decisions. This prompt works best with models that have strong instruction-following and JSON output capabilities. For real-time chat applications, use a fast model with strict response_format or tool-calling constraints to enforce the output schema. For async or batch safety review pipelines, a more capable model can be used with a longer timeout. The harness should log every request and response—including the raw prompt, model version, latency, confidence score, threshold applied, and final routing decision—to a structured logging system. This audit trail is essential for calibration analysis, threshold tuning, and compliance review. Never log the raw user input if it contains PII; instead, log a hashed or redacted identifier that can be correlated in a separate secure system.

Threshold configuration should be externalized from the prompt and managed in the harness as a feature flag or dynamic config. Start with a conservative threshold (e.g., block if confidence_score > 0.7 for high-severity categories) and use the audit logs to measure false-positive and false-negative rates against human reviewer judgments. The harness should support per-category thresholds, as a 0.6 score for self-harm may warrant immediate escalation while a 0.6 score for spam may only warrant a warning. Implement an A/B testing framework in the harness that can route a percentage of traffic to different threshold configurations and compare refusal rates, user satisfaction metrics, and safety incident reports between arms.

Human review integration is the harness's most important safety net. When the confidence score falls into a configured uncertainty band (e.g., 0.4–0.7), the harness should route the request to a review queue with full context: the original input, the model's classification, confidence score, evidence citations, and the threshold that triggered the review. The review interface should allow human reviewers to confirm or override the classification, and the harness must capture these overrides to feed back into calibration monitoring. For high-severity categories, consider requiring dual review or manager approval before an override is accepted. The harness should also track review queue depth and SLA compliance, alerting operations teams if the queue grows beyond capacity.

What to avoid: Do not use the raw confidence score as a probability in downstream calculations without calibration evidence. Do not hardcode thresholds in the prompt itself—thresholds are operational controls that must be changeable without prompt versioning. Do not skip the validation layer; malformed JSON from the model will crash your pipeline. Do not log sensitive user input in plaintext alongside safety scores. Do not treat this prompt as a replacement for content filtering at the infrastructure layer; it should complement, not replace, existing safety infrastructure. Finally, do not deploy without a human review escape hatch—every automated safety classifier will encounter edge cases that require human judgment.