Safety Score with Evidence Citation Prompt

This prompt is for trust-and-safety analysts, compliance engineers, and ML platform teams who need to justify risk scores to reviewers or auditors. It produces a calibrated risk score with specific citations to policy clauses, input excerpts, and a reasoning chain. Use it when a downstream human reviewer, audit log, or compliance system requires traceable evidence for why a request was scored at a particular risk level. This prompt is not a real-time safety classifier. It is designed for post-hoc analysis, audit sample review, and building golden evaluation datasets where explainability matters as much as the score itself.

If you need a low-latency binary refuse/allow decision, use the Risk Threshold Gating Decision Prompt instead. This prompt assumes you already have a defined safety policy document and a user input that needs scoring. The output is a structured JSON object containing the risk score, a list of evidence citations, and a reasoning chain that connects specific policy clauses to excerpts from the input. The prompt is designed to be run on a sample of production traffic, not on every request. Typical use cases include: generating audit artifacts for a compliance review, creating a labeled dataset for fine-tuning a faster classifier, or investigating a specific user report where the original refusal decision needs to be explained.

Before using this prompt, ensure your safety policy document is versioned and accessible as part of the prompt context. The prompt will fail silently or produce hallucinated citations if the policy text is missing, outdated, or ambiguous. Also ensure that the input text is preserved in its original form—do not pre-redact or summarize it, as the citation mechanism depends on verbatim excerpts. For high-risk domains such as regulated finance or healthcare, always route the output through a human reviewer before it becomes part of a permanent audit record. The prompt is a reasoning aid, not a replacement for professional judgment.

This template is designed to be pasted directly into your system instructions for a trust-and-safety classification workflow. It forces the model to justify its risk score by anchoring it to specific evidence, making the output auditable for reviewers and downstream governance systems. The prompt is structured to accept a policy document, a user input, and a scoring scale as dynamic variables, ensuring it can be adapted to different organizational policies without rewriting the core logic.

textCopy
You are a safety classifier. Your task is to evaluate the user's request against the provided safety policy and produce a risk score with explicit evidence citations.

## POLICY
[POLICY_DOCUMENT]

## USER INPUT
[USER_INPUT]

## SCORING SCALE
[SCORING_SCALE]

## INSTRUCTIONS
1.  **Analyze the Input:** Read the user input and identify any clauses, requests, or implications that may violate the policy.
2.  **Map to Policy:** For each potential violation, cite the specific policy clause(s) from the POLICY section that apply. Use the exact clause identifier if available.
3.  **Cite Evidence:** Quote the specific excerpt(s) from the USER INPUT that triggered the policy match.
4.  **Determine a Score:** Assign a final risk score based on the provided SCORING SCALE. The score must be justified by the severity and number of policy matches.
5.  **Construct Reasoning:** Write a concise reasoning chain that connects the evidence citations to the assigned score.

## OUTPUT FORMAT
Return ONLY a valid JSON object with the following schema. Do not include any text outside the JSON block.
{
  "risk_score": <integer or float as defined by SCORING_SCALE>,
  "findings": [
    {
      "policy_clause": "<exact clause text or ID>",
      "input_excerpt": "<verbatim quote from user input>",
      "rationale": "<one-sentence explanation of the violation>"
    }
  ],
  "reasoning_chain": "<step-by-step reasoning from evidence to final score>",
  "refusal_recommendation": "<block | review | allow>"
}

## CONSTRAINTS
- If no policy violation is found, `risk_score` must be the lowest possible value on the scale, `findings` must be an empty array, and `refusal_recommendation` must be "allow".
- Every finding in the `findings` array must have a corresponding `policy_clause` and `input_excerpt`.
- Do not fabricate policy clauses. If a violation is ambiguous, state the ambiguity in the `rationale` and assign a score reflecting uncertainty.

To adapt this template, replace the square-bracket placeholders with your specific operational context. [POLICY_DOCUMENT] should contain the full text of your safety policy, with each clause having a unique identifier for easy citation. [SCORING_SCALE] must define a clear numeric or categorical scale (e.g., '1-5 where 5 is a critical violation'). Before deploying, run this prompt against a golden evaluation set to calibrate the scoring scale and ensure the model correctly maps edge cases to the right scores. A common failure mode is the model hallucinating policy clauses; mitigate this by strictly enforcing the output schema and validating that all cited clauses exist in the source [POLICY_DOCUMENT] via a post-processing script.

This prompt is designed to be called after an upstream safety classifier has flagged a request for review, or as the primary scoring step in a human-in-the-loop audit workflow. The implementation harness must treat the model's output as a structured artifact that requires validation before it can be trusted by downstream systems. The core integration pattern is: receive input text and applicable policy clauses, call the model with this prompt template, validate the returned JSON against the expected schema, check citation accuracy, and route the result to either an audit log, a review queue, or an automated decision engine based on the risk score and confidence level.

Validation and Retry Logic: The model returns a JSON object containing risk_score, confidence, policy_citations, input_excerpts, and reasoning_chain. Before accepting this output, validate that: (1) risk_score is a number between 0.0 and 1.0, (2) confidence is a number between 0.0 and 1.0, (3) every policy_citation references a clause ID that exists in the provided [POLICY_CLAUSES] input, (4) every input_excerpt is a verbatim substring of the original [INPUT_TEXT] (use exact string matching, not fuzzy search), and (5) the reasoning_chain contains at least one step linking each citation to its corresponding excerpt. If validation fails, retry up to two times with the validation errors appended to the [CONSTRAINTS] field. If the third attempt still fails validation, log the failure and route to a human review queue with a validation_failure flag. Never silently accept a malformed or unverifiable output.

Citation Accuracy Eval: Implement a post-processing check that verifies each cited excerpt genuinely supports the policy violation claim. This can be done with a secondary LLM call using a narrow eval prompt: 'Given this policy clause and this input excerpt, does the excerpt violate the clause? Answer only YES or NO with a brief reason.' If any citation fails this check, flag the entire score as citation_accuracy: failed and reduce the effective confidence by a configurable penalty factor (e.g., multiply by 0.5). This prevents the model from generating plausible-sounding but incorrect citations. Log all citation eval results alongside the original output for auditability.

Model Choice and Latency: This prompt benefits from models with strong instruction-following and structured output capabilities. GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro are suitable defaults. For high-throughput pipelines, consider using a smaller model (e.g., Claude 3 Haiku or GPT-4o-mini) for the initial scoring pass and escalating to a larger model only when confidence falls below 0.85 or when citation_accuracy fails. Always set response_format to JSON mode if the model API supports it, and include the full JSON schema in the API call rather than relying solely on the prompt's [OUTPUT_SCHEMA] description. Log the model ID, prompt version, and latency for every call to enable debugging and cost attribution.

Audit Trail and Downstream Routing: Every scored request must produce an immutable audit record containing: the original input, the policy clauses in effect, the full model output, the validation results, the citation eval results, the model ID and version, and the timestamp. Store this in a structured logging system (not just stdout). Route decisions based on the validated risk score: scores below 0.3 can be auto-approved, scores between 0.3 and 0.7 should surface a warning but allow the workflow to continue with logging, scores above 0.7 should block the request and require human review, and any output with citation_accuracy: failed or confidence below 0.6 should escalate regardless of the risk score. These thresholds must be configurable per policy category and per deployment environment. Never hardcode threshold values in the prompt or application code; load them from a configuration store that supports A/B testing and gradual rollout.