Cumulative Policy Violation Scoring Prompt

This prompt is designed for trust-and-safety engineers and abuse detection system builders who need to move beyond isolated, single-turn content classification. Its job is to score the cumulative severity of policy violations across an entire multi-turn conversation session. It does this by analyzing each user and assistant turn, weighting contributions by recency, intent ambiguity, and the specific policy category violated, and then producing a structured violation ledger with a running cumulative score. The primary use case is detecting adversarial strategies that single-turn classifiers miss, such as gradual jailbreak attempts, context poisoning spread across multiple messages, or coordinated probing of policy boundaries.

Deploy this prompt when you have a defined safety policy taxonomy, a complete session transcript, and a human-reviewed calibration set for tuning the scoring weights. It assumes your system can provide the full conversation history as input and that you have established severity weights for each policy category (e.g., hate speech, self-harm, data exfiltration). The prompt is most effective as a secondary scoring layer behind a real-time, single-turn classifier, acting as a session-level auditor that triggers escalations, tool restrictions, or human review queues when a cumulative risk threshold is breached. It is not a replacement for real-time blocking classifiers, nor is it designed for latency-sensitive, per-turn gating.

Do not use this prompt for real-time, single-message intervention. It is too heavy for per-turn blocking and requires full session context to function correctly. Avoid deploying it without a calibration harness; unscored or uncalibrated policy weights will produce arbitrary, unactionable scores. This prompt is also inappropriate for domains where session history cannot be retained for privacy or compliance reasons, as its value depends entirely on cross-turn analysis. If your safety system only needs to block a single violating message, use a single-turn classification prompt. If you need to understand how a user is attempting to manipulate the model over time, this is the correct tool.

The following prompt template is designed to be dropped directly into your AI system's message array. It instructs the model to act as a session-level policy auditor, evaluating each turn in a conversation for violations and producing a structured ledger with a cumulative risk score. Before pasting, ensure you have access to the full session transcript, your organization's policy taxonomy, and a defined severity matrix. This prompt is not a single-turn classifier; it requires multi-turn context to detect probing patterns, gradual jailbreaks, and cumulative harm that individual messages might not trigger.

textCopy
SYSTEM:
You are a session-level policy auditor. Your task is to evaluate the entire conversation history for violations of the provided safety policies. You must track cumulative risk, not just isolated incidents.

INPUT SESSION:
[FULL_SESSION_TRANSCRIPT]

POLICY TAXONOMY:
[POLICY_CATEGORIES_AND_DEFINITIONS]

SEVERITY MATRIX:
[SEVERITY_LEVELS_WITH_EXAMPLES]

CONSTRAINTS:
[CONSTRAINTS]

OUTPUT_SCHEMA:
{
  "session_id": "string",
  "cumulative_risk_score": "number (0.0 to 1.0)",
  "breach_threshold": "number (0.0 to 1.0)",
  "threshold_breached": "boolean",
  "violation_ledger": [
    {
      "turn_index": "integer",
      "user_message": "string",
      "violation_detected": "boolean",
      "policy_id": "string | null",
      "severity": "string (LOW, MEDIUM, HIGH, CRITICAL) | null",
      "intent_ambiguity": "number (0.0 to 1.0)",
      "recency_weight": "number (0.0 to 1.0)",
      "turn_contribution_score": "number (0.0 to 1.0)",
      "evidence": "string | null"
    }
  ],
  "escalation_recommendation": "string (NONE, WARN, RESTRICT_TOOLS, HUMAN_REVIEW, TERMINATE)",
  "summary": "string"
}

INSTRUCTIONS:
1. Iterate through each turn in [FULL_SESSION_TRANSCRIPT].
2. For each user message, determine if it violates any policy in [POLICY_CATEGORIES_AND_DEFINITIONS]. Consider the context of prior turns when assessing intent.
3. Assign a severity based on [SEVERITY_MATRIX]. If no violation, severity is null.
4. Calculate `intent_ambiguity` as a score from 0.0 (clear malicious intent) to 1.0 (clearly benign). Ambiguous cases should be scored near 0.5.
5. Calculate `recency_weight` using an exponential decay function where the most recent turn has weight 1.0. Older turns decay based on their distance from the final turn.
6. Calculate `turn_contribution_score` as a product of severity weight, (1 - intent_ambiguity), and recency_weight.
7. Sum all `turn_contribution_score` values to produce the `cumulative_risk_score`, capped at 1.0.
8. Compare `cumulative_risk_score` against the `breach_threshold` to set `threshold_breached`.
9. Generate an `escalation_recommendation` based on the score and the nature of violations.
10. Write a concise `summary` explaining the risk trajectory.

Do not refuse to evaluate. If the session is benign, the score will naturally be 0.0.

To adapt this template, replace the square-bracket placeholders with your specific data. [FULL_SESSION_TRANSCRIPT] should be a JSON array of {“role”: “user”, “content”: “...”} objects. [POLICY_CATEGORIES_AND_DEFINITIONS] must be a structured list of your organization's disallowed content categories (e.g., hate speech, self-harm, illegal acts) with clear definitions. [SEVERITY_LEVELS_WITH_EXAMPLES] should map your severity labels to numeric weights for the scoring formula. [CONSTRAINTS] is where you inject hard rules, such as “ignore system messages” or “treat all mentions of [REDACTED_TOPIC] as CRITICAL.” The output schema is strict JSON; validate it in your application layer before acting on the escalation recommendation. For high-stakes domains, always route HUMAN_REVIEW and TERMINATE decisions to a human moderator queue and log the full prompt and response for auditability.

This prompt is designed to run once per user turn inside a stateful session handler. The application must maintain a rolling session ledger—an append-only log of prior violation records—and pass it into the prompt alongside the current turn's user message and assistant response. The prompt returns a structured JSON payload containing a cumulative score, a per-turn contribution, and a threshold breach alert. The application layer is responsible for persisting the returned ledger, comparing the cumulative score against configured risk thresholds, and triggering downstream actions such as tool restriction, human review queue insertion, or session termination.

Validation and retry logic is critical because a malformed JSON response can corrupt the session ledger. Implement a strict schema validator (using Pydantic, Zod, or JSON Schema) that checks: (a) the cumulative_score is a float between 0.0 and 1.0, (b) the per_turn_contributions array contains exactly one entry for the current turn with a non-negative contribution field, (c) all policy_category values match an allowed enum, and (d) the threshold_breach boolean is present. If validation fails, retry the prompt once with the error message appended as a [REPAIR_INSTRUCTION]. If the retry also fails, log the raw output, flag the session for human review, and fall back to the last known valid ledger state. Model choice matters: use a model with strong structured-output support (GPT-4o with response_format, Claude 3.5 Sonnet with tool-use mode, or equivalent). Avoid models known to drift on complex JSON under long context windows.

Logging and observability should capture the full prompt input, the raw model response, the validated ledger, and any threshold breach decisions. Emit structured log events that include session_id, turn_index, cumulative_score, breach_triggered, and action_taken. This audit trail is essential for calibrating thresholds, investigating false positives, and demonstrating policy enforcement to compliance reviewers. Human review integration should be gated on breach severity: low-severity breaches can queue for asynchronous review, while high-severity or rapid-escalation breaches should trigger real-time handoff with the full session context and violation ledger included in the review packet. Never allow an unvalidated or failed prompt response to silently downgrade a session's risk state—when in doubt, preserve the higher risk score and escalate.