Severity Scoring Rubric Prompt for Escalation Triage

This prompt is for safety engineers, compliance officers, and trust and safety teams who need a consistent, calibrated method for scoring the severity of guardrail violations before they enter a human review queue. The job-to-be-done is not just classification, but calibrated triage: converting a messy detection event into a structured severity score, a rationale grounded in organizational impact levels, and a recommended priority that downstream systems and human reviewers can act on. The ideal user is someone who already has a guardrail detection system producing violation records and needs to add a repeatable severity layer on top, rather than relying on ad-hoc reviewer judgment that drifts over time and across individuals.

Use this prompt when you have a defined set of organizational impact levels (e.g., 'P0-Critical Safety', 'P1-High Regulatory Risk', 'P2-Policy Violation', 'P3-Minor') and a library of golden-labeled examples to calibrate against. The prompt expects a violation record as input, including the policy that was triggered, the content or action in question, and any surrounding context. It is designed to be run after detection and before the evidence package is assembled for human review, so the severity score can inform routing priority and reviewer assignment. Do not use this prompt for real-time blocking decisions where latency is measured in milliseconds; it is a triage and queue-management tool, not an inline enforcement mechanism. Do not use it without first running calibration tests against your golden set to ensure the rubric's language maps correctly to your organization's risk tolerance.

The primary constraint is inter-rater consistency. The prompt includes calibration instructions that reference your golden-labeled examples, and you should evaluate its output against those examples before deployment. If your organization lacks a labeled severity dataset, invest time in creating one with at least 50–100 examples that span your severity levels and edge cases. The prompt also requires human review for any violation scored as high-severity, and you should log every severity decision with the rationale for auditability. Next, move to the prompt template section to copy the base structure, then to the implementation harness to wire it into your escalation pipeline with validation and retries.

The following prompt template is designed to be copied directly into your prompt management system, IDE, or orchestration layer. It takes a structured violation record and a defined severity rubric as input, then produces a consistent severity score with supporting rationale. The template uses square-bracket placeholders for all dynamic inputs, making it straightforward to parameterize in code. Before using this in production, you must define your own severity levels, impact dimensions, and calibration examples that reflect your organization's risk tolerance and operational realities.

textCopy
You are a severity scoring specialist for an AI safety operations team. Your task is to assign a calibrated severity score to a guardrail violation using the provided rubric.

## INPUT

### Violation Record
[VIOLATION_RECORD]

### Severity Rubric
[RUBRIC_DEFINITION]

### Calibration Examples
[CALIBRATION_EXAMPLES]

## INSTRUCTIONS

1. Analyze the violation against each dimension in the rubric.
2. Consider the context, affected users, potential harm, and reversibility of the violation.
3. Assign a severity score that maps to the defined levels in the rubric.
4. Provide a clear rationale that references specific rubric criteria and evidence from the violation record.
5. If the violation spans multiple severity dimensions, explain how you weighted each.
6. Flag any ambiguity or missing information that prevented a higher-confidence assessment.

## OUTPUT SCHEMA

Return a JSON object with this exact structure:
{
  "severity_score": "string (must match one of the defined rubric levels)",
  "confidence": number (0.0 to 1.0),
  "rationale": "string (explain which rubric criteria were met and why)",
  "impact_assessment": {
    "affected_users": "string (scope and characteristics of affected users)",
    "potential_harm": "string (nature and severity of possible harm)",
    "reversibility": "string (how easily the violation's effects can be undone)"
  },
  "missing_information": ["string (list any gaps that affected scoring)"],
  "escalation_recommendation": "string (immediate_review | standard_review | low_priority | no_escalation)"
}

## CONSTRAINTS

- Do not invent or assume violation details not present in the input.
- If the rubric does not cleanly fit the violation, explain the mismatch rather than forcing a score.
- For violations involving [RISK_LEVEL] risk categories, flag for immediate human review regardless of computed score.
- If confidence is below 0.7, recommend escalation even if the computed score is low.
- Do not include markdown formatting inside JSON string values.

To adapt this template for your environment, start by replacing [RUBRIC_DEFINITION] with your organization's actual severity levels, impact dimensions, and scoring criteria. A well-formed rubric defines clear boundaries between levels (e.g., critical, high, medium, low) with concrete examples of what constitutes each. The [CALIBRATION_EXAMPLES] placeholder should contain 3-5 golden-labeled violation examples that demonstrate correct scoring across your severity spectrum. These examples serve as few-shot anchors that dramatically improve scoring consistency. The [RISK_LEVEL] placeholder in the constraints section should be replaced with your highest-risk categories (e.g., "child safety," "financial fraud," "PII exposure") that always warrant human review regardless of the computed score.

Before deploying this prompt, validate it against a held-out set of 20-50 violations that your team has independently scored. Measure inter-rater agreement between the prompt's scores and your human reviewers using Cohen's kappa or a similar metric. Pay particular attention to boundary cases where violations sit between two severity levels—these are where scoring drift most commonly occurs. If you observe systematic over-scoring or under-scoring, adjust the rubric language and calibration examples rather than adding more instructions to the prompt. After deployment, log every severity score alongside the eventual human adjudication outcome to build a feedback loop for continuous rubric improvement.

The Severity Scoring Rubric Prompt is not a standalone tool; it is a decision component inside a larger escalation pipeline. The prompt expects a structured violation record as input and returns a calibrated severity score with rationale. To integrate this into an application, you must first ensure the upstream guardrail detection system produces a clean, structured violation object containing at minimum the policy violated, the offending content excerpt, and the surrounding context. The scoring prompt should be called synchronously after detection and before the case is routed to a human review queue, so that the severity score can inform queue priority and SLA assignment.

Wire the prompt into a dedicated scoring service with the following contract: Input is a JSON object with violation_type, policy_id, content_excerpt, context_snippet, and optional user_history_summary. Output must conform to a strict schema: { severity_score: integer (1-5), severity_label: string, rationale: string, impact_level: string, confidence: float }. Implement a post-processing validator that rejects any output where severity_score is outside the 1-5 range, confidence is below a configurable threshold (e.g., 0.7), or required fields are missing. On validation failure, retry once with the same input and an added instruction: Your previous output failed schema validation. Ensure all required fields are present and within specified ranges. If the retry also fails, log the raw output and escalate the case to a high-priority review queue with a SCORING_FAILURE flag so a human can assign severity manually. This prevents a prompt formatting error from silently dropping a high-severity violation into a low-priority queue.

For model choice, use a model with strong instruction-following and structured output capabilities. The prompt relies on the model internalizing a multi-factor rubric (impact, urgency, regulatory exposure, user harm potential) and applying it consistently. If your deployment supports structured output modes (e.g., JSON mode, function calling with a strict schema), use them instead of relying solely on the prompt's formatting instructions. This reduces repair-loop frequency. Log every scoring call with the input violation ID, the raw model output, the validated score, and the retry count. This audit trail is essential for calibrating the rubric over time and for demonstrating consistent severity assignment during compliance reviews. Finally, periodically run a golden-set evaluation: a labeled dataset of violations with known severity scores, run through the prompt, with inter-rater agreement measured against human adjudicators. Drift in agreement signals that the rubric language needs recalibration or that the model's behavior has shifted.