Tool Error Severity Classification Rubric Prompt

This prompt is designed for incident management and platform reliability teams that need a consistent, repeatable method for classifying tool failures into severity tiers: critical, major, minor, or cosmetic. The core job-to-be-done is translating raw error telemetry, user impact signals, and recovery timeline estimates into a structured severity label that triggers the correct response SLA and escalation policy. Use this prompt when you have already instrumented your tool calls with error codes, latency metrics, and user-facing impact data, and you need to remove human variability from the initial triage step. The ideal user is an SRE, on-call engineer, or incident commander who is building or refining an automated incident response pipeline where tool failures must be classified before they reach a human responder or an automated remediation runbook.

Do not use this prompt for real-time agent decision loops where sub-millisecond classification latency is required; the model inference time makes it unsuitable for inline circuit-breaker decisions or per-request routing. This prompt also fails when the error context is too sparse—if you cannot provide at least an error type, a user impact description, and a recovery estimate, the classification will be unreliable and should default to a human triage step. The prompt assumes you have a defined SLA framework with clear response time targets per severity level. If your organization lacks these definitions, the output will be a severity label without an operational contract, which undermines the entire purpose. Wire this prompt into your incident pipeline as a post-detection classification step, not as a replacement for error detection itself.

Before deploying this prompt, ensure your error telemetry pipeline can populate the required input placeholders with structured data. The prompt expects concrete values for error characteristics (e.g., HTTP 503, timeout, partial response), user impact scope (e.g., single user, entire region, read-only vs. write failure), and recovery timeline (e.g., self-healing within seconds, requires manual intervention, unknown duration). If any of these inputs are missing or inferred, the classification confidence drops, and you should route the incident to a human for review. The next section provides the copy-ready prompt template with all required placeholders and adaptation guidance.

This prompt template standardizes how your incident management team classifies tool errors. Replace each square-bracket placeholder with your specific incident data before sending it to the model. The template produces a severity classification (critical, major, minor, cosmetic) based on error characteristics, user impact, and recovery timeline, mapped to your team's response SLAs. Use this when you need consistent, defensible severity decisions across on-call rotations and incident responders.

textCopy
You are an incident severity classifier for a production AI system. Your job is to assign a severity level to a tool error based on the evidence provided. You must follow the classification rubric exactly and explain your reasoning.

## CLASSIFICATION RUBRIC

**CRITICAL (P0)**
- Complete tool unavailability affecting all users
- Data loss, corruption, or unauthorized access confirmed or suspected
- Recovery timeline unknown or exceeds 4 hours
- No viable fallback or workaround exists
- Response SLA: Immediate escalation, page on-call, begin incident within 5 minutes

**MAJOR (P1)**
- Significant degradation affecting >25% of requests
- Partial feature loss with user-visible errors
- Recovery expected within 1-4 hours
- Fallback available but degraded (stale data, reduced functionality)
- Response SLA: Escalate within 15 minutes, begin incident within 30 minutes

**MINOR (P2)**
- Intermittent errors affecting <25% of requests
- Latency increase within 2x baseline but still functional
- Recovery expected within 1 hour
- Workaround available with minimal user impact
- Response SLA: Create ticket, address within business hours or next 4 hours

**COSMETIC (P3)**
- Error messages visible but no functional impact
- Deprecation warnings, non-breaking schema changes
- Recovery not time-sensitive
- No user-visible degradation
- Response SLA: Log for review, address in next maintenance window

## INPUT DATA

**Error Details:**
[ERROR_TYPE]
[ERROR_MESSAGE]
[HTTP_STATUS_CODE]
[TOOL_NAME]
[ERROR_FREQUENCY]
[ERROR_DURATION]

**User Impact:**
[AFFECTED_USER_COUNT_OR_PERCENTAGE]
[USER_VISIBLE_SYMPTOMS]
[WORKAROUND_AVAILABLE]

**Recovery Information:**
[ESTIMATED_RECOVERY_TIME]
[FALLBACK_TOOL_AVAILABLE]
[DATA_INTEGRITY_STATUS]

**Additional Context:**
[DEPENDENCY_GRAPH_NOTES]
[RECENT_CHANGES]
[HISTORICAL_PATTERN]

## OUTPUT SCHEMA

Return a JSON object with this exact structure:
{
  "severity": "critical|major|minor|cosmetic",
  "confidence": 0.0-1.0,
  "rationale": "Brief explanation mapping evidence to rubric criteria",
  "response_sla": "The SLA action from the matched severity level",
  "borderline_considerations": ["Any factors that nearly pushed this to a different severity"],
  "recommended_immediate_action": "What the responder should do first"
}

## CONSTRAINTS

- If evidence spans multiple severity levels, choose the highest applicable level
- If data integrity status is unknown or suspect, default to CRITICAL
- If user impact cannot be determined, state this in rationale and flag for human review
- Do not downgrade severity based on recovery optimism unless recovery is confirmed
- If error frequency is unknown, assume worst case for the error type

## BORDERLINE CALIBRATION CASES

Use these examples to calibrate your judgment:

**Case 1: Partial outage with fast recovery**
- 30% error rate, recovery confirmed within 30 minutes, no data loss
- Classification: MAJOR (not critical because recovery is imminent; not minor because impact exceeds 25%)

**Case 2: Silent data corruption**
- 2% error rate, but data integrity checks failing, recovery unknown
- Classification: CRITICAL (data integrity concern overrides low error rate)

**Case 3: Cosmetic error during peak traffic**
- Deprecation warning on 100% of requests, no functional impact, users complaining about log noise
- Classification: COSMETIC (no functional impact, but note user sentiment in borderline_considerations)

Now classify the error in the INPUT DATA section above.

After copying the template, replace each placeholder with data from your monitoring systems, error logs, and incident context. The [ERROR_TYPE] should use a consistent taxonomy (e.g., timeout, 5xx, connection_refused, schema_mismatch). The [DATA_INTEGRITY_STATUS] field is especially important—if you cannot confirm data integrity, leave it as unknown and the prompt will default to CRITICAL. The calibration cases at the bottom of the prompt help the model handle borderline situations consistently. If your team uses different severity names or SLA timings, edit the rubric section directly rather than trying to override it with additional instructions.

Before deploying this prompt into an automated classification pipeline, test it against at least 10 historical incidents where you know the correct severity. Pay special attention to cases where error frequency is low but data integrity is at risk, and cases where recovery is fast but user impact is high. These are the situations where human responders often disagree, and the prompt's calibration cases are designed to produce consistent decisions. If the model's confidence score falls below 0.7, route the classification to a human for review rather than auto-escalating.

This prompt is designed to be called programmatically as part of an incident management pipeline, not as a standalone chat interaction. The typical integration point is a webhook receiver or event bus consumer that fires when a new tool error event is ingested. The harness should construct the prompt by injecting the error payload into the [ERROR_LOG] placeholder, the affected service's dependency map into [DEPENDENCY_GRAPH], the current incident state into [ACTIVE_INCIDENT_CONTEXT], and the organization's SLA definitions into [RESPONSE_SLA_TABLE]. The model call should be made with temperature=0 and a strict JSON mode enabled to ensure the output matches the expected severity classification schema without creative drift.

Before the classification result is acted upon, the harness must run a validation layer. Check that the returned severity_level is one of the allowed enum values (critical, major, minor, cosmetic), that the confidence_score is a float between 0.0 and 1.0, and that the evidence_summary array contains at least one entry mapped to a specific log line or metric. If validation fails, retry once with the same payload and an explicit instruction appended to the retry prompt: Your previous output failed schema validation for field [FIELD_NAME]. Return a valid JSON object. If the retry also fails, route the event to a human triage queue rather than silently defaulting to a lower severity. For critical and major classifications where confidence_score < 0.85, the harness should automatically escalate to a human on-call reviewer via your incident management platform's API (e.g., PagerDuty, Opsgenie) and attach the full classification payload as a note to the incident.

Log every classification call as a structured event with the input error signature, the model's raw response, the validated output, and the routing decision. This audit trail is essential for calibrating the rubric over time. Run weekly evaluations against a golden dataset of 50-100 manually classified error events, paying special attention to borderline cases where major and critical overlap—these are where the model is most likely to misclassify and where human review gates should be tightest. Avoid wiring this prompt directly to automated remediation actions (like circuit breaker flips or service restarts) without a human-in-the-loop approval step for any severity above minor.