Error Severity Scoring Prompt for Agent Steps

This prompt is for monitoring systems and agent orchestrators that must assign a consistent severity level to step failures. It is designed for operational dashboards, alerting pipelines, and post-execution analysis where heterogeneous errors from different tools, APIs, and models need a single, calibrated scoring method. Use it when you need to prioritize on-call response, trigger specific recovery workflows, or generate incident reports from agent execution traces. The prompt ingests a structured failure record—including the error message, the step's role in the overall plan, and any available downstream dependency information—and outputs a severity level (critical, high, medium, or low) with a blast radius estimate, data loss risk, and an impact assessment on subsequent steps.

Do not use this prompt for real-time step-level recovery decisions. It is an analytical scoring tool, not a retry or skip decision engine. If you wire this prompt directly into an agent's execution loop to decide whether to abort a workflow, you will introduce latency and a point of failure that is better served by a deterministic rule engine. Instead, use this prompt in offline or nearline analysis paths: post-execution trace review, batch incident report generation, or as a signal into a monitoring system that aggregates severity scores across many agent runs. The prompt assumes you have already classified the failure type (transient, permanent, permission, etc.) using a separate classification step. Pair it with the Agent Step Failure Classification Prompt Template to ensure the input to this scorer is well-structured.

The primary risk with severity scoring prompts is severity inflation, where the model overestimates the impact of a failure to appear cautious. To mitigate this, you must calibrate the prompt with clear definitions for each severity level, provide few-shot examples that demonstrate appropriate restraint, and run inter-rater consistency evals comparing model-assigned scores against a golden set of failures scored by your operations team. A secondary risk is context blindness, where the model fails to account for the blast radius because it doesn't understand your specific system architecture. Always include a concise dependency map in the [CONTEXT] field showing which downstream steps or services are affected by the failing step. Start with a human-reviewed calibration set of 20-30 failures before trusting this prompt in an automated pipeline.

This prompt template is the core instruction set for your severity scoring system. It is designed to be pasted directly into your monitoring harness or evaluation pipeline. The model is forced to reason about the failure's blast radius, data loss risk, and downstream impact before committing to a severity level, which prevents the common failure mode of severity inflation for transient errors.

textCopy
You are an agent reliability scorer. Your task is to analyze a failed agent step and assign a consistent severity level.

## INPUT
[AGENT_STEP_FAILURE_RECORD]

## CONTEXT
[AGENT_PLAN_CONTEXT]

## INSTRUCTIONS
1. Analyze the failure record against the provided plan context.
2. Determine the severity level based on the following criteria:
   - **critical**: The failure makes the entire plan unrecoverable, causes irreversible side effects, or risks significant data loss.
   - **high**: A core plan objective is now unachievable, or the failure blocks multiple downstream critical steps.
   - **medium**: The step can be skipped or replaced with a fallback without breaking core objectives, but it requires a plan modification.
   - **low**: The failure is transient, has no downstream impact, or can be resolved by a simple retry without plan changes.
3. Estimate the blast radius: the number of downstream steps directly impacted.
4. Assess data loss risk as `none`, `possible`, or `confirmed`.
5. Identify the specific downstream step IDs from the plan context that are now blocked or at risk.

## CONSTRAINTS
- Do not inflate severity for transient errors like timeouts or 429 rate limits unless retries are exhausted.
- A `critical` severity must be justified by irreversible harm, not just task importance.
- If the failure record is incomplete, note this in the `assessment_notes` field and score conservatively.

## OUTPUT_SCHEMA
You must return a single, valid JSON object with no additional text or markdown fences:
{
  "severity": "critical" | "high" | "medium" | "low",
  "blast_radius": <integer>,
  "data_loss_risk": "none" | "possible" | "confirmed",
  "impacted_step_ids": ["<string>", ...],
  "justification": "<concise, evidence-based reason for the score>",
  "assessment_notes": "<any caveats about incomplete data or low confidence>"
}

To adapt this template for your environment, replace the [AGENT_STEP_FAILURE_RECORD] placeholder with a structured object containing the error message, step ID, tool name, and raw output. The [AGENT_PLAN_CONTEXT] should include the original plan's dependency graph and the status of all preceding steps. If your agent framework uses different severity labels, adjust the enum values in both the instructions and the OUTPUT_SCHEMA, but keep the decision criteria anchored to concrete, observable consequences rather than subjective importance. After pasting this into your harness, immediately run it against your golden set of historical failures to calibrate the scoring against your team's operational definitions.

This prompt is not designed for real-time agent decision loops. It belongs in a post-execution analysis pipeline or a monitoring sidecar that observes agent traces asynchronously. The primary job is to produce consistent, structured severity scores across heterogeneous agent failures so that downstream alerting, reporting, and retrospective systems can operate on a normalized signal. Running this prompt in the hot path of an agent's error-recovery logic adds latency and token cost without improving the immediate recovery decision. Instead, call it after a step failure is logged, or batch it across multiple failures in a periodic analysis window.

Integration pattern: Wire the prompt as a consumer of your agent trace store. When a step execution record transitions to a failed or error terminal state, publish an event to a message queue or invoke a webhook. A lightweight worker—a cloud function, a background job, or a monitoring sidecar—pulls the failure record, assembles the prompt with the required [STEP_CONTEXT], [ERROR_PAYLOAD], and [DOWNSTREAM_DEPENDENCIES] placeholders, and calls the model. Validation layer: Before writing the output to your observability database, validate that severity is one of the allowed enum values (critical, high, medium, low), that blast_radius_estimate is a non-empty string, and that data_loss_risk is a boolean. Reject and log any response that fails schema validation; do not silently ingest malformed severity records. Retry logic: If the model call fails or returns invalid JSON, retry once with exponential backoff (1s, then 2s). After two failures, write a fallback record with severity: "unknown" and a validation_error note so the failure is still visible in your monitoring dashboards.

Model choice and cost control: Use a fast, cost-efficient model for this task—a lightweight instruction-tuned model or a small hosted model with JSON mode support. The prompt's output schema is flat and the reasoning required is classification, not generation. Avoid routing this to a large, expensive frontier model unless your severity decisions are demonstrably worse on smaller models and the cost difference is justified by alerting accuracy. Logging and audit: Store the full prompt, the model response, the validated severity record, and the trace ID of the original failure in a structured log. This audit trail is essential for calibrating severity thresholds over time and for debugging cases where a critical alert fired on a transient DNS blip. Human review gate: For any failure scored as critical, route the record to a human-review queue before triggering a pager alert. The model can inflate severity on novel failure patterns, and a quick human sanity check prevents 3 AM wake-ups for non-critical issues. After review, feed the corrected severity back into your eval dataset to improve future calibration.