On-Call Alert Severity Classification Prompt Template

This prompt is designed for NOC and SRE teams drowning in monitoring alerts where manual triage is no longer sustainable. The core job-to-be-done is reducing alert fatigue by automatically classifying incoming alerts into severity levels (P0-Critical through P4-Informational) using few-shot examples that encode your incident response policy. You should use this prompt when your alert volume exceeds what on-call engineers can reasonably triage, when alert descriptions vary in format across monitoring tools (Datadog, PagerDuty, Grafana, CloudWatch), and when you need consistent severity decisions that downstream automation—like paging, ticket creation, or suppression rules—can consume programmatically. The few-shot approach adapts faster to new alert patterns than brittle rule chains, and it handles the natural language variation that regex-based classifiers miss.

The ideal user has access to a historical corpus of labeled alerts that reflect real severity decisions made by experienced on-call engineers. You need at least 10–20 representative examples per severity level, including edge cases where the severity is ambiguous. Required context includes your organization's severity definitions (what constitutes a P0 outage versus a P3 degradation), any suppression rules for known noisy alerts, and escalation criteria that trigger human review. The prompt expects structured input containing the alert title, description, affected service, and any available metadata like metric values or error counts. It produces a structured JSON output with the severity level, confidence score, and a brief justification that can be logged for auditability.

Do not use this prompt for real-time closed-loop remediation—it classifies alerts but does not act on them, execute runbooks, or restart services. It is also not a replacement for your alerting rules engine; use it to triage alerts that have already fired, not to decide whether an alert should fire in the first place. For high-stakes environments where a misclassification could delay response to a customer-impacting outage, always pair this prompt with a human review escalation path for low-confidence classifications (below 0.85 confidence) and for any P0 determination. The prompt works best as one component in a pipeline: classify, then route high-severity alerts to on-call engineers and suppress or batch low-severity alerts for later review. Before deploying, validate against a golden test set of at least 50 alerts with known severity labels, and monitor for severity drift as your services and failure modes evolve.

This prompt template is designed to be dropped directly into an AI gateway or model call. It uses few-shot examples to teach the model the difference between a critical outage (SEV0) and non-actionable noise (SEV3), and it enforces suppression rules for known maintenance windows. The template expects you to provide the raw alert payload, a set of labeled examples, and the current maintenance schedule. The model's only job is to return a single, valid JSON object that matches the output schema—no conversational preamble, no markdown fences.

textCopy
System: You are an on-call alert classifier for a production engineering team. Your task is to classify the provided monitoring alert into a severity level: SEV0, SEV1, SEV2, or SEV3. Use the provided examples to distinguish critical outages from noise. Apply suppression rules when the alert matches a known maintenance window or a non-actionable pattern. Return only a valid JSON object matching the output schema. Do not include any other text.

[EXAMPLES]

[SUPPRESSION_RULES]

User: Classify the following alert:

[ALERT_PAYLOAD]

Assistant:

To adapt this template, replace the square-bracket placeholders with your actual data. [EXAMPLES] should contain 3-6 few-shot demonstrations of alerts and their correct severity classifications, formatted as clear input-output pairs. [SUPPRESSION_RULES] should list active maintenance windows and known non-actionable alert fingerprints (e.g., specific hostnames or error messages that are safe to ignore). [ALERT_PAYLOAD] is the raw JSON or text of the incoming alert. The [OUTPUT_SCHEMA] is implied by the examples, but for production use, you should validate the model's JSON output against a strict schema (e.g., {"severity": "string", "reasoning": "string", "suppressed": "boolean"}) in your application code before taking any action. If the output fails validation, retry once with the error message appended to the prompt. If it fails again, escalate to a human on-call engineer.

Integrating this prompt into an operational harness requires treating it as a deterministic classification step within a broader incident management workflow, not as a standalone chatbot. The prompt should be called by a service that receives raw alerts from monitoring systems like PagerDuty, Datadog, or Prometheus Alertmanager. Before the prompt is invoked, the harness must pre-process the alert payload to extract and normalize the fields required by the prompt's [ALERT_PAYLOAD] placeholder, such as alert_name, service, metrics, thresholds, and runbook_url. This ensures the model receives a clean, structured input regardless of the source monitoring tool's format.

The application layer must enforce a strict contract around the model's output. Configure your API call to request a JSON object matching the [OUTPUT_SCHEMA] defined in the prompt, using structured output features like OpenAI's response_format with a JSON schema or function calling with tool_choice: "required". After receiving the response, a validation layer must confirm the presence and type of critical fields: severity (must be one of CRITICAL, WARNING, NOISE), confidence (a float between 0.0 and 1.0), and rationale (a non-empty string). If validation fails, implement a retry loop with a maximum of two attempts, feeding the validation error message back into the model's context as a [CORRECTION] instruction. For any CRITICAL classification, the harness should bypass standard routing and immediately trigger the escalation policy via the incident management API, logging the full prompt, response, and validation result for post-incident review.

To prevent alert fatigue and ensure consistent behavior, you must build an evaluation harness that runs offline against a golden dataset of historical alerts. This dataset should contain 100-200 labeled examples covering clear CRITICAL outages, ambiguous WARNING states, and known NOISE patterns like transient CPU spikes. Use an LLM-as-judge or a simple script to compare the model's classification against the ground truth, measuring precision and recall for each severity class. Pay special attention to false negatives for CRITICAL alerts, as these represent missed outages. Run this evaluation suite before deploying any prompt changes and integrate it into your CI/CD pipeline as a release gate. Finally, log every classification to an observability platform, attaching the alert_id, model_version, prompt_version, and confidence score, so you can track drift in model behavior and identify new alert patterns that require updating the few-shot examples.