Error Cluster Triage Prompt for On-Call Engineers

This prompt is designed for on-call engineers and incident responders who are facing an alert storm: a situation where dozens or hundreds of alerts fire simultaneously across multiple services, creating a high-noise, high-stress environment. The primary job-to-be-done is to rapidly reduce this noise into a structured, prioritized investigation queue. The ideal user is an SRE, DevOps engineer, or backend developer who has access to a raw stream of alerts (from PagerDuty, Opsgenie, Datadog, or similar) and needs to identify which clusters of alerts represent distinct incidents, which are duplicates, and which require immediate escalation to specialized teams.

To use this prompt effectively, you must provide the raw alert payloads, including error messages, source service names, timestamps, and any available stack traces or log snippets. The prompt instructs the LLM to group related errors by a calculated fingerprint, the originating service, and a defined timeframe, then assign a preliminary severity and suggest an owning team. The output is a triage summary, not a root cause analysis. It is a first-pass filter that transforms an unmanageable alert flood into a manageable list of investigation targets, allowing the responder to focus on the most critical and unique problems first.

Do not use this prompt for deep root cause analysis of a single, already-isolated error. It is not designed to trace a single stack frame back to a specific code commit or to analyze the performance characteristics of one slow query. For those tasks, pair this triage prompt with a more focused diagnostic prompt, such as the Stack Trace Root Cause Analysis Prompt Template, after the triage phase is complete. Using this prompt for a single error will result in an unnecessary and confusing grouping analysis. Its value is exclusively in the initial, high-level clustering of a large, noisy dataset of related and unrelated failures.

This template is designed to be copied directly into your alerting pipeline, Slack bot, or on-call runbook. It expects raw alert data, recent deployment events, and service topology context. Replace each square-bracket placeholder with live data before sending the prompt to the model. The output is a structured triage report that reduces dozens of alerts into a single prioritized investigation queue, with clear ownership assignments and severity justifications.

textCopy
You are an on-call triage assistant. Your task is to group a batch of production alerts into error clusters, assign severity, and recommend ownership. Follow these instructions exactly.

## INPUT DATA

### Alerts
[ALERTS_JSON_ARRAY]

### Recent Deployments (last 2 hours)
[DEPLOYMENTS_JSON_ARRAY]

### Service Dependency Map
[SERVICE_TOPOLOGY_JSON]

### On-Call Rotation
[ONCALL_ROTATION_JSON]

## OUTPUT SCHEMA

Return a valid JSON object with this structure:

{
  "triage_summary": {
    "total_alerts": <int>,
    "unique_error_fingerprints": <int>,
    "affected_services": [<string>],
    "time_window_start": <ISO 8601>,
    "time_window_end": <ISO 8601>,
    "highest_severity": "CRITICAL" | "HIGH" | "MEDIUM" | "LOW"
  },
  "clusters": [
    {
      "cluster_id": <string>,
      "fingerprint": <string>,
      "error_type": <string>,
      "error_message_excerpt": <string>,
      "count": <int>,
      "affected_services": [<string>],
      "first_seen": <ISO 8601>,
      "last_seen": <ISO 8601>,
      "severity": "CRITICAL" | "HIGH" | "MEDIUM" | "LOW",
      "severity_justification": <string>,
      "likely_owner_team": <string>,
      "owner_assignment_reason": <string>,
      "correlated_deployment": <string | null>,
      "correlation_confidence": "HIGH" | "MEDIUM" | "LOW" | "NONE",
      "recommended_action": "INVESTIGATE_NOW" | "MONITOR" | "ACKNOWLEDGE" | "SUPPRESS",
      "investigation_lead": <string>
    }
  ],
  "unclustered_alerts": [
    {
      "alert_id": <string>,
      "reason_unclustered": <string>
    }
  ]
}

## FINGERPRINTING RULES

1. Normalize error messages by removing timestamps, UUIDs, IP addresses, port numbers, and numeric IDs before fingerprinting.
2. Group alerts that share the same normalized error message and originate from the same service.
3. If two error types consistently appear together within a 5-minute window, consider merging them into a single cluster with a combined fingerprint.
4. Alerts that cannot be grouped with any other alert should appear in `unclustered_alerts` with a reason.

## SEVERITY RULES

- CRITICAL: Error rate exceeds 5% of total requests for the affected service, OR user-facing writes are failing, OR data loss is possible.
- HIGH: Error rate between 1% and 5%, OR a single service is returning errors for more than 10 minutes, OR a deployment correlates with the error spike with HIGH confidence.
- MEDIUM: Error rate below 1%, OR errors are limited to non-critical async paths, OR correlation with deployment is MEDIUM or LOW.
- LOW: Isolated errors with no user impact, OR errors in non-production environments.

## OWNERSHIP RULES

- Assign ownership based on the service that originates the error, using the on-call rotation data.
- If the error correlates with a recent deployment, assign ownership to the team that deployed.
- If multiple teams could be responsible, assign to the service owner and note the secondary team in `investigation_lead`.

## CONSTRAINTS

- Do not invent error messages, services, or deployments not present in the input data.
- If deployment correlation is uncertain, set `correlation_confidence` to LOW and do not fabricate a link.
- Every cluster must have a `severity_justification` that references specific thresholds or evidence.
- Preserve the original alert IDs so downstream systems can trace back to raw alerts.
- If the input contains fewer than 3 alerts, still produce valid output but note the low sample size in `triage_summary`.

## EXAMPLE

Input: 3 alerts from payment-service with "Connection refused" errors starting at 14:03, plus a deployment of payment-service at 14:01.
Output: Single cluster with fingerprint "payment-service_connection-refused", severity HIGH, correlated_deployment set to the deployment ID, correlation_confidence HIGH.

Adapt this template by replacing the four input placeholders with data from your monitoring and deployment systems. The [ALERTS_JSON_ARRAY] should contain alert objects with at minimum id, service, error_message, timestamp, and severity fields. The [DEPLOYMENTS_JSON_ARRAY] should list recent deployments with service, timestamp, and deployer_team. The [SERVICE_TOPOLOGY_JSON] should map services to their upstream and downstream dependencies. The [ONCALL_ROTATION_JSON] should map services to current on-call teams. If your alerting system uses different field names, adjust the fingerprinting and severity rules accordingly, but keep the output schema stable so downstream automation can consume the triage report reliably.

This prompt is designed to sit behind a triage API endpoint or a Slack/Teams bot command that on-call engineers invoke during an alert storm. The application layer is responsible for gathering the raw alert payload, normalizing it into the [ALERT_STREAM] placeholder format, and injecting the [SERVICE_CATALOG] and [ON_CALL_ROTATION] context. The model call should be wrapped in a function that enforces a strict timeout (e.g., 30 seconds) and a maximum token budget to prevent runaway costs during high-severity incidents when latency matters.

Validation and retry logic is critical because the output drives incident response actions. Before presenting the triage summary to the engineer, validate the JSON output against a schema that checks: (1) every cluster has a non-empty fingerprint, severity, and owner field; (2) severity values are constrained to the enum [CRITICAL, HIGH, MEDIUM, LOW]; (3) owner values match entries in the provided on-call rotation; and (4) the total_alerts_clustered count matches the input alert count. If validation fails, retry once with the validation errors appended to the [CONSTRAINTS] block. If the second attempt also fails, surface the raw model output alongside a warning that manual triage is required—never silently drop malformed triage results during an active incident.

Logging and observability should capture the full prompt, model response, validation result, and retry count for every invocation. Attach these to the incident channel or ticket so postmortem reviewers can audit the AI-assisted triage decisions. For high-severity incidents where the model assigns CRITICAL to any cluster, the harness should automatically page the designated owner and post the cluster summary to the incident channel without waiting for human acknowledgment. For MEDIUM and LOW clusters, queue them for review during the next on-call handoff. Model choice matters: use a model with strong JSON mode and low latency (e.g., Claude 3.5 Sonnet or GPT-4o with structured outputs enabled). Avoid models that struggle with strict schema adherence, since malformed triage output during an incident erodes trust and wastes time. If your organization requires data residency, deploy a local model that has been fine-tuned on your service catalog and alert taxonomy.

Tool integration can enhance this prompt by giving the model access to a lookup_recent_deploys(service_name, time_window) function or a query_incident_history(fingerprint) tool. If you add tools, update the [TOOLS] placeholder with the function schemas and instruct the model to call them before finalizing severity assignments. Be cautious: tool calls add latency and can fail during infrastructure incidents. Implement a tool-call timeout and a fallback that proceeds with the information already available in the alert stream. Human review is mandatory when the model assigns CRITICAL severity to a cluster it cannot confidently fingerprint, or when the alert stream contains security-related signals (e.g., authentication failures, permission escalations). In these cases, the harness should flag the cluster for mandatory human acknowledgment before any automated paging or remediation is triggered.