Alert Trigger Validation Prompt for Monitoring Systems

This prompt is built for SREs and platform engineers who need to validate alert behavior against known conditions. The core job-to-be-done is confirming that each alert rule triggers when it should, stays silent when it shouldn't, and has an accurate runbook linked to it. You bring the alert rule definitions, threshold values, and runbook documentation. The prompt produces a structured validation table that maps each alert condition to expected versus actual trigger behavior, flags false positives and false negatives, and cross-checks runbook accuracy. This is not a prompt for designing new alerts from scratch or for general observability health checks—it assumes you already have concrete alert definitions ready to test.

Before using this prompt, gather your inputs: the alert rule YAML or JSON definitions, the current threshold values, any silence or inhibition rules, and the corresponding runbook URLs or text. The prompt works best when you provide these as structured [ALERT_DEFINITIONS] and [RUNBOOK_CONTEXT] blocks. It will produce a validation table with columns for alert name, condition, expected trigger state, actual trigger state, match status, false positive/negative flags, and runbook accuracy notes. For high-severity alerts—P1 or P2 incidents—always require human review of the validation output before deploying changes. The prompt includes explicit instructions to flag any alert where the runbook is missing, stale, or references commands that no longer apply to the current service version.

Do not use this prompt when you need to design new alert rules, tune thresholds based on historical data, or evaluate alert fatigue across an entire fleet. Those are separate workflows covered by the Alert Fatigue Risk Assessment and Runbook Accuracy Verification prompts in this pillar. After running this validation, take the output table and feed discrepancies into your alert tuning process. If you find false positives, adjust thresholds or add aggregation windows. If you find false negatives, check your PromQL or metric query logic. If runbooks are inaccurate, update them before the alert fires in production. The next section provides the copy-ready prompt template you can adapt with your own alert definitions and runbook context.

This prompt template is the core engine for auditing your monitoring system's alert rules. It is designed to be copied directly into your AI harness, where you will replace the square-bracket placeholders with data from your observability stack. The prompt instructs the model to act as an SRE auditor, cross-referencing your defined alert conditions against actual trigger logs to produce a rigorous validation table. The primary output is a structured analysis that separates expected behavior from anomalies, giving you a clear picture of alert rule health.

markdownCopy
You are an SRE auditor validating the accuracy of monitoring alert rules. Your task is to compare a set of defined alert conditions against a log of actual alert trigger events. You will produce a structured validation table that identifies true positives, false positives, and false negatives, and you will cross-check the accuracy of any provided runbook links.

**INPUT DATA**

- **Alert Definitions:**
[ALERT_DEFINITIONS]

- **Trigger Event Log:**
[TRIGGER_EVENT_LOG]

- **Runbook Mappings (Optional):**
[RUNBOOK_MAPPINGS]

**OUTPUT SCHEMA**

You must respond with a valid JSON object conforming to this exact structure. Do not include any text outside the JSON object.

```json
{
  "alert_validation_report": {
    "analysis_metadata": {
      "analysis_timestamp": "[string] ISO 8601 timestamp of when this analysis was generated",
      "alert_definitions_count": "[integer] Total number of alert definitions provided",
      "trigger_events_count": "[integer] Total number of trigger events in the provided log"
    },
    "validation_table": [
      {
        "alert_definition_id": "[string] Unique identifier from the alert definition",
        "alert_name": "[string] Human-readable name of the alert",
        "defined_condition": "[string] The threshold or condition that should trigger the alert",
        "expected_trigger_count": "[integer] Number of times the alert should have triggered based on the event log",
        "actual_trigger_count": "[integer] Number of times the alert actually fired in the event log",
        "true_positives": "[integer] Events where the condition was met and the alert fired",
        "false_positives": "[integer] Events where the alert fired but the condition was not met",
        "false_negatives": "[integer] Events where the condition was met but the alert did not fire",
        "runbook_accuracy": {
          "runbook_present": "[boolean] Whether a runbook was mapped to this alert",
          "runbook_steps_accurate": "[boolean or null] Whether the runbook steps match the alert's diagnostic needs",
          "runbook_accuracy_notes": "[string or null] Explanation of any inaccuracies found"
        },
        "overall_status": "[string] One of: 'HEALTHY', 'MISFIRING', 'MISSING', 'UNVERIFIABLE'",
        "recommendation": "[string] A concise, actionable recommendation for tuning or investigation"
      }
    ],
    "summary": {
      "total_alerts_analyzed": "[integer]",
      "healthy_alerts": "[integer]",
      "misfiring_alerts": "[integer]",
      "missing_alerts": "[integer]",
      "unverifiable_alerts": "[integer]",
      "overall_assessment": "[string] A brief summary of the alert system's health and the most critical action to take"
    }
  }
}

CONSTRAINTS

• Base your analysis strictly on the provided [ALERT_DEFINITIONS] and [TRIGGER_EVENT_LOG]. Do not infer or assume external context.
• If a trigger event cannot be definitively matched to an alert definition, classify the alert as 'UNVERIFIABLE' and explain why in the recommendation field.
• For runbook accuracy, if no runbook is provided, set runbook_present to false and runbook_steps_accurate to null.
• If the data is insufficient to calculate any metric, use -1 for integer fields and explain the gap in the recommendation.
• Ensure all counts are internally consistent: true_positives + false_positives must equal actual_trigger_count.

To adapt this template for your environment, replace the placeholders with data from your monitoring stack. The [ALERT_DEFINITIONS] should be a structured list of your alert rules, including their IDs, names, and the exact PromQL, LogQL, or threshold condition. The [TRIGGER_EVENT_LOG] should be a time-bounded export of firing events from Alertmanager, PagerDuty, or your equivalent system. For the optional [RUNBOOK_MAPPINGS], provide a mapping of alert IDs to their runbook text or URLs. After running the prompt, always validate the output JSON against the schema before ingesting it into your dashboard or reporting system. For high-severity alerts, a human SRE should review the 'MISFIRING' and 'MISSING' classifications before any alert rule is tuned or silenced.

This prompt is designed to be integrated into a CI/CD pipeline or a scheduled SRE workflow, not run as a one-off chat. The primary integration point is a script or microservice that fetches alert rule definitions from your monitoring system (e.g., Prometheus, Datadog, PagerDuty APIs), retrieves recent incident timelines and metric data, and then constructs the prompt with concrete [ALERT_RULES], [THRESHOLD_DEFINITIONS], and [OBSERVED_BEHAVIOR] blocks. The model's output—a structured validation table—should be parsed and compared against a known schema before any action is taken.

For a robust harness, implement a multi-step validation pipeline. First, validate the model's JSON output against a strict schema that expects fields like alert_name, condition, expected_trigger, actual_trigger, false_positive, false_negative, and runbook_accuracy. Use a library like jsonschema in Python to reject malformed responses. Second, implement a retry layer with exponential backoff (max 3 attempts) if the output fails schema validation or if the model returns a refusal. Third, log every prompt and response pair, including the model's system_fingerprint, to an append-only audit store for post-incident review. For high-severity alerts (e.g., severity: critical), route the output to a human review queue in your incident management tool before auto-updating any alert rules.

Model choice matters here. Use a model with strong JSON mode and function-calling capabilities, such as gpt-4o or claude-3.5-sonnet, and set response_format to json_object with the expected schema. Avoid smaller, less instruction-following models that may conflate alert names or hallucinate threshold values. Do not feed raw, unbounded log streams into the prompt; pre-summarize the observed behavior into a concise timeline of state changes and metric values. The prompt is a validation and analysis tool, not a replacement for your monitoring system's rule engine. Always cross-check the model's runbook accuracy assessment against the actual runbook source of truth, as the model may misinterpret stale or ambiguous documentation.