Continuous Monitoring Coverage Audit Prompt Template

This prompt is designed for AI operations leads, platform engineers, and SREs who need to move from ad-hoc trace review to comprehensive continuous monitoring. The job-to-be-done is a systematic coverage audit: you provide a current monitoring configuration inventory (what alerts, dashboards, and SLOs you already have) and a catalog of known failure modes (what can go wrong in your AI system), and the prompt produces a structured gap report showing which failure dimensions lack alert coverage, which existing alerts may be misconfigured, and which gaps should be prioritized for new alert conditions. This is not a prompt for reviewing individual traces or debugging a single session. It is a strategic operations tool for assessing the completeness of your observability posture before an incident exposes blind spots.

The prompt assumes you already have production trace data flowing and a monitoring system in place—it does not help you set up tracing infrastructure from scratch. You need to bring two concrete inputs: a current monitoring configuration inventory (listing each alert, its trigger condition, evaluation window, severity, and the failure dimension it covers) and a failure mode catalog (enumerating known failure categories such as hallucination, tool-call errors, latency spikes, token cost anomalies, refusal rate changes, retrieval degradation, prompt injection attempts, and output format drift). The prompt works best when your monitoring inventory is detailed enough to map each alert to specific trace spans or metrics, and when your failure mode catalog is drawn from postmortems, red-team findings, and production incident history rather than generic lists. If you lack either input, the audit will produce vague recommendations that are not actionable.

Do not use this prompt as a substitute for incident response or root-cause analysis workflows. It is a planning and assessment tool, not a real-time diagnostic. The output is a coverage gap report with prioritization recommendations—it tells you where to invest monitoring effort next, but it does not configure alerts, deploy dashboards, or validate that new alert conditions work correctly. After running this audit, you should feed the prioritized gaps into sibling playbooks like Alert Condition Definition Prompt for Trace Patterns or Trace-Based SLO Threshold Evaluation Prompt Template to build the actual monitoring rules. Also avoid using this prompt when you have not yet established baseline observability—if you cannot describe your current monitoring configuration, start with instrumenting your traces and defining initial alerts before attempting a coverage audit.

This prompt template is designed to be copied directly into your AI observability workflow. It expects a structured inventory of your current monitoring configuration and a description of the failure modes you care about. Replace every square-bracket placeholder with your actual data before execution. The prompt instructs the model to produce a coverage gap report, identify unmonitored failure modes, and recommend new alert conditions with clear prioritization.

textCopy
You are an AI observability auditor. Your task is to evaluate the completeness of a production monitoring configuration for an AI system.

You will receive:
- A current monitoring configuration inventory describing what is already monitored.
- A failure mode catalog listing known or anticipated failure modes for the system.

Produce a structured coverage audit report with the following sections:

1. **Coverage Summary**: A table mapping each failure mode to its monitoring status (Covered, Partially Covered, Unmonitored).
2. **Gap Analysis**: For each unmonitored or partially covered failure mode, explain what trace dimensions, metrics, or signals are missing.
3. **Prioritization Matrix**: Rank gaps by risk severity (Critical, High, Medium, Low) based on potential user impact, frequency likelihood, and detectability without monitoring.
4. **Recommended Alert Conditions**: For the top five gaps, propose specific alert condition definitions including metric name, threshold value, evaluation window, and severity level.
5. **Coverage Heatmap**: A dimension-by-failure-mode matrix showing where monitoring density is weakest across trace dimensions (e.g., latency, token consumption, tool-call errors, hallucination rate, refusal rate, retrieval quality, output format compliance).

Constraints:
- Do not invent failure modes not present in the input catalog.
- If a failure mode cannot be monitored with available trace data, flag it as "Requires New Instrumentation."
- For each recommended alert, include a false-positive risk assessment (Low, Medium, High).
- Use the exact metric names and dimension labels from the monitoring configuration inventory where they exist.

Inputs:
[MONITORING_CONFIG_INVENTORY]
[FAILURE_MODE_CATALOG]

Output format: Valid JSON matching this schema:
{
  "coverage_summary": [{"failure_mode": string, "status": "Covered"|"Partially Covered"|"Unmonitored", "existing_monitors": [string]}],
  "gaps": [{"failure_mode": string, "missing_signals": [string], "trace_dimensions_affected": [string], "requires_new_instrumentation": boolean}],
  "prioritization": [{"failure_mode": string, "risk_severity": "Critical"|"High"|"Medium"|"Low", "rationale": string}],
  "recommended_alerts": [{"metric_name": string, "threshold": string, "evaluation_window": string, "severity": string, "false_positive_risk": "Low"|"Medium"|"High"}],
  "coverage_heatmap": {"dimensions": [string], "failure_modes": [string], "coverage_scores": [[number]]},
  "uninstrumented_gaps": [{"failure_mode": string, "required_instrumentation": string}]
}

Adaptation guidance: Replace [MONITORING_CONFIG_INVENTORY] with a structured description of your current dashboards, alert rules, and trace queries—include metric names, evaluation windows, and the trace dimensions each monitor covers. Replace [FAILURE_MODE_CATALOG] with your known failure modes drawn from incident postmortems, red-team findings, eval regressions, and user-reported issues. If your monitoring config is large, consider splitting it by subsystem and running the audit per subsystem. For high-risk production systems, always have a human SRE review the gap prioritization before acting on recommended alerts. Validate the output JSON against the schema before ingesting it into your alerting pipeline or runbook automation.

The Continuous Monitoring Coverage Audit prompt is designed to be called on a schedule—weekly or per deployment—not as a one-off chat interaction. Wire it into a scripted pipeline that fetches the current monitoring configuration inventory from your observability platform's API (e.g., Datadog monitors, Grafana alert rules, or a configuration database), assembles the [MONITORING_CONFIG_INVENTORY] input, and invokes the model. The output should be parsed as structured JSON and compared against the previous audit run to detect regressions in coverage. Store each audit result in a versioned audit log for trend analysis and compliance evidence.

Validation is critical because the prompt produces a coverage gap report with severity classifications. Implement a post-processing validator that checks: (1) every identified gap references at least one trace dimension from the input inventory, (2) severity values are constrained to the allowed enum [CRITICAL, HIGH, MEDIUM, LOW], (3) each recommendation includes a concrete alert condition template, and (4) the output schema matches the expected JSON structure. If validation fails, retry once with the validation errors appended to the [CONSTRAINTS] field. After two failures, log the raw output and escalate to the on-call AI operations engineer. For high-severity gaps (CRITICAL or HIGH), route the finding to a human review queue before any alert condition is deployed to production—this is a safety gate, not optional.

Model choice matters here. Use a model with strong structured output support and a large context window (e.g., Claude 3.5 Sonnet or GPT-4o with structured outputs enabled) because the monitoring inventory can be large and the output schema is complex. Set temperature=0 to maximize consistency across audit runs. Log every invocation with the full prompt, response, validation result, and any retry attempts to your trace store. This creates an audit trail that proves coverage reviews are happening and captures the reasoning behind gap prioritization. Avoid wiring this directly into an auto-remediation pipeline that creates or modifies alert rules without human approval—coverage gaps often require architectural decisions, not just configuration changes.