Production Anomaly Detection Alert Prompt Template

This prompt is designed for AI SREs and platform engineers who need to detect statistical anomalies in production trace data before they become incidents. Use it when you have aggregated trace metrics—token consumption, latency distributions, error rates—and a historical baseline, and you need the model to identify deviations, score their severity, and pinpoint the affected spans. The ideal user is an engineer or operator responsible for AI service reliability who already has a monitoring pipeline that pre-aggregates raw trace data into time-series windows. The prompt's job is to perform multivariate pattern analysis on those windows, catching drift that simple threshold-based alerting would miss.

This prompt belongs in a continuous monitoring pipeline, not a one-off debugging session. It expects structured input: a current observation window with metric values, a historical baseline window for comparison, and configuration parameters like sensitivity thresholds and minimum deviation requirements. The output is a structured anomaly report containing severity scores, affected span identifiers, and drift descriptions. Do not use this prompt for real-time per-request alerting—it is designed for aggregated windows, not individual traces. Do not use it as a replacement for static threshold alerts; it complements them by detecting subtle multivariate shifts. Do not use it when you lack a reliable historical baseline, as the model cannot establish normal behavior without one.

Before deploying this prompt, ensure your trace aggregation pipeline produces consistent window sizes and metric definitions. Validate that your baseline windows are representative of normal operation and free from known incidents. Wire the prompt into a scheduled evaluation loop—every 5 or 15 minutes is typical—and route its output to your alerting platform with severity-based routing rules. The next step after reading this section is to review the prompt template and adapt the placeholders to your specific trace schema and monitoring infrastructure.

This prompt template is designed to be copied directly into your AI operations harness. It expects production trace data, a statistical baseline, and drift sensitivity parameters as inputs. The model's job is to identify anomalous spans, compute anomaly scores, and assign severity ratings—not to generate generic summaries or conversational explanations. Every placeholder in square brackets must be replaced with real production data before the prompt is sent to the model. The template is structured to produce a machine-readable JSON output that can feed directly into your alerting pipeline, dashboard, or incident response workflow.

textCopy
You are an AI operations anomaly detector. Your task is to analyze production trace data against a provided baseline and identify statistical anomalies across token consumption, latency distributions, and error rates.

## INPUT

### Trace Data
[TRACE_DATA]

### Baseline Statistics
[BASELINE_STATS]

### Drift Sensitivity Configuration
[DRIFT_SENSITIVITY_CONFIG]

## OUTPUT SCHEMA

Return a valid JSON object with this exact structure:
{
  "anomalies_detected": boolean,
  "anomaly_count": integer,
  "anomalies": [
    {
      "anomaly_id": "string",
      "affected_spans": ["span_id_1", "span_id_2"],
      "metric_type": "token_consumption | latency | error_rate",
      "observed_value": number,
      "baseline_value": number,
      "deviation_z_score": number,
      "anomaly_score": number (0.0 to 1.0),
      "severity": "critical | high | medium | low | informational",
      "timestamp": "ISO 8601",
      "description": "Brief factual description of the deviation"
    }
  ],
  "summary": {
    "total_spans_analyzed": integer,
    "anomalous_span_percentage": number,
    "highest_severity": "critical | high | medium | low | informational | none",
    "primary_affected_metric": "token_consumption | latency | error_rate | none"
  }
}

## CONSTRAINTS

1. Only flag a span as anomalous if its z-score exceeds the threshold defined in [DRIFT_SENSITIVITY_CONFIG].
2. Severity must be derived from the combination of z-score magnitude and the metric's business criticality as defined in [CONSTRAINTS].
3. If no anomalies are detected, set "anomalies_detected" to false, "anomaly_count" to 0, and return an empty "anomalies" array. Do not fabricate anomalies.
4. Every anomaly description must reference the specific metric, observed value, baseline value, and affected span IDs. Do not use vague language.
5. Do not include remediation advice, root-cause speculation, or conversational commentary in the output.
6. If the input trace data is malformed, incomplete, or missing required fields, return a JSON object with {"error": "INVALID_INPUT", "detail": "<specific issue>"} and stop.

## EXAMPLES

### Example 1: Latency Anomaly
Input: A span with P99 latency of 4500ms against a baseline P99 of 1200ms with z-score threshold of 3.0.
Expected output includes: severity "critical", metric_type "latency", z-score > 3.0, anomaly_score > 0.8.

### Example 2: No Anomalies
Input: All spans within 1.5 standard deviations of baseline across all metrics.
Expected output: anomalies_detected false, anomaly_count 0, highest_severity "none".

### Example 3: Malformed Input
Input: Trace data missing span duration fields.
Expected output: {"error": "INVALID_INPUT", "detail": "Trace data missing required field: span_duration_ms"}.

## RISK_LEVEL
[RISK_LEVEL]

If RISK_LEVEL is "high", you must additionally flag any anomaly with severity "critical" or "high" with a field "requires_immediate_review": true and include the affected service name from the trace metadata.

To adapt this template for your environment, start by replacing [TRACE_DATA] with a structured representation of your production spans—typically a JSON array containing span IDs, timestamps, durations, token counts, error flags, and service metadata. Replace [BASELINE_STATS] with precomputed statistical baselines including mean, median, standard deviation, and percentile values for each metric you monitor. The [DRIFT_SENSITIVITY_CONFIG] placeholder should contain your z-score thresholds per metric type and any metric-specific criticality weights. The [CONSTRAINTS] block should define your severity mapping rules—for example, a z-score above 4.0 on latency might map to "critical" while the same z-score on token consumption might map to "high" depending on your SLO definitions. The [RISK_LEVEL] placeholder accepts "low", "medium", or "high" and controls whether the output includes the additional immediate-review flagging field. Before deploying this prompt to production, validate that your trace data schema matches the expected span fields referenced in the constraints and examples. Run this prompt against a set of known-clean traces and known-anomalous traces to confirm that anomaly scores and severity ratings align with your operational expectations. If the prompt will feed directly into an automated alerting pipeline, add a post-processing validation step that confirms the output JSON conforms to the schema before triggering any alerts.

Integrating this prompt into a production system requires a harness that treats the prompt as a stateless function within a broader detection pipeline. The harness is responsible for fetching the required baseline metrics, assembling the trace window data, invoking the model, and then validating the structured output before any alert is fired. Because anomaly detection directly influences on-call pager notifications, the harness must be designed to fail closed: if the model call fails, returns unparseable JSON, or produces scores that fail validation, the system should log the failure and escalate to a human operator rather than silently dropping the alert or generating a false positive. The prompt itself is the analytical core, but the surrounding code handles data sourcing, retries, schema enforcement, and severity routing.

A concrete implementation should start with a data assembly function that queries your observability backend (e.g., Grafana, Datadog, or an internal trace store) for the metrics specified in the prompt's [BASELINE_METRICS] and [TRACE_WINDOW] placeholders. This function must enforce a strict lookback window and return a structured JSON object matching the expected input schema. Next, a model invocation wrapper should handle retries with exponential backoff (max 3 attempts) and enforce a strict timeout (e.g., 10 seconds) to prevent a hung model call from delaying incident response. The model's raw output must pass through a JSON schema validator that checks for required fields (anomaly_score, affected_spans, severity, drift_indicators) and enforces type constraints (e.g., anomaly_score must be a float between 0.0 and 1.0). Any output that fails validation should trigger a ModelOutputError event in your logging system, capturing the raw response for later debugging. For high-severity anomalies (score > 0.85), the harness should automatically attach the validated output to an incident ticket and page the on-call rotation, while lower-severity findings can be routed to a dashboard for review.

Before deploying this harness to production, you must establish evaluation criteria specific to the anomaly detection workflow. Create a golden dataset of known anomaly traces (both true positives and true negatives) and run the full pipeline against them nightly. Measure precision, recall, and mean time-to-detection. Monitor the false-positive rate closely during the first week of production; a rate exceeding 10% indicates that the [DRIFT_SENSITIVITY] parameter or the baseline comparison logic needs tuning. Additionally, log every model invocation's anomaly_score distribution to detect model drift over time—if the median score suddenly shifts upward without a corresponding change in actual system behavior, the prompt or the underlying model may have changed. Avoid wiring this prompt directly to automated remediation actions (e.g., auto-scaling or circuit breaking) until you have at least two weeks of stable, validated output. Start with alert-only mode, review every high-severity alert manually for the first week, and only then graduate to automated responses.