Alert Condition Definition Prompt for Trace Patterns

This prompt is designed for AI SREs and platform engineers who have moved beyond one-off trace investigations and need to operationalize their findings. The job-to-be-done is converting a known, recurring failure pattern observed in production traces into a precise, deployable alert rule definition. This pattern could be a latency spike in a specific tool call, a grounding gap in RAG responses, a schema drift event, or an anomalous token consumption profile. The ideal user has already completed the diagnostic work—they have access to structured trace data, understand the failure's shape and frequency, and now need to encode that understanding into a condition expression, threshold value, evaluation window, and severity mapping that a monitoring platform like Datadog, Grafana, or a custom alerting service can execute.

This prompt does not replace your observability platform; it produces the rule definition that your platform ingests. It assumes you are working with a specific trace pattern, not exploring data for unknown anomalies. You must provide the prompt with a structured description of the pattern, the target metric (e.g., tool_call_latency_ms), the statistical function (e.g., p95), the evaluation window (e.g., 5m), and the desired severity levels. The output is a machine-readable alert condition, not a dashboard or a report. Before using this prompt, ensure you have already isolated the pattern to a specific span attribute, operation name, or resource label. If you are still in the discovery phase, use a trace review or anomaly detection prompt first.

Do not use this prompt when the failure pattern is undefined, when you lack access to structured trace data, or when the alert logic requires complex multi-dimensional baselines that are better defined in code. This prompt is also not a substitute for an incident response runbook; it defines the trigger, not the remediation steps. Once you have generated the alert rule definition, the next step is to validate it against historical trace data to measure its precision and recall, tune the threshold to balance signal and noise, and then commit the rule to your monitoring-as-code repository. Always pair the generated alert condition with a test case that simulates the expected breach to confirm it fires correctly before deployment.

This prompt template is designed to take the output of a trace pattern analysis and convert it into a structured, machine-readable alert condition definition. The goal is to move from ad-hoc trace review to continuous monitoring by producing the exact parameters—condition expression, threshold value, evaluation window, and severity mapping—that an alerting system needs. Before using this template, you should have already identified a specific, recurring trace pattern (e.g., a spike in tool-call errors for a particular function, or a latency breach in a specific span) that warrants an alert. The template uses square-bracket placeholders; replace each one with concrete details from your analysis.

textCopy
You are an AI SRE converting an observability finding into an executable alert rule. Your task is to produce a complete alert condition definition based on the provided trace pattern, platform constraints, and operational policies.

## Inputs
- **Trace Pattern Description:** [TRACE_PATTERN_DESCRIPTION]
- **Affected Spans/Services:** [AFFECTED_SPANS]
- **Platform Alerting Syntax:** [PLATFORM_SYNTAX] (e.g., PromQL, Datadog monitor JSON, custom DSL)
- **Operational Policies:** [OPERATIONAL_POLICIES] (e.g., SLO targets, error budgets, on-call rotation rules)

## Output Schema
Produce a JSON object with the following structure:
{
  "alert_name": "string",
  "condition_expression": "string",
  "threshold_value": "number",
  "evaluation_window": "string",
  "severity": "critical|warning|info",
  "alert_message": "string",
  "runbook_url": "string",
  "test_cases": [
    {
      "description": "string",
      "input_trace_data_summary": "string",
      "expected_alert_status": "firing|resolved"
    }
  ]
}

## Constraints
- The `condition_expression` must be valid syntax for the specified [PLATFORM_SYNTAX].
- The `threshold_value` must be derived from the [TRACE_PATTERN_DESCRIPTION] and align with [OPERATIONAL_POLICIES].
- The `evaluation_window` must be long enough to avoid flapping but short enough to meet detection time objectives.
- Generate at least two `test_cases`: one that should trigger the alert and one that should not.
- If the trace pattern is intermittent, the `evaluation_window` should account for this.
- If the alert is high-severity, the `alert_message` must include a direct link to the [RUNBOOK_URL].

## Risk Level
[RISK_LEVEL] (This should be 'high', 'medium', or 'low'. If 'high', you must add a note requiring human approval before deployment.)

To adapt this template, start by pasting your specific trace pattern description into the [TRACE_PATTERN_DESCRIPTION] placeholder. Be precise: instead of 'slow responses,' write 'P95 latency for the /api/v1/search endpoint in the retrieval-service span exceeds 2000ms.' Next, specify the exact alerting syntax your platform expects in [PLATFORM_SYNTAX]. This ensures the generated condition_expression is not just a logical statement but a copy-pasteable rule. The [OPERATIONAL_POLICIES] placeholder is critical for setting a meaningful threshold; for example, if your SLO for search latency is 1500ms, the alert threshold should be set at or near that boundary. Finally, always set the [RISK_LEVEL] accurately. A high-risk alert (e.g., one that pages an on-call engineer) should always be reviewed by a human before being deployed to production. After generating the alert definition, use the provided test_cases to validate the condition in a staging environment before activating it.

This prompt is designed to be called programmatically as part of an AI SRE workflow, not as a one-off chat interaction. The typical integration point is a backend service that receives trace pattern analysis results—often from a prior diagnosis or anomaly detection step—and needs to convert those findings into executable alert rules. The harness must supply structured inputs for trace patterns, SLO targets, and the target monitoring system's expression language, then validate the generated conditions before they touch production alerting infrastructure.

Wire the prompt into a pipeline with these stages: Input assembly—collect trace pattern evidence, affected span types, metric baselines, and SLO thresholds into the [TRACE_PATTERN_ANALYSIS], [SLO_TARGETS], and [ALERT_EXPRESSION_LANGUAGE] placeholders. Model call—use a model with strong structured output support (GPT-4o, Claude 3.5 Sonnet, or equivalent) with response_format set to JSON and the output schema enforced. Output validation—parse the generated alert conditions and run them through a schema validator that checks required fields (condition_expression, threshold_value, evaluation_window, severity, test_cases) and type correctness. Reject any output where the condition expression references undefined metrics or where test cases don't exercise both triggering and non-triggering scenarios. Expression safety check—before deployment, run each generated condition expression through a dry-run evaluation against recent production trace data to verify it doesn't produce false-positive storms or evaluate to null unexpectedly. Human review gate—for severity levels of critical or high, route the alert definition to an on-call SRE for approval before it goes live. For medium and low severities, automated deployment is acceptable if the dry-run passes and the false-positive rate estimate is below 5%.

Retry logic should be conservative. If validation fails, retry once with the validation errors appended to the prompt as [PREVIOUS_VALIDATION_ERRORS]. If the second attempt also fails, escalate to a human operator with the failed output and validation report rather than looping indefinitely. Log every generation attempt—including the input trace pattern summary, the generated conditions, validation results, and the final deployment decision—to an audit table. This audit trail is essential for post-incident review when an alert either failed to fire or fired spuriously. Avoid wiring this prompt directly to alerting infrastructure without the validation and dry-run gates; a malformed condition expression can silently fail to evaluate or, worse, trigger a flood of false alerts that erode on-call trust.