Latency Budget Violation Alert Prompt Template

This prompt is designed for AI operations teams and Site Reliability Engineers (SREs) who are responsible for maintaining Service Level Objectives (SLOs) on AI-powered features. The core job-to-be-done is converting raw, pre-filtered production trace data into a structured, actionable alert payload. The ideal user is an engineer building or maintaining an observability pipeline that already identifies traces exceeding a defined per-step or end-to-end latency budget. The prompt's value is in standardizing the alert format for downstream systems like PagerDuty, incident management platforms, or automated remediation runbooks, ensuring that on-call personnel receive consistent, diagnostic-rich notifications instead of raw data dumps.

You should use this prompt when your monitoring system has already performed the initial threshold comparison and data collection. The required context includes a specific trace ID, the violated latency budget (in milliseconds), the measured latency, the step or span where the violation occurred, and the service name. The prompt does not perform data collection, query your observability backend, or decide if a violation is severe enough to page someone. Its sole function is to take a structured set of violation facts and produce a templated alert. This separation of concerns is critical: your data pipeline owns the detection logic, and this prompt owns the presentation logic.

Avoid using this prompt for real-time, sub-millisecond alerting where the overhead of an LLM call is unacceptable, or for generating alerts from unsanitized traces that may contain sensitive user data. It is also not a replacement for a root-cause analysis engine; it suggests an investigation path based on the provided data but does not perform a deep diagnostic. The next step after generating this alert is to feed the structured JSON payload into your incident management system and, if the alert severity is high, trigger an automated runbook that gathers more context from the affected service.

This prompt template is designed to be wired directly into your observability pipeline. It expects a structured trace payload and a defined set of latency budgets, and it produces a machine-readable alert object. The prompt enforces strict JSON output, severity classification, and an investigation path—removing the need for brittle regex or hard-coded threshold logic in your alerting layer.

textCopy
You are an AI operations analyst reviewing production trace data for latency budget compliance.

Analyze the provided [TRACE_DATA] against the latency budgets defined in [LATENCY_BUDGETS].

For each step in the trace, compare its observed duration to its budget. If any step exceeds its budget, or if the end-to-end duration exceeds the total budget, generate a violation alert.

[OUTPUT_SCHEMA]
Return a single JSON object with this exact structure:
{
  "alert_type": "latency_budget_violation",
  "trace_id": "string",
  "timestamp": "ISO 8601 string from trace",
  "severity": "critical | warning | info",
  "violations": [
    {
      "step_name": "string",
      "observed_latency_ms": number,
      "budget_ms": number,
      "excess_ms": number,
      "excess_percent": number
    }
  ],
  "end_to_end_observed_ms": number,
  "end_to_end_budget_ms": number,
  "affected_service": "string or null",
  "suggested_investigation": [
    "string"
  ]
}

[CONSTRAINTS]
- Severity is "critical" if any step exceeds its budget by more than 50% or if the end-to-end budget is exceeded.
- Severity is "warning" if any step exceeds its budget by 20-50%.
- Severity is "info" if any step exceeds its budget by less than 20%.
- If no violations exist, return a JSON object with "alert_type": "no_violation", "trace_id", and an empty "violations" array.
- The "suggested_investigation" array must contain 2-4 concrete, actionable steps based on the specific steps that violated their budgets. Do not return generic advice.
- All numeric values must be integers.
- Do not include markdown fences, explanations, or any text outside the JSON object.

[TRACE_DATA]
[INSERT_TRACE_JSON_HERE]

[LATENCY_BUDGETS]
[INSERT_BUDGET_JSON_HERE]

To adapt this template, replace the [INSERT_TRACE_JSON_HERE] and [INSERT_BUDGET_JSON_HERE] placeholders with your actual data at runtime. The [LATENCY_BUDGETS] input should be a JSON object mapping step names to their maximum allowed latency in milliseconds, plus an end_to_end_ms key. The [TRACE_DATA] input should be a JSON object containing at minimum a trace_id, a timestamp, and a spans array where each span has a step_name and duration_ms. If your trace format differs, adjust the field mapping instructions in the prompt's analysis paragraph before deploying. For high-severity alerts in production, always route the output through a human-review step or a ticketing system before triggering automated rollbacks.

This prompt is designed to be called programmatically as part of an automated monitoring pipeline, not as a one-off chat interaction. The typical integration point is a scheduled job or a streaming alert processor that queries your observability backend (e.g., Datadog, Grafana, Honeycomb, or an internal trace store) for traces exceeding defined latency thresholds. The raw trace spans, step-level timing data, and the relevant SLO budget definitions are assembled into the [TRACE_DATA] and [LATENCY_BUDGETS] placeholders before the prompt is submitted to the model. The output is a structured alert payload that your incident management system (PagerDuty, Opsgenie, a webhook) can ingest directly.

Validation and Retry Logic: The model's output must conform to a strict JSON schema before any alert is dispatched. Implement a post-generation validation step that checks for required fields (violation_severity, affected_steps, investigation_path), valid enum values for severity (e.g., critical, warning, info), and non-empty arrays for affected steps. If validation fails, retry the prompt once with the validation error message appended to the [CONSTRAINTS] field. If the second attempt also fails, log the raw output and fall back to a templated alert with the raw trace ID and a generic 'manual review required' message. Never silently drop a failed alert generation; the absence of an alert for a real violation is a higher risk than a malformed alert.

Model Choice and Latency Budget: This is a structured extraction and summarization task, not a creative generation task. Favor models with strong JSON mode and instruction-following capabilities (e.g., GPT-4o, Claude 3.5 Sonnet, or a fine-tuned smaller model if you have sufficient trace-alert training pairs). Set a tight generation timeout (e.g., 10 seconds) because the alert prompt itself must not become a source of latency during an incident. If the model call exceeds this timeout, abort and trigger the fallback alert. For high-throughput environments, batch multiple trace violations into a single prompt call by providing an array of traces in [TRACE_DATA] and requesting an array of alert objects in the [OUTPUT_SCHEMA].

Human Review and Escalation: For critical severity violations, the generated alert should be routed to an on-call engineer immediately, but the prompt's output should be treated as a diagnostic aid, not an autonomous decision. The investigation_path field provides a suggested starting point; it does not replace human judgment. For warning severity, consider aggregating alerts into a daily digest rather than paging on-call staff. Log every generated alert with the full prompt input, model output, and validation result to create an audit trail for retrospective analysis and prompt improvement. If the same trace repeatedly triggers alerts, implement a deduplication key based on the trace ID and violation type to prevent alert storms.