Latency Threshold Breach Detection Prompt Template

This prompt is designed for infrastructure engineers and AI SREs who need to programmatically detect latency threshold breaches across distributed trace spans. Use it when you have raw trace data from an observability platform (such as OpenTelemetry, Datadog, or Grafana Tempo) and need to identify which spans violated P50, P95, or P99 latency targets within a configurable evaluation window. The prompt expects structured span data as input and produces a breach report with timestamps, affected components, bottleneck attribution, and severity classification.

This is not a dashboard replacement or a real-time alerting system. It is a diagnostic and analysis tool that should be wired into a monitoring pipeline where human review or automated escalation can act on the output. The prompt assumes spans include start timestamps, duration in milliseconds, service names, and operation names. Before wiring this into production, validate that your trace data conforms to this schema and that your percentile targets are calibrated against historical baselines—running breach detection against misconfigured thresholds will generate noise that erodes on-call trust.

Do not use this prompt when you lack structured trace data, when you need sub-millisecond precision, or when the evaluation window is undefined. It is also unsuitable for real-time stream processing where per-span latency decisions must happen inline; this prompt is designed for batch evaluation over a completed window. If your traces lack service-name or operation-name fields, the bottleneck attribution step will degrade and produce generic rather than actionable output. For high-severity breaches affecting customer-facing SLOs, always route the output through a human review step before triggering an incident.

This prompt template is designed to be dropped directly into your observability workflow. It expects structured trace span data as input and produces a strict JSON report of all latency threshold breaches. The template enforces attribution discipline: every breach must be tied to a specific service and operation present in the input data. It will not invent spans, durations, or causal relationships not supported by the evidence. If a span lacks required fields for analysis, the template instructs the model to flag it as incomplete rather than guessing.

textCopy
You are a production observability analyst. Your task is to analyze the provided trace span data and identify all latency threshold breaches across P50, P95, and P99 percentiles within the specified evaluation window.

## INPUT DATA
[TRACE_SPAN_DATA]

## CONFIGURATION
- Evaluation Window: [EVALUATION_WINDOW_START] to [EVALUATION_WINDOW_END]
- P50 Threshold (ms): [P50_THRESHOLD]
- P95 Threshold (ms): [P95_THRESHOLD]
- P99 Threshold (ms): [P99_THRESHOLD]

## OUTPUT SCHEMA
Return a valid JSON object with this exact structure:
{
  "evaluation_window": {
    "start": "ISO8601 timestamp",
    "end": "ISO8601 timestamp"
  },
  "thresholds": {
    "p50_ms": number,
    "p95_ms": number,
    "p99_ms": number
  },
  "breaches": [
    {
      "breach_id": "string",
      "timestamp": "ISO8601 timestamp of breach detection",
      "percentile": "P50 | P95 | P99",
      "observed_latency_ms": number,
      "threshold_ms": number,
      "excess_ms": number,
      "severity": "LOW | MEDIUM | HIGH | CRITICAL",
      "affected_service": "string",
      "affected_operation": "string",
      "span_id": "string",
      "trace_id": "string",
      "attribution": {
        "bottleneck_service": "string",
        "bottleneck_operation": "string",
        "step_level_detail": "string explaining where latency accumulated"
      }
    }
  ],
  "incomplete_spans": [
    {
      "span_id": "string",
      "trace_id": "string",
      "missing_fields": ["field_name"],
      "reason": "string explaining why analysis was not possible"
    }
  ],
  "summary": {
    "total_spans_analyzed": number,
    "total_breaches": number,
    "breaches_by_percentile": {
      "p50": number,
      "p95": number,
      "p99": number
    },
    "breaches_by_severity": {
      "low": number,
      "medium": number,
      "high": number,
      "critical": number
    },
    "incomplete_spans_count": number
  }
}

## CONSTRAINTS
[CONSTRAINTS]
- Do not invent spans, services, operations, or durations not present in the input data.
- If a span lacks service name, operation name, or duration fields, flag it in `incomplete_spans` rather than guessing.
- Severity classification: LOW (excess < 10% of threshold), MEDIUM (10-25%), HIGH (25-50%), CRITICAL (>50%).
- Attribution must reference specific spans and operations from the input data.
- If multiple spans contribute to a single breach, identify the primary bottleneck.
- Return only valid JSON. No markdown fences, no commentary.

To adapt this template, replace the square-bracket placeholders with your specific configuration. [TRACE_SPAN_DATA] should contain your structured span data in JSON format, including at minimum span_id, trace_id, service_name, operation_name, duration_ms, and timestamp fields. [EVALUATION_WINDOW_START] and [EVALUATION_WINDOW_END] define the time range for analysis. The three threshold values set your latency SLOs. [CONSTRAINTS] can be extended with additional rules, such as excluding specific services from analysis or adding custom severity mappings. Before deploying this prompt into an automated monitoring pipeline, validate the output against known breach scenarios to ensure the severity classification and attribution logic matches your operational expectations. For high-severity breaches, route the output to a human reviewer before triggering automated incident response.

The latency threshold breach detection prompt is designed to operate as a batch analysis step within an existing observability pipeline, not as a real-time streaming processor. Feed it a window of trace spans extracted from your tracing backend (Jaeger, Grafana Tempo, Datadog APM, or similar) at a cadence that matches your SLO evaluation interval—typically every 5 to 15 minutes. The prompt expects pre-aggregated span data with timestamps, duration values, service names, and operation identifiers. Do not send raw, unsampled trace payloads; pre-filter to the relevant services and time window before invoking the model.

Integration flow: (1) A scheduled job queries your trace store for spans in the evaluation window, filtered by service and operation. (2) The span data is serialized into the [TRACE_SPANS] input block as a structured JSON array with fields: span_id, service, operation, duration_ms, start_time, parent_span_id, and status. (3) The prompt is invoked with [PERCENTILE_THRESHOLDS] set to your SLO targets (e.g., {"p50": 200, "p95": 1000, "p99": 3000}) and [WINDOW_CONFIG] specifying the evaluation window size and granularity. (4) The model returns a structured JSON payload with breach events, affected components, and bottleneck attribution. (5) A post-processing validator checks that every breach entry contains a timestamp, percentile, observed_value, threshold, and affected_service before the output is forwarded to your alerting system.

Validation and retry logic: Implement a schema validator that rejects outputs missing required fields or containing malformed timestamps. If validation fails, retry once with the same input and an explicit [CONSTRAINTS] amendment flagging the missing field. If the retry also fails, log the raw output and escalate to the on-call channel with a latency_breach_detection_failed event. For high-severity environments, add a human-review gate before auto-paging: route breach events with severity: critical to a review queue where an SRE confirms the bottleneck attribution before an incident is declared. This prevents alert fatigue from transient spikes that the model may over-classify.

Model choice and cost: This task requires strong structured output adherence and numerical reasoning. Use a model with reliable JSON mode and low hallucination rates on data-extraction tasks—GPT-4o, Claude 3.5 Sonnet, or equivalent. Avoid smaller models that may conflate percentile values or fabricate breach events when span data is sparse. Token consumption scales with the number of spans in the window; if you are processing thousands of spans per evaluation cycle, pre-aggregate to service-level duration distributions rather than sending individual spans. Cache the system prompt prefix across invocations to reduce per-request cost.

Alert routing and observability: Map the prompt's output fields directly to your alerting system's data model. The breach_id should be a deterministic hash of window_start + percentile + service to enable deduplication in your alert manager. Attach the full prompt input and output as metadata on the alert so incident responders can trace the breach back to the source spans. Log every invocation—input, output, latency, token count, and validation result—to your prompt observability store for later audit and threshold tuning. If breach frequency increases after a model or prompt version change, compare invocation logs to distinguish a real latency regression from a detection sensitivity shift.