Latency Spike Root-Cause Trace Prompt

This prompt is designed for performance engineers and AI SREs who are responding to an active latency alert and have access to structured trace data from the spike window. The job-to-be-done is isolating the bottleneck across model inference, tool calls, retrieval, and context assembly spans. Instead of manually scanning individual traces in an observability dashboard, you feed a batch of normalized traces into this prompt to receive a latency attribution breakdown, a ranked list of suspect spans, and the specific timing evidence that supports each hypothesis. The prompt assumes traces are already collected, normalized, and contain span-level timing data—it does not query your observability backend or perform real-time data extraction.

Do not use this prompt when you lack structured trace data, when the spike is still ongoing and traces are incomplete, or when the root cause is already known to be an infrastructure outage (e.g., a dead model endpoint or a network partition). This prompt is also not a replacement for your observability dashboard or alerting system; it is a diagnostic accelerator for the phase of an incident where you have trace data in hand and need to move from 'latency is high' to 'the retrieval step in the vector store is adding 800ms of p99 latency due to a cold cache.' For ambiguous cases where multiple spans show correlated degradation, the prompt's ranked suspect list helps prioritize your next investigation step rather than delivering a single definitive answer. Always validate the output against raw trace data before taking remediation actions such as rolling back a prompt version, scaling a service, or adjusting a timeout.

After using this prompt, you should have a clear, evidence-backed hypothesis about which component is the primary latency contributor. The next step is to validate that hypothesis against additional traces from the spike window and, if confirmed, to execute the appropriate remediation—whether that means tuning retrieval parameters, adjusting model routing, or escalating to the infrastructure team. If the prompt's output is inconclusive or the top suspect has low confidence, expand the input batch to include more traces or narrow the time window to isolate the spike's onset more precisely.

This prompt template is designed to be dropped directly into your observability or incident response tooling. It expects structured trace data from the latency spike window and a set of baseline traces for comparison. The model's job is to attribute added latency to specific spans—model inference, tool calls, retrieval, context assembly—and produce a ranked suspect list with per-span timing evidence. The output is a structured latency attribution breakdown that your engineering team can act on immediately.

textCopy
You are an AI performance engineer analyzing a latency spike incident.

Your task is to compare traces from the spike window against baseline traces and attribute the added latency to specific spans.

## INPUT DATA

### Spike Window Traces
[SPIKE_TRACES]

### Baseline Traces (pre-spike, same or similar request types)
[BASELINE_TRACES]

### Spike Window Time Range
[SPIKE_WINDOW_START] to [SPIKE_WINDOW_END]

### Latency Threshold for Alert
[LATENCY_THRESHOLD_MS]

## OUTPUT SCHEMA

Return a JSON object with this exact structure:
{
  "summary": {
    "spike_window": "string",
    "total_traces_analyzed": number,
    "traces_exceeding_threshold": number,
    "mean_latency_increase_ms": number,
    "p95_latency_increase_ms": number
  },
  "span_attribution": [
    {
      "span_type": "model_inference | tool_call | retrieval | context_assembly | other",
      "mean_added_latency_ms": number,
      "p95_added_latency_ms": number,
      "percentage_of_total_increase": number,
      "affected_trace_count": number,
      "evidence_trace_ids": ["string"],
      "observations": "string"
    }
  ],
  "ranked_suspects": [
    {
      "rank": number,
      "span_type": "string",
      "specific_target": "string (e.g., 'tool:vector_search', 'model:gpt-4-turbo', 'retrieval:pinecone_query')",
      "confidence": "high | medium | low",
      "evidence_summary": "string",
      "recommended_action": "string"
    }
  ],
  "anomalies": [
    {
      "trace_id": "string",
      "anomaly_type": "string",
      "description": "string",
      "severity": "critical | high | medium | low"
    }
  ],
  "needs_human_review": boolean,
  "human_review_reason": "string (if applicable)"
}

## CONSTRAINTS

1. Only attribute latency to spans present in the trace data. Do not speculate about spans not captured.
2. If a span type shows no statistically significant increase, set its values to 0 and note it in observations.
3. For each suspect, cite at least one specific trace ID as evidence.
4. If the spike window contains fewer than [MIN_TRACES_FOR_CONFIDENCE] traces, set confidence to "low" for all suspects and set needs_human_review to true.
5. Distinguish between latency caused by the span itself vs. latency caused by queuing or contention before the span started.
6. Flag any trace where the root cause appears to be external to the AI system (network timeout, upstream dependency failure) in the anomalies array.
7. If tool-call latency increased but the tool response payload size also increased, note this correlation rather than assuming a tool degradation.

## ANALYSIS STEPS

1. Compute per-span latency deltas between spike and baseline traces.
2. Group spans by type and compute aggregate statistics.
3. Rank span types by contribution to total latency increase.
4. Within each high-contribution span type, identify the specific target (model name, tool name, retrieval index).
5. Check for correlated changes: did context size grow? Did tool response payloads increase? Did model change?
6. Flag any outlier traces that don't fit the dominant pattern.
7. Assess whether the evidence is sufficient for high-confidence attribution or requires human review.

Return only the JSON object. No markdown fences, no additional commentary.

To adapt this template, replace the square-bracket placeholders with your actual data. [SPIKE_TRACES] and [BASELINE_TRACES] should contain structured trace objects with at minimum: trace_id, span_type, span_name, duration_ms, start_time, and any relevant metadata like model name or tool endpoint. [MIN_TRACES_FOR_CONFIDENCE] should be set based on your traffic volume—typically 10–20 traces for a meaningful comparison. If your observability platform exports traces in OpenTelemetry format, pre-process them into the span-level structure this prompt expects before injection. The output schema is designed to be machine-readable for downstream automation: pipe the ranked suspects into your incident channel, use the needs_human_review flag to gate automated escalation, and store the full attribution breakdown in your incident document for postmortem use.

This prompt is designed to be called programmatically as part of an incident response runbook, not as a one-off chat interaction. The harness should fetch traces from your observability platform (e.g., Datadog, Grafana, Honeycomb) for the spike window, format them into the [TRACE_DATA] placeholder, and invoke the model via your production API. The prompt expects raw span data with timing fields—if your trace exporter omits duration_ms or start_time, preprocess the spans to compute these before injection. Run this prompt only after you have isolated the time window and service boundaries; feeding it unfiltered firehose data will produce noisy, low-confidence attributions.

Validation and output contract: Parse the model's JSON output against a strict schema before acting on it. The latency_attribution array must contain objects with span_name, component, p50_ms, p99_ms, delta_from_baseline_ms, and evidence_span_ids. Reject any response where evidence_span_ids reference spans not present in the input trace data—this is a hallucination signal. Implement a retry loop with a maximum of 2 attempts, appending the validation error to the retry prompt as [PREVIOUS_ERROR]. If the model fails to produce valid JSON after 2 retries, escalate to a human responder with the raw trace data and the partial output. Model choice: Use a model with strong JSON mode and long-context handling (e.g., GPT-4o, Claude 3.5 Sonnet) because trace payloads can exceed 50k tokens during spike windows. Set temperature=0 to maximize reproducibility of the attribution breakdown.

Logging and audit trail: Log every invocation—input trace count, spike window parameters, model response, validation result, and final attribution—to your incident management system (e.g., PagerDuty, FireHydrant, or a dedicated audit table). This creates an evidence trail for postmortems and allows you to compare attributions across incidents over time. Human review gate: If the ranked_suspects list includes a component marked as [RISK_LEVEL] = high (e.g., the primary database connection pool or the model inference endpoint), require a human to acknowledge the finding before automated remediation fires. This prevents a misattribution from triggering a rollback or scale-up on the wrong component. Next step: After the harness returns a validated attribution, feed the top suspect into your remediation runbook—whether that means scaling a service, rolling back a model version, or failing over to a secondary region—and attach the prompt's output as the justification artifact.