Latency Spike Root-Cause Analysis Prompt

This prompt is designed for infrastructure engineers, SREs, and FinOps teams who need to perform a surgical root-cause analysis on a single production trace where end-to-end latency exceeded a defined threshold. The job-to-be-done is isolating the specific span—whether it's a retrieval call, model inference, tool execution, or network hop—that is responsible for the latency spike. It requires a raw trace as input, complete with timestamps and span metadata, and produces a structured latency attribution report that decomposes the total latency into per-step measurements. The ideal user is someone who has already identified an anomalous session through monitoring dashboards and now needs to understand exactly what happened inside that session before deciding on a fix.

This prompt is not a replacement for aggregate observability tools or real-time alerting systems. Do not use it to analyze trends across thousands of traces, to calculate p95 latency across a fleet, or to trigger automated alerts. It is purpose-built for deep, single-session investigation. The prompt assumes you have already isolated a trace of interest—perhaps flagged by a user complaint, a cost anomaly, or a spike on a latency histogram—and you need to explain the spike before escalating to the model provider, adjusting a timeout, or rewriting a tool. The required context includes the raw trace data, a defined latency threshold that was breached, and optionally a baseline percentile for comparison. Without a specific trace to analyze, this prompt will produce a generic report that lacks diagnostic value.

Before using this prompt, confirm that the trace data is complete and includes parent-child span relationships, start and end timestamps, and operation names. If your observability pipeline samples traces or drops spans, the analysis will be incomplete and may misattribute the latency spike. After running the analysis, use the output to decide your next action: if a retrieval span is the culprit, investigate the vector database or embedding service; if model inference is slow, check for token limits, model version changes, or provider throttling; if a tool call is the bottleneck, review the tool's timeout and retry logic. Avoid using this prompt when the latency spike is caused by client-side network conditions or browser rendering, as those factors are typically invisible in server-side traces.

This prompt template is designed to be pasted directly into your investigation notebook or an automated diagnosis harness. It instructs the model to act as an SRE performing a structured latency breakdown on a single trace. The goal is to move from 'the trace was slow' to a precise attribution, identifying whether the spike originated in retrieval, model inference, a specific tool call, or network transit. The output is a structured latency attribution report that can be compared against your service's baseline percentiles.

textCopy
You are an SRE specializing in AI system performance. Analyze the provided trace data to identify the root cause of a latency spike.

## Input Data
- Trace JSON: [TRACE_JSON]
- Baseline Latency Percentiles (p50, p95, p99) for each span type: [BASELINE_PERCENTILES]

## Analysis Steps
1. Parse the trace into a chronological list of spans. For each span, extract the `span_id`, `operation_name`, `start_time`, and `duration_ms`.
2. Categorize each span into one of the following types: `retrieval`, `model_inference`, `tool_call`, `network`, `other`.
3. For each span, calculate its deviation from the provided p95 baseline for its type. Flag any span where `duration_ms > p95_baseline`.
4. Identify the single span with the highest positive deviation from its p95 baseline. This is the primary latency contributor.
5. Trace the causal chain backward from the primary contributor to the initial user request to identify any upstream triggers (e.g., a large retrieved context causing a slow tool call).

## Output Format
You must respond with a valid JSON object conforming to this schema:
{
  "trace_id": "string",
  "total_duration_ms": number,
  "primary_latency_contributor": {
    "span_id": "string",
    "operation_name": "string",
    "span_type": "string",
    "duration_ms": number,
    "p95_baseline_ms": number,
    "deviation_from_baseline_ms": number,
    "upstream_trigger": "string or null"
  },
  "span_breakdown": [
    {
      "span_id": "string",
      "operation_name": "string",
      "span_type": "string",
      "duration_ms": number,
      "p95_baseline_ms": number,
      "deviation_from_baseline_ms": number,
      "is_flagged": boolean
    }
  ],
  "root_cause_hypothesis": "A concise, evidence-based explanation of why the primary contributor was slow."
}

## Constraints
- Do not invent data. If a baseline is missing for a span type, set `p95_baseline_ms` to null and `deviation_from_baseline_ms` to null.
- If multiple spans have similar high deviations, select the one with the largest absolute duration as the primary contributor.
- The `root_cause_hypothesis` must reference specific evidence from the trace (e.g., 'The high latency in the tool_call span is directly correlated with a 3x increase in input token size from the preceding retrieval span.').

To adapt this template, replace the [TRACE_JSON] placeholder with the raw span data from your observability platform, such as LangSmith, Arize, or a custom trace store. The [BASELINE_PERCENTILES] placeholder should be populated with your service's established performance baselines. If you lack formal baselines, you can initially use a recent healthy trace's span durations as a rough proxy, but be explicit about this in your harness. For automated use, the JSON output schema is the contract your harness should validate against. If the model fails to return valid JSON or the primary_latency_contributor is missing, implement a retry with a stricter prompt or fall back to a rule-based analysis of the raw spans.

This prompt is designed to be the first step in a latency investigation, not a standalone report. It should be wired into an observability pipeline where a single trace, already flagged by a p99 latency alert, is fed into the model along with its baseline percentile data. The prompt expects a pre-processed trace object containing per-span timing, so your harness must first extract spans from your tracing backend (e.g., LangSmith, Arize, Datadog, or custom OpenTelemetry collectors) and format them into the [TRACE_SPANS] placeholder. The baseline percentiles in [BASELINE_PERCENTILES] should be computed from a rolling window of healthy traffic—do not hardcode them or use global averages, as that will produce false attributions during low-traffic periods.

The implementation should include a validation layer that checks the model's output against the expected [OUTPUT_SCHEMA] before the report reaches an engineer. At minimum, validate that the attribution array sums to 100% (±1% tolerance), that every span ID referenced in the report exists in the input trace, and that the dominant_span field points to a real span. If validation fails, retry once with the same prompt and a prepended error message describing the schema violation. If the retry also fails, log the raw trace and model output to a human review queue rather than silently dropping the investigation. For high-severity incidents where latency directly impacts an SLO, route the validated report to an on-call channel with a link to the full trace.

Model choice matters here. Use a model with strong structured output support and a large enough context window to hold the full trace JSON plus baseline data. GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro are all suitable. Avoid small or quantized models for this task—they frequently miscalculate percentage attributions or hallucinate span IDs. If you must use a smaller model, add a post-processing step that recalculates the latency percentages from the raw span durations in the input trace and only uses the model's output for the textual root-cause hypothesis and recommendations. This hybrid approach keeps the numerical analysis trustworthy while still getting the narrative value from the LLM.

Finally, store every analysis result alongside the trace ID and prompt version in your observability database. This creates an audit trail for post-incident review and lets you measure whether the prompt's attributions align with the actual remediation actions taken by engineers. Over time, you can use this data to fine-tune the baseline percentile windows, adjust the prompt's emphasis on specific span types, or build a few-shot example set from confirmed root causes that improves accuracy on recurring failure patterns.