Latency Regression Trace Comparison Prompt Template

This prompt is for Site Reliability Engineers (SREs) and AI infrastructure engineers who need to isolate the root cause of a latency regression after a deployment, model upgrade, or configuration change. The job-to-be-done is a structured, step-by-step comparison of two production trace windows—one baseline and one suspect—to produce a statistically grounded latency diff. The ideal user has access to structured trace data (spans with operation_name, start_time, duration_ms, and metadata like model_id or prompt_version) and needs a report they can act on, not just a raw data dump. The prompt assumes the traces have already been collected and normalized into a consistent JSON schema.

Do not use this prompt when you lack a clear baseline for comparison, when the regression is purely in throughput or error rate without a latency component, or when the trace data is so sparse that statistical significance is impossible. This prompt is also inappropriate for real-time alerting; it is designed for post-hoc incident review and root-cause analysis. If your traces do not include per-step span identifiers and timing metadata, you must first enrich them with an instrumentation layer before this prompt can produce a meaningful diff. For continuous monitoring, pair this with a Latency Budget Violation Alert prompt that triggers on threshold breaches, then use this comparison prompt during the investigation phase.

After running this prompt, you will receive a step-by-step latency breakdown with absolute and relative deltas, statistical significance flags (using a basic threshold like p95 comparison or Cohen's d if you provide distribution data), and a ranked list of hypothesis-driven root causes. The output is designed to be pasted directly into an incident postmortem or a deployment rollback decision document. The next step is to validate the top hypothesis by isolating the suspect component—such as rolling back a specific prompt version or disabling a tool call—and re-running the trace comparison to confirm the regression disappears.

The following prompt template is designed to compare a baseline trace against a regression trace, producing a structured latency diff. It isolates which pipeline stages (e.g., model inference, tool calls, retrieval) regressed, calculates the magnitude of change, and generates a root-cause hypothesis. The template uses square-bracket placeholders for all dynamic inputs—replace these with actual trace data, thresholds, and formatting requirements before use. The prompt assumes traces are provided in a structured format (e.g., JSON spans with start/end timestamps and step labels) and that a statistical significance threshold has been pre-defined.

textCopy
You are an SRE analyzing latency regression in an AI pipeline. Compare the [BASELINE_TRACE] (pre-deployment) against the [REGRESSION_TRACE] (post-deployment) and produce a structured latency diff report.

## Input Data
- Baseline Trace: [BASELINE_TRACE]
- Regression Trace: [REGRESSION_TRACE]
- Significance Threshold: [P_VALUE_THRESHOLD] (default 0.05)
- Latency Budget per Step (ms): [LATENCY_BUDGET_MAP]
- Known Deployment Changes: [DEPLOYMENT_CONTEXT]

## Output Schema
Return a JSON object with the following structure:
{
  "summary": {
    "total_latency_delta_ms": number,
    "regression_detected": boolean,
    "primary_regression_stage": string,
    "confidence": "high" | "medium" | "low"
  },
  "step_diffs": [
    {
      "step_name": string,
      "baseline_p50_ms": number,
      "regression_p50_ms": number,
      "delta_ms": number,
      "percent_change": number,
      "p_value": number,
      "significant": boolean,
      "exceeds_budget": boolean
    }
  ],
  "root_cause_hypothesis": {
    "primary_suspect": string,
    "supporting_evidence": [string],
    "alternative_explanations": [string],
    "recommended_investigation": string
  },
  "budget_violations": [
    {
      "step_name": string,
      "budget_ms": number,
      "actual_ms": number,
      "severity": "critical" | "warning"
    }
  ]
}

## Constraints
- Use p50 (median) latency for comparisons unless raw data requires mean.
- Flag a step as "significant" only if p_value < [P_VALUE_THRESHOLD].
- If fewer than [MIN_SAMPLE_COUNT] spans exist for a step, set confidence to "low" and note the sample size limitation.
- Do not fabricate causes. Base the root-cause hypothesis only on the trace data and [DEPLOYMENT_CONTEXT].
- If no regression is detected, set "regression_detected" to false and leave "primary_regression_stage" as null.
- For any step exceeding its budget in [LATENCY_BUDGET_MAP], populate "budget_violations" with severity "critical" if delta > 2x budget, otherwise "warning".

## Example Output
[FEW_SHOT_EXAMPLE]

Analyze the traces now.

To adapt this template, replace each placeholder with concrete data from your observability pipeline. [BASELINE_TRACE] and [REGRESSION_TRACE] should contain span-level timing data—ideally extracted from your tracing system (e.g., OpenTelemetry, LangSmith, or custom trace storage). [LATENCY_BUDGET_MAP] should be a JSON object mapping step names to their maximum acceptable latency in milliseconds. [DEPLOYMENT_CONTEXT] should include any known changes between the two traces (model version bump, tool schema change, context window expansion, infrastructure migration). [FEW_SHOT_EXAMPLE] should contain a correctly formatted output example matching your expected schema to improve reliability. If your traces lack sufficient samples for statistical testing, lower [MIN_SAMPLE_COUNT] or switch to descriptive comparison only. Always validate the model's output against the schema before surfacing results in dashboards or incident reports—malformed JSON or missing fields are common failure modes.

This prompt is designed to be a critical step in a deployment verification pipeline, not a one-off chat interaction. The primary integration point is your existing observability stack (e.g., Datadog, Grafana, Honeycomb) or a custom trace store. The application harness must programmatically fetch two sets of traces—a baseline window before a deployment and a comparison window after—and format them into the [BASELINE_TRACES] and [COMPARISON_TRACES] placeholders. The prompt's value is in its structured differential analysis, so the harness must enforce that the input traces are from the same logical operation (e.g., the same API endpoint or agent task) and are sampled at a statistically meaningful rate. The harness should also inject a [DEPLOYMENT_CONTEXT] placeholder with metadata like the deployment timestamp, version tags, and any relevant infrastructure changes to help the model form a root-cause hypothesis.

To make this production-grade, wrap the LLM call in a function that validates the output against a strict JSON schema before the results are shown to an SRE. The model's output must contain a step_diffs array with fields for step_name, baseline_p50_ms, comparison_p50_ms, delta_ms, and regression_confidence. A post-processing validation layer should check that all delta_ms values are mathematically consistent and that the regression_confidence field is a valid enum (e.g., high, medium, low). If validation fails, the harness should implement a retry strategy with a backoff, feeding the validation error back into the [CONSTRAINTS] field of the prompt on the subsequent attempt. For high-severity regressions (e.g., a delta exceeding a configurable threshold like 200% or 500ms), the harness must not automatically post the analysis to a shared channel; instead, it should enrich the output with links to a full trace query and route it for human review before broadcasting to the incident response team.

Model choice is a key architectural decision here. A model with a large context window and strong reasoning capabilities (like Claude 3.5 Sonnet or GPT-4o) is required to reliably process the potentially large JSON blocks of trace data and perform the nuanced statistical reasoning. Do not use a small or fast model for this task, as the cost of a hallucinated or missed regression is far higher than the inference cost. Log the raw prompt, the validated output, and the model's system_fingerprint alongside the deployment event for a complete audit trail. The next step for the reader is to implement the trace-fetching function and the JSON schema validator; avoid the temptation to skip the output validation layer, as it is the primary defense against a malformed analysis that could misdirect an on-call engineer during an incident.