Incident Trace Comparison Prompt

This prompt is built for incident responders, AI SREs, and platform engineers who are staring at two production traces from the same time window: one that represents a healthy baseline and one that is failing. The job-to-be-done is a structured differential diagnosis. Instead of manually scanning thousands of spans, you provide two pre-reconstructed chronological narratives and the prompt isolates the first divergent event, a configuration change, or a behavioral difference that likely caused the failure. This is a precision instrument for narrowing the search space before you escalate to broader incident correlation or pull in the full on-call team.

You should use this prompt when you already have two clean, reconstructed traces. If you are starting from raw spans, you must run the End-to-End Trace Reconstruction Prompt first to convert those spans into the required chronological narratives. The prompt assumes the traces are from the same production window and share a common entry point, such as the same user query or API call. It is not designed for comparing traces from different code versions, different users, or different time periods unless you explicitly frame those as the baseline and failing conditions. The ideal input includes metadata like model version, prompt version, and deployment tags so the diff can surface configuration drift alongside behavioral drift.

Do not use this prompt for aggregate trend analysis, capacity planning, or cost attribution across many sessions—those tasks belong to monitoring dashboards and continuous trace analysis pipelines. This prompt is also a poor fit if the failure is already known to be a single-point infrastructure error like a network timeout or an out-of-memory kill; in those cases, the trace diff will only confirm what your metrics already show. After running this prompt, the next step is to take the isolated divergent event and feed it into a root-cause classification prompt or an incident runbook for remediation. If the diff is ambiguous, you may need to run the comparison against a second healthy baseline to rule out flukes.

This prompt template is designed to be the first executable step in your incident response workflow. When you have identified a failing trace and a known-healthy baseline trace from the same production window, this prompt performs a structured differential diagnosis. It isolates the first point of divergence—whether that is a tool-call argument change, a missing retrieval step, a prompt version mismatch, or an unexpected model output—and produces a report that an engineer can act on immediately. The template is model-agnostic and expects pre-extracted trace data to be injected via the placeholders.

textCopy
You are an AI incident responder specializing in production trace analysis. Your task is to compare two traces from the same time window and isolate the root cause of a failure.

## INPUT DATA

### BASELINE (Healthy Trace)
- Trace ID: [BASELINE_TRACE_ID]
- Prompt Version: [BASELINE_PROMPT_VERSION]
- Model: [BASELINE_MODEL]
- Trace Events (chronological JSON):
[BASELINE_TRACE_EVENTS]

### INCIDENT (Failing Trace)
- Trace ID: [INCIDENT_TRACE_ID]
- Prompt Version: [INCIDENT_PROMPT_VERSION]
- Model: [INCIDENT_MODEL]
- Trace Events (chronological JSON):
[INCIDENT_TRACE_EVENTS]

## CONSTRAINTS
- Compare events step-by-step in chronological order.
- Ignore fields that are expected to differ (e.g., timestamps, unique IDs) unless they indicate a timing anomaly.
- Flag the **first** event where the incident trace deviates from the baseline in a way that could cause downstream failure.
- If prompt versions differ, note this as a primary suspect before comparing events.
- If models differ, flag this as a potential cause but continue the event comparison.

## OUTPUT SCHEMA
Return a JSON object with the following structure:
{
  "divergence_point": {
    "step_index": <integer, the first step where divergence occurred>,
    "baseline_event": <the baseline event at that step>,
    "incident_event": <the incident event at that step>,
    "divergence_type": <"tool_call_change" | "retrieval_gap" | "output_drift" | "missing_step" | "extra_step" | "config_change" | "model_change">
  },
  "root_cause_hypothesis": <string, concise explanation of why this divergence caused the failure>,
  "confidence": <"high" | "medium" | "low">,
  "recommended_action": <string, immediate next step for the engineer>,
  "additional_observations": [<string, any other notable differences>]
}

## RISK LEVEL
[RISK_LEVEL]

## INSTRUCTIONS
1. Parse both trace event arrays.
2. Align them by step index where possible. If step counts differ, note the first missing or extra step.
3. For each aligned pair, compare the event type, tool name, arguments, and output summary.
4. Stop at the first meaningful divergence and populate the output schema.
5. If [RISK_LEVEL] is "high", prefix your root_cause_hypothesis with "URGENT: " and prioritize speed over exhaustive comparison.

Before running this prompt in production, ensure that the trace event arrays are normalized to a consistent schema. Raw spans from OpenTelemetry, LangSmith, or custom logging must be converted into a chronological list of objects containing at minimum step_index, event_type, and a payload summary. If your traces contain sensitive data, redact PII before injection. For high-severity incidents, pair this prompt with a human-in-the-loop review step: the model's output should be treated as a diagnostic aid, not a final root-cause determination. Validate the divergence point against your runbooks before taking automated remediation actions.

The Incident Trace Comparison Prompt is designed to be called programmatically during an active incident, not just used in a chat interface. The typical integration point is an internal incident response tool or a runbook automation that fetches two trace payloads—one from a known-healthy baseline window and one from the failing session—and passes them as structured inputs. The prompt expects a [BASELINE_TRACE] and a [FAILING_TRACE], each containing the full span tree, tool-call logs, retrieval events, and metadata. Before calling the model, the harness should validate that both traces are from the same application version, share the same prompt version identifier, and fall within the same production deployment window. If these preconditions are not met, the comparison is likely to produce spurious divergence signals, so the harness should abort early and request a better baseline.

The output of this prompt is a structured differential diagnosis, so the harness must enforce a strict output contract. Wrap the model call in a validation layer that parses the response against the expected [OUTPUT_SCHEMA], which includes fields for first_divergent_event, divergence_type (e.g., tool_call_missing, retrieval_ranking_change, model_output_drift), confidence_score, and root_cause_hypothesis. If the model returns malformed JSON or missing required fields, implement a retry loop with a maximum of three attempts, appending the parse error to the next request as a correction hint. For high-severity incidents where the divergence type is classified as model_output_drift or safety_filter_activation, route the diagnosis to a human responder for review before publishing to the incident channel. Log every comparison—including the raw traces, the model's response, the validation result, and the reviewer's decision—to an audit table so the diagnosis is reproducible and can be used for post-incident review.

Model choice matters here. Use a model with strong instruction-following and long-context capabilities, such as claude-sonnet-4-20250514 or gpt-4o, because the combined trace payloads can exceed 20,000 tokens. Do not use a lightweight routing model for this task; the comparison requires careful reasoning over structured spans, and a smaller model will miss subtle divergences. If your observability platform supports trace differencing natively, use that as a pre-filter to reduce the trace payload to only the divergent segments before passing them to the model. This reduces token consumption and improves comparison accuracy. Finally, avoid running this prompt on every failure automatically—reserve it for incidents where the root cause is not immediately obvious from alert dashboards or where multiple teams are reporting inconsistent behavior from the same deployment.