Incident Trace Summarization Prompt for SRE Handoff

This prompt is designed for an on-call Site Reliability Engineer (SRE) or incident responder who is handing off an active investigation to the next shift. The primary job-to-be-done is to condense a messy, multi-trace incident investigation into a structured, concise handoff brief. The ideal user has already done the initial triage—they have a collection of relevant trace IDs, have formed a preliminary hypothesis, and need to transfer diagnostic signal without forcing the next responder to replay hours of raw trace analysis. The prompt requires a specific set of inputs: a list of trace IDs, the current working hypothesis, and any open questions. Without these, the output will be generic and unsafe for operational use.

Do not use this prompt for real-time alerting, as it is designed for asynchronous handoff, not automated incident detection. It is also inappropriate for postmortem authoring; a postmortem requires a broader business-impact and timeline analysis that this prompt explicitly scopes out. The prompt assumes the input traces are already correlated to the same incident. If you feed it traces from unrelated failures, it will hallucinate a false correlation. Always validate that the input trace set shares a common incident fingerprint—such as a matching deployment marker, time window, or error signature—before invoking this prompt. In high-severity incidents, the output should be treated as a draft that requires human review before being shared as the authoritative handoff document.

Before using this prompt, ensure you have gathered the required inputs: affected components, key trace IDs, your current hypothesis, and a list of open questions. The prompt is most effective when the incoming responder has enough context to continue the investigation without re-interviewing the outgoing shift. After generating the summary, validate that every claim in the 'Key Findings' section can be traced back to a specific trace ID in the input set. If the model introduces a component or failure mode not present in the provided traces, flag it for removal before handoff.

This prompt template is designed to be the core of your incident handoff automation. It takes a batch of raw or pre-processed production traces and instructs the model to produce a concise, structured brief. The goal is to preserve diagnostic signal—affected components, key trace IDs, the current working hypothesis, and open questions—while stripping away the raw verbosity that slows down the next on-call engineer. Use this template as the foundation for a tool, a Slack command, or an automated runbook step.

textCopy
You are an AI SRE assistant preparing a shift handoff for an active incident.

Your input is a set of production traces related to incident [INCIDENT_ID].

[TRACE_DATA]

Analyze the traces and produce a structured handoff summary. Follow these rules:
- Focus on patterns, not individual log lines.
- If a trace is incomplete or ambiguous, note it as an open question.
- Do not invent root causes not supported by the trace evidence.
- If the evidence is contradictory, state the contradiction explicitly.

[OUTPUT_SCHEMA]
Return a JSON object with the following fields:
- "incident_id": string
- "summary": string (a 2-3 sentence executive summary of the incident state)
- "affected_components": list of strings (services, models, tools, or infrastructure pieces showing errors)
- "key_trace_ids": list of strings (the 3-5 most representative trace IDs for this incident)
- "current_hypothesis": string (the leading theory for the root cause, with a confidence level: High/Medium/Low)
- "open_questions": list of strings (specific, actionable unknowns to investigate next)
- "recommended_next_checks": list of strings (concrete diagnostic steps for the next shift, e.g., 'Check auth service logs for 403 spikes')
- "evidence_summary": list of objects with "trace_id" and "finding" (a 1-sentence key finding from that trace)

[CONSTRAINTS]
- If the trace data is insufficient to form a hypothesis, set "current_hypothesis" to "Insufficient data" and confidence to "Low".
- Do not include raw stack traces in the summary.
- Limit "key_trace_ids" to a maximum of 5.

To adapt this template, replace the [TRACE_DATA] placeholder with your actual trace payload. This can be a pre-processed JSON array of spans and errors, or raw log lines if your context window allows. The [OUTPUT_SCHEMA] is the contract your downstream tooling should validate against. If you are wiring this into an automation, ensure the JSON output is parsed and checked for required fields before it is posted to an incident channel or ticket. For high-severity incidents, always flag the output for human review before the shift handoff is finalized.

This prompt is designed to run as a batch summarization step during incident handoff, not as a real-time streaming call. The implementation harness should accept a structured payload containing the incident ID, a list of trace objects (each with trace ID, spans, error messages, and timestamps), the current hypothesis, and any open questions. The harness calls the LLM once per handoff batch, validates the output against the expected schema, and writes the result to the handoff channel (e.g., Slack, PagerDuty, incident management platform). Because this is an SRE workflow, the harness must never silently drop a failed summary—if validation fails after retries, the system should post a degraded handoff note with raw trace IDs and a human_review_required flag.

Model choice and tool use: Use a model with strong structured-output support (GPT-4o with response_format, Claude 3.5 Sonnet with tool-use mode, or equivalent). Set temperature to 0 or near-zero to maximize consistency across handoff shifts. The prompt does not require external tools or retrieval—all trace data must be provided inline in the [TRACE_BATCH] placeholder. If the trace batch exceeds the model's context window, the harness should pre-filter traces by severity, recency, or anomaly score before calling the prompt. A common pattern is to sort traces by error_budget_impact descending and take the top N that fit within a safe token budget (e.g., 80% of the model's context limit).

Validation and retry logic: After the model returns, validate the output against a strict schema: affected_components must be a non-empty array of strings, key_trace_ids must be a non-empty array of valid trace ID strings, current_hypothesis must be a non-empty string, open_questions must be an array (can be empty), and recommended_next_checks must be a non-empty array of strings. If validation fails, retry once with the validation errors appended to the prompt as [PREVIOUS_ERRORS]. If the second attempt also fails, log the raw response, emit a handoff_summary_failed metric, and fall back to a template-based summary that includes the raw trace IDs and a note that automated summarization was unavailable. Never retry more than twice—SRE handoffs are time-sensitive, and a partial summary with trace IDs is better than a delayed handoff.

Logging and audit trail: Every summarization call should log the incident ID, the number of traces provided, the model used, the token counts (input and output), the validation result, and the final summary text. Store these logs in the same observability platform that holds the raw traces (e.g., Datadog, Grafana, Honeycomb) so that incident reviewers can trace a handoff summary back to its source data. If the summary is used in a postmortem document, include a reference to the log entry so that the evidence chain is auditable. For regulated environments, consider adding a human approval step before the summary is posted to the handoff channel—the harness can route the validated output to an on-call reviewer with a 5-minute timeout before auto-posting.

What to avoid: Do not call this prompt on every individual trace—it is designed for batch summarization of an incident's trace set. Do not use it for real-time alerting; latency-tolerant models and validation steps make it unsuitable for sub-second response requirements. Do not include raw PII or secrets in the trace data passed to the model; redact sensitive fields in the harness before assembly. Finally, do not treat the summary as the sole source of truth for the incident—always preserve links to the raw traces so the next shift can drill into details the summary may have compressed or omitted.