Multi-Trace Failure Pattern Detection Prompt

This prompt is designed for AI SREs and incident responders who need to analyze a batch of production traces during an active incident. Instead of manually inspecting each trace, you feed a set of traces into this prompt to identify recurring failure signatures such as repeated tool-call errors, identical refusal patterns, or shared latency spikes. The prompt outputs a ranked list of common failure clusters with supporting trace IDs, enabling rapid root-cause isolation. Use this when you have 10 or more traces from the incident window and need to determine whether failures are correlated or independent.

The prompt requires structured trace data that includes span details, error messages, tool-call arguments, and latency measurements. It is most effective when traces are pre-normalized into a consistent JSON schema before submission. Do not use this for single-trace diagnosis—the clustering logic depends on cross-trace comparison and will produce unreliable results with fewer than 10 samples. Similarly, avoid using this prompt for traces that lack structured span and error data, as the failure pattern detection relies on these fields to compute similarity scores and cluster assignments.

Before running this prompt, ensure you have isolated traces from the incident time window and excluded healthy baseline traces to reduce noise. The prompt includes eval criteria for cluster purity and false-positive suppression, but you should still review the output for clusters that contain fewer than three traces or have low confidence scores. For high-severity incidents, always pair the automated clustering with human review of the representative exemplar traces before declaring a root cause or initiating a fix.

This prompt is designed to be pasted directly into your incident response workflow. It ingests a set of production traces and performs comparative analysis to surface common failure patterns—such as repeated tool-call errors, identical refusal messages, or shared latency spikes. The goal is to move you from reviewing individual traces to identifying systemic issues quickly. Replace every square-bracket placeholder with actual data from your observability platform before execution.

textCopy
You are an AI SRE analyzing a batch of production traces during an active incident. Your task is to compare the provided traces and identify recurring failure signatures. Do not summarize individual traces in isolation. Instead, look for patterns that repeat across multiple traces.

## INPUT TRACES
[TRACE_BATCH]

## ANALYSIS INSTRUCTIONS
1. **Group traces by failure mode.** Look for shared error messages, identical tool-call failures, repeated refusal patterns, common latency spike locations, or matching malformed output structures.
2. **Rank the failure clusters** by frequency (number of affected traces) and severity (user-facing error, silent degradation, timeout).
3. **For each cluster**, provide:
   - A concise name for the failure pattern.
   - The number of traces exhibiting the pattern.
   - A representative trace ID.
   - The specific span, event, or log line that is the strongest evidence of the pattern.
   - A hypothesis for the root cause (e.g., model change, tool outage, prompt regression, infrastructure issue).
4. **Flag any traces that do not fit a cluster** as outliers with a brief note on why they are unique.
5. **Suppress false positives.** If a pattern appears in fewer than [MIN_CLUSTER_SIZE] traces, do not report it as a cluster. Note it only if it is high severity.

## OUTPUT FORMAT
Return a JSON object with the following schema:
{
  "incident_id": "[INCIDENT_ID]",
  "analysis_timestamp": "[TIMESTAMP]",
  "total_traces_analyzed": [NUMBER],
  "failure_clusters": [
    {
      "cluster_name": "string",
      "trace_count": [NUMBER],
      "severity": "critical|high|medium|low",
      "representative_trace_id": "string",
      "evidence_span_id": "string",
      "evidence_summary": "string",
      "root_cause_hypothesis": "string",
      "affected_component": "string"
    }
  ],
  "outliers": [
    {
      "trace_id": "string",
      "reason": "string"
    }
  ]
}

## CONSTRAINTS
- Do not fabricate evidence. Only use data present in the provided traces.
- If you cannot determine a pattern with confidence, state that explicitly.
- Do not recommend actions outside of observability and incident response.
- If the traces contain PII or sensitive data, do not include it in your analysis output.

To adapt this prompt, adjust the MIN_CLUSTER_SIZE threshold based on your traffic volume—higher for noisy systems, lower for low-traffic services where even two similar failures are significant. The TRACE_BATCH placeholder should be populated with structured trace data, ideally as a JSON array of spans, events, and metadata. If your observability platform outputs a different format, add a brief mapping instruction before the analysis instructions. For high-severity incidents, route the output to a human reviewer before taking automated remediation action.

This prompt is designed to be called programmatically during an active incident, not as a one-off chat interaction. The implementation harness should treat the prompt as a stateless function that accepts a batch of traces and returns a structured failure cluster report. The calling system—typically an incident management bot, a runbook automation script, or an observability platform's webhook handler—is responsible for assembling the trace batch, injecting it into the prompt template, parsing the JSON output, and routing the results to the incident channel. Because the prompt operates over potentially sensitive production traces, the harness must enforce data access controls and redaction rules before any trace data reaches the model.

The harness should implement a multi-step pipeline: 1) Trace Assembly: Query your observability backend (e.g., LangSmith, Arize, Datadog LLM, or a custom trace store) for traces within the incident window. Filter to traces with error spans, refusal signals, or latency exceeding a threshold. Serialize each trace into a compact representation—include trace_id, span_summaries, tool_call_sequences, error_messages, model_version, and prompt_version. 2) Prompt Injection: Insert the serialized trace batch into the [TRACE_BATCH] placeholder. Set [MIN_CLUSTER_SIZE] to a value appropriate for your traffic volume (start with 3–5 for low-volume services, 10+ for high-volume). Set [OUTPUT_SCHEMA] to require a JSON array of cluster objects, each with cluster_id, failure_signature, trace_ids, shared_root_cause_hypothesis, and confidence_score. 3) Validation: Parse the model's JSON output and validate it against the expected schema. Reject responses where trace_ids reference traces not present in the input batch, where confidence_score falls outside 0.0–1.0, or where clusters overlap beyond an acceptable Jaccard threshold. On validation failure, retry once with an explicit error message injected into the prompt context. 4) Routing: Post the validated cluster report to the incident channel, attach it to the incident ticket, and store it alongside the input trace batch for postmortem traceability.

Model choice matters here. Use a model with strong structured output capabilities and a context window large enough to hold your trace batch. Claude 3.5 Sonnet or GPT-4o are good defaults; avoid smaller models that may collapse multiple failure modes into a single cluster or hallucinate trace IDs. Set temperature=0 or a very low value to maximize deterministic clustering. For high-stakes production incidents, always include a human-review step before taking automated remediation action based on the prompt's output. The prompt identifies patterns—it does not authorize fixes. Log every invocation with the input trace count, output cluster count, validation status, and reviewer decision to build an audit trail for incident retrospectives and to measure the prompt's precision and recall over time.