Trace Similarity Scoring Prompt for Incident Clusters

This prompt is designed for Site Reliability Engineers (SREs) and AI Operations engineers who are responding to an active incident and need to group a large volume of related production traces. The core job-to-be-done is to move from a noisy, unorganized list of failing traces to a structured set of incident clusters, each defined by a shared failure signature. The prompt computes pairwise similarity scores based on critical trace attributes—failure mode, tool-call sequence, error messages, and latency profile—and outputs clustered groups with representative exemplars. This allows the responder to quickly understand how many distinct problems are occurring, which traces are most representative of each problem, and where to focus diagnostic effort first.

Use this prompt when you have already collected a batch of failing traces from your observability platform and suspect they represent multiple underlying issues. The required input is a structured dataset of traces, each containing at minimum a trace ID, a failure mode or error code, a sequence of tool calls or spans, and a latency profile. The prompt is most effective when the traces are pre-filtered for a specific time window and service context. Do not use this prompt for real-time alert triage on individual traces; it is designed for batch correlation during an active incident review. It is also not a replacement for a root-cause analysis—it identifies clusters of similar failures, which then become the input for deeper diagnostic prompts like the Root-Cause Isolation Prompt for Agent Loop Failures.

Before using this prompt, ensure you have a clear threshold for what constitutes a 'similar' trace. The prompt will recommend a similarity threshold for automated grouping, but you should validate this against your own operational context. A threshold that is too low will create many small, noisy clusters; a threshold that is too high will merge distinct incidents into one. After running the prompt, the next step is to take the top one or two representative exemplar traces from each cluster and feed them into a detailed trace review or root-cause isolation workflow. Avoid the temptation to treat the similarity scores as ground truth—they are a prioritization and grouping aid, and ambiguous cases should be flagged for human review.

The prompt below accepts a batch of production traces and returns clustered groups with similarity scores and representative exemplar traces. It is designed to be dropped into an incident response workflow where an SRE or AI operations engineer needs to quickly determine whether multiple failures share a common root cause or represent distinct problems. The template uses square-bracket placeholders for all variable inputs so you can wire it into a script, observability pipeline, or manual triage tool without modification.

textCopy
You are an incident trace analyst. Your task is to compute pairwise similarity scores across a set of production traces and group them into clusters that likely share a common root cause.

## INPUT TRACES
[TRACE_BATCH]

## SIMILARITY DIMENSIONS
Score each trace pair on a 0.0 to 1.0 scale across these dimensions:
1. **Failure mode** — Do the traces end with the same error type, refusal pattern, or output degradation?
2. **Tool-call sequence** — Do the traces share a similar sequence of tool calls, including argument patterns and ordering?
3. **Error message signature** — Do error messages, stack traces, or status codes match or share structural similarity?
4. **Latency profile** — Do the traces exhibit similar per-step latency patterns, timeouts, or bottleneck locations?
5. **Input characteristics** — Do the user inputs or triggering requests share semantic or structural similarity?

## OUTPUT REQUIREMENTS
Return a JSON object with this exact schema:
{
  "clusters": [
    {
      "cluster_id": "string",
      "trace_ids": ["string"],
      "mean_similarity_score": 0.0,
      "dominant_failure_mode": "string",
      "exemplar_trace_id": "string",
      "exemplar_summary": "string",
      "shared_characteristics": ["string"],
      "recommended_grouping_threshold": 0.0
    }
  ],
  "unclustered_trace_ids": ["string"],
  "pairwise_scores": [
    {
      "trace_id_a": "string",
      "trace_id_b": "string",
      "failure_mode_score": 0.0,
      "tool_call_score": 0.0,
      "error_signature_score": 0.0,
      "latency_profile_score": 0.0,
      "input_similarity_score": 0.0,
      "composite_score": 0.0
    }
  ],
  "grouping_recommendation": "string"
}

## CONSTRAINTS
[CONSTRAINTS]

## EXAMPLES
[EXAMPLES]

## OUTPUT
Return only the JSON object. Do not include explanations, markdown fences, or additional text.

Adapting the template. Replace [TRACE_BATCH] with a structured array of trace objects, each containing at minimum a trace_id, spans with tool-call and error details, latency_per_step, and final_output or error. The [CONSTRAINTS] placeholder should specify operational boundaries such as "Only group traces where the composite similarity score exceeds 0.7" or "Do not merge clusters if the dominant failure mode differs." The [EXAMPLES] placeholder should contain one or two worked examples showing correct clustering output for a small trace set—this is critical for calibration because similarity scoring is subjective without demonstrations. If you are wiring this into an automated pipeline, consider adding a [RISK_LEVEL] field that triggers human review when clusters contain traces from high-severity incidents or when the model assigns low confidence to a grouping decision.

Validation and next steps. Before relying on this prompt in production, validate the output against a hand-labeled set of incident traces where you already know the correct clusters. Measure cluster purity (do grouped traces actually share a root cause?) and false-positive rate (are unrelated traces being merged?). If the model consistently over-groups or under-groups, adjust the similarity thresholds in [CONSTRAINTS] or add counterexamples to [EXAMPLES]. For high-stakes incidents, always require a human responder to review the clustering output before taking automated action such as merging incident tickets or suppressing alerts. Wire the JSON output into your incident management tool so responders can click through from a cluster to the individual trace views.

The Trace Similarity Scoring Prompt is designed to be called programmatically during an active incident, not as a one-off chat interaction. The implementation harness must accept a batch of production traces—typically extracted from your observability platform (e.g., LangSmith, Arize, Datadog LLM)—and feed them into the prompt with a consistent schema. Each trace should be pre-processed into a canonical representation that includes: failure mode classification, ordered tool-call sequence with status codes, error messages and stack traces, latency profile per span, and the final output or refusal message. This normalization step is critical because the prompt's similarity scoring depends on structured, comparable fields rather than raw log lines. Without pre-processing, the model will spend tokens on parsing inconsistent trace formats instead of computing meaningful similarity scores.

Wire the prompt into your incident response pipeline as a stateless function with the following contract: input is a JSON array of trace objects conforming to the [TRACE_BATCH] schema, and output is a JSON object containing cluster assignments, pairwise similarity scores, representative exemplar traces, and a recommended grouping threshold. Implement a validation layer that checks the output schema before accepting results—verify that every trace ID in the input appears in exactly one cluster, that similarity scores are numeric values between 0 and 1, and that exemplar traces are valid references to input trace IDs. If validation fails, retry once with the same input and an added [CONSTRAINTS] instruction emphasizing schema compliance. After a second failure, log the malformed output and escalate to the on-call engineer with the raw model response attached. For model selection, prefer a model with strong JSON mode and long-context capabilities (e.g., Claude 3.5 Sonnet or GPT-4o with structured outputs enabled) because trace batches can be large and the pairwise comparison task is token-intensive. Set temperature=0 to maximize deterministic clustering behavior.

The recommended grouping threshold from the prompt's output should be treated as a suggestion, not an automatic action. Implement a human review gate that displays the proposed clusters, similarity scores, and exemplar traces in your incident dashboard. The SRE should be able to adjust the threshold, merge or split clusters, and annotate the rationale before the grouping is committed to the incident timeline. This review step is essential because similarity scoring can produce false clusters when traces share superficial features (e.g., both timed out) but have different root causes (e.g., network partition vs. API key expiration). Log every invocation—input trace count, cluster count, threshold recommendation, human adjustments, and final grouping—for post-incident analysis and prompt improvement. Avoid wiring this prompt directly into automated remediation without human approval, as incorrect clustering during an incident can misdirect the response effort and delay root-cause isolation.