Agent Behavior Anomaly Summary Prompt Template

This prompt is designed for AI reliability engineers and SREs who need to brief product leads on agent behavior anomalies detected in production traces. Use it when you have already identified a set of anomalous agent sessions through automated monitoring or trace review, and you need to produce a structured summary that explains what deviated, how severe the anomalies were, and whether patterns cluster across sessions. The output is a stakeholder-ready report that preserves diagnostic signal without requiring the reader to interpret raw trace data.

Do not use this prompt for real-time alerting, single-trace root-cause analysis, or executive-level metric dashboards. It assumes anomaly detection has already occurred and focuses on aggregation, severity scoring, and pattern explanation. The prompt requires a pre-filtered set of anomalous sessions as input—feeding it unfiltered production traces will produce noisy, unactionable output. Before invoking this prompt, confirm that each session in the input set has been flagged by a monitoring system, eval failure detector, or human review queue. The prompt also expects a defined severity rubric (e.g., minor deviation, significant plan divergence, critical tool misuse) and a clustering dimension (e.g., by failure mode, by agent capability, by time window) to produce meaningful pattern analysis.

This prompt is not a substitute for incident response workflows. When an active incident is in progress, use the Incident Root-Cause Summary Prompt Template instead to isolate causal chains from correlated traces. Reserve this prompt for post-incident analysis, recurring anomaly review cadences, or pre-briefing product leads before a stakeholder meeting. The output should be treated as a diagnostic summary, not a final decision document—always link back to the underlying trace IDs so reviewers can drill into raw evidence if needed.

This template is designed to be pasted directly into your orchestration layer. It instructs the model to act as an AI reliability engineer, analyzing aggregated trace data to produce a structured anomaly report for a product lead. The prompt uses square-bracket placeholders for all dynamic inputs, such as the raw trace data, the specific agent's intended plan, and the desired output schema. Before using this template, ensure you have assembled the necessary trace data, including agent plans, tool-call logs, and state snapshots, from your observability platform. The prompt's strict instructions and output schema are designed to prevent the model from inventing patterns or smoothing over critical diagnostic signals.

textCopy
You are an AI Reliability Engineer briefing a product lead. Your task is to analyze aggregated agent trace data and produce a structured report on behavior anomalies. Be precise, evidence-based, and avoid speculation. Do not invent data. If the evidence is inconclusive, state that clearly.

**Input Data:**
- Agent Plan: [AGENT_PLAN]
- Tool-Call Logs: [TOOL_CALL_LOGS]
- State Snapshots: [STATE_SNAPSHOTS]
- Expected Behavior Policy: [EXPECTED_BEHAVIOR_POLICY]

**Instructions:**
1.  **Identify Deviations:** Compare the [TOOL_CALL_LOGS] and [STATE_SNAPSHOTS] against the [AGENT_PLAN] and [EXPECTED_BEHAVIOR_POLICY]. List every significant deviation.
2.  **Cluster Anomalies:** Group related deviations into coherent anomaly clusters (e.g., 'Repetitive Tool Loop', 'Incorrect Argument Pattern', 'State Update Failure').
3.  **Assess Severity:** For each cluster, assign a severity score (LOW, MEDIUM, HIGH, CRITICAL) based on the deviation from the expected plan and the potential for downstream failure. Justify the score with trace evidence.
4.  **Generate Summary:** Write a concise executive summary of the findings, highlighting the most critical anomalies and their potential business impact.

**Output Format:**
You must produce a JSON object conforming to this exact schema:
{
  "executive_summary": "string",
  "anomaly_clusters": [
    {
      "cluster_name": "string",
      "description": "string",
      "severity": "LOW" | "MEDIUM" | "HIGH" | "CRITICAL",
      "severity_justification": "string",
      "related_trace_ids": ["string"],
      "frequency": "integer"
    }
  ],
  "overall_assessment": "string"
}

**Constraints:**
- Base every finding on explicit evidence from the provided [TOOL_CALL_LOGS] and [STATE_SNAPSHOTS].
- Do not hallucinate trace IDs or data points.
- If no anomalies are found, the `anomaly_clusters` array should be empty, and the `executive_summary` should state that behavior was nominal.

To adapt this template, replace the placeholders with data from your production trace pipeline. The [AGENT_PLAN] should be the high-level task decomposition the agent generated. [TOOL_CALL_LOGS] should be a structured list of function calls with their arguments and results. [STATE_SNAPSHOTS] are key-value pairs representing the agent's memory or working state at critical steps. The [EXPECTED_BEHAVIOR_POLICY] is a natural language description of the rules the agent should follow, such as 'never call the same tool with identical arguments more than twice' or 'always update the task status after a tool call.' For high-stakes applications, always route the final JSON output through a validation step that checks for schema compliance and the presence of related_trace_ids before it reaches a human reviewer.

The Agent Behavior Anomaly Summary prompt is designed to run as a batch or on-demand analysis step within an observability pipeline, not as a real-time user-facing endpoint. It consumes structured trace segments—agent plans, tool-call sequences, and state snapshots—and produces a structured anomaly report. The primary integration point is a scheduled job (e.g., a cron-triggered serverless function or an Airflow DAG) that queries a trace database, assembles the input payload, calls the model, validates the output, and writes the report to a dashboard or notification system. This prompt is high-stakes for diagnosis but low-risk for direct user harm, so the harness should prioritize diagnostic accuracy and reproducibility over real-time latency.

Input assembly is the most critical pre-processing step. Before calling the prompt, the harness must: (1) query the trace store for sessions within the analysis window that contain agent plan deviations, unexpected tool sequences, or state anomalies; (2) deduplicate and normalize trace events into a consistent schema (e.g., {timestamp, agent_id, plan_step, tool_name, tool_args, state_diff, expected_next_step}); (3) chunk sessions if the trace volume exceeds the model's context window, producing per-session summaries that can be aggregated later. Validation after the model responds must check: (a) the output is valid JSON matching the expected schema; (b) each anomaly has a severity_score between 0.0 and 1.0; (c) pattern_cluster_id values are consistent across similar anomalies; (d) no hallucinated trace IDs appear that weren't in the input. If validation fails, retry with a repair prompt that includes the validation errors. After three failed retries, escalate to a human reviewer with the raw trace and partial output.

Model choice matters here. Use a model with strong structured-output capabilities and a large context window (e.g., Claude 3.5 Sonnet or GPT-4o with response_format set to json_schema). For high-volume weekly runs, consider a cheaper model for the initial pass and escalate ambiguous cases to a more capable model. Logging must capture: the input trace IDs, the model version, the raw and validated output, validation errors, retry count, and any human review decisions. Store these in an append-only audit table so anomaly trends can be tracked over time. Human review is required when: severity scores exceed 0.8, a new pattern cluster appears that wasn't seen in prior runs, or the model's confidence (if requested) falls below a threshold. The review interface should display the original trace alongside the model's summary, with one-click accept, edit, or reject actions.

What to avoid: Do not run this prompt on raw, unnormalized traces—the model will hallucinate structure. Do not skip schema validation; malformed anomaly reports will break downstream dashboards. Do not treat the severity score as a ground-truth metric without human calibration; use it as a triage signal. Finally, do not use this prompt for real-time alerting. It is a diagnostic summarization tool, not a paging system. Pair it with a separate rules-based alerting layer that triggers on known failure signatures, and use this prompt for the deeper investigation that follows.