Tool-Call Error Rate Monitoring Prompt Template

This prompt is designed for agent platform engineers and AI SREs who need to transform raw production trace data into actionable tool-call error rate reports. Use it when you have a batch of tool-call spans from your observability pipeline and need to produce a structured analysis that includes error rate trends by tool name, failure type classification, argument validation failure breakdowns, and error budget burn rate calculations. The ideal user has access to structured trace exports—typically JSON arrays of spans with tool_name, status, error_type, and timestamp fields—and needs to generate a diagnostic report that feeds into alert definitions and SLO reviews. This prompt assumes you have already extracted tool-call events from your tracing system and can provide them as structured input. It is not a replacement for real-time metric dashboards but serves as a diagnostic and reporting layer that bridges raw trace data and operational decision-making.

Before using this prompt, verify that your input data includes the minimum required fields: tool_name, timestamp, status (success/failure), and error_type where applicable. The prompt works best with batched data covering a defined evaluation window—typically 1 hour, 24 hours, or 7 days—aligned with your SLO reporting cadence. If your traces lack structured error classification or argument validation metadata, the prompt will still produce a partial analysis but will flag missing dimensions rather than hallucinating values. Do not use this prompt for real-time alerting; it is a batch analysis tool. For live threshold monitoring, pair this with a streaming metrics pipeline and use the output of this prompt to calibrate alert conditions. The prompt also assumes single-tenant analysis: if you are processing traces across multiple tenants or applications, run separate analyses to avoid cross-contamination of error budgets.

After running this prompt, you should have a structured report suitable for inclusion in an SLO review document or as input to an alert configuration workflow. The next step is to validate the output against your known error budget policies and compare the burn rate calculations with your incident response thresholds. Avoid using this prompt when you need sub-second latency on error detection—this is a diagnostic tool, not a real-time circuit breaker. If your tool-call volume exceeds 100,000 spans per batch, consider pre-aggregating by tool name and hour before passing data to the prompt to stay within context window limits. For high-risk production systems where tool failures can cause user-facing incidents, always have a human review the burn rate severity classification before triggering an incident response.

This prompt template is designed to be pasted directly into your monitoring workflow. It instructs the model to analyze a batch of production traces, focusing exclusively on tool-call invocations. The goal is to produce a structured report that identifies error rate trends by tool name, classifies failure types, and calculates the error budget burn rate. Replace every square-bracket placeholder with your specific trace data, tool catalog, and SLO configuration before execution.

textCopy
You are an AI SRE analyzing production traces for an AI agent. Your task is to monitor tool-call error rates.

## INPUT DATA
[TRACE_DATA]

## CONFIGURATION
Tool Catalog: [TOOL_CATALOG]
SLO Target (success rate): [SLO_TARGET]%
Evaluation Window: [EVALUATION_WINDOW]
Error Budget: [ERROR_BUDGET] minutes of downtime or failed calls

## TASK
1.  **Calculate Error Rates:** For each tool in the Tool Catalog, calculate the error rate as (failed calls / total calls) * 100 within the Evaluation Window.
2.  **Classify Failures:** Categorize each failed tool call into one of the following failure types:
    - `INVALID_ARGUMENT`: The model provided malformed or incorrect arguments.
    - `TOOL_NOT_FOUND`: The model attempted to call a non-existent tool.
    - `EXECUTION_FAILURE`: The tool was found and arguments were valid, but the tool's execution returned an error.
    - `TIMEOUT`: The tool call exceeded its allowed execution time.
    - `PERMISSION_DENIED`: The agent lacked the authorization to execute the tool.
3.  **Calculate Error Budget Burn Rate:** Determine the rate at which the error budget is being consumed. The burn rate is the percentage of the error budget consumed within the Evaluation Window, extrapolated to an hourly rate.
4.  **Identify Top Failure Contributors:** List the top 3 tools with the highest error rates and their primary failure type.

## OUTPUT FORMAT
You must respond with a single JSON object conforming to this schema:
{
  "evaluation_window": "string",
  "slo_target_percent": number,
  "overall_error_rate_percent": number,
  "error_budget_burn_rate_percent_per_hour": number,
  "tool_error_breakdown": [
    {
      "tool_name": "string",
      "total_calls": number,
      "failed_calls": number,
      "error_rate_percent": number,
      "failure_type_breakdown": {
        "INVALID_ARGUMENT": number,
        "TOOL_NOT_FOUND": number,
        "EXECUTION_FAILURE": number,
        "TIMEOUT": number,
        "PERMISSION_DENIED": number
      },
      "primary_failure_type": "string"
    }
  ],
  "top_failure_contributors": [
    {
      "tool_name": "string",
      "error_rate_percent": number,
      "primary_failure_type": "string"
    }
  ],
  "alert_triggered": boolean,
  "alert_reason": "string or null"
}

## CONSTRAINTS
- Only analyze tool-call spans within the provided [TRACE_DATA]. Ignore other spans.
- If a tool is not present in the [TOOL_CATALOG], flag it as `TOOL_NOT_FOUND`.
- If the error budget burn rate exceeds a critical threshold of 10% per hour, set `alert_triggered` to `true` and explain the reason.
- Base all calculations strictly on the provided data. Do not infer or assume data not present in the trace.

To adapt this template, start by replacing [TRACE_DATA] with a structured JSON array of your trace spans. Ensure each span object includes name, attributes (containing tool call details and error messages), and status. The [TOOL_CATALOG] should be a simple list of approved tool names. Set [SLO_TARGET] to your agreed-upon success rate (e.g., 99.9) and [EVALUATION_WINDOW] to the time range you are monitoring (e.g., 'last 1 hour'). The [ERROR_BUDGET] must be a concrete number representing your allowed failures. For high-stakes production environments, always route the output of this prompt to a validation step that checks the JSON schema before forwarding alerts to an on-call system. A common failure mode is an incorrect burn rate calculation; verify the math with a separate script if this metric directly triggers paging.

This prompt is designed to run as a scheduled evaluation step inside your existing observability stack, not as a one-off debugging query. It expects a batch of production trace spans—specifically tool-call events with their status codes, error messages, argument payloads, and timestamps—as structured input. The prompt's job is to classify failures, compute error rates by tool, and calculate error budget burn rates against your defined SLO targets. Wire it into a cron-triggered workflow (e.g., a GitHub Action, Airflow DAG, or AWS Step Function) that queries your trace store (Datadog, Grafana Tempo, or a custom log sink), formats the spans into the [TRACE_SPANS] placeholder, invokes the model, and routes the output to your alerting platform.

Validation and retry logic is critical because this prompt produces structured JSON that downstream alert rules consume. Implement a post-generation validation step that checks: (1) the output parses as valid JSON matching the expected schema, (2) all error_rate values are floats between 0 and 1, (3) burn_rate calculations are non-negative, and (4) failure_type classifications match your internal taxonomy. If validation fails, retry once with the validation errors appended to the [CONSTRAINTS] field. If the retry also fails, log the raw output and escalate to the on-call channel rather than silently dropping the monitoring window. For high-severity findings—burn rates above 5x or error rates crossing critical thresholds—route the structured alert payload directly to PagerDuty or your incident management platform with the trace correlation IDs attached.

Model choice and latency considerations matter here. This prompt performs classification and arithmetic reasoning over structured data, not creative generation. A fast model (e.g., Claude 3.5 Haiku, GPT-4o-mini, or a fine-tuned Llama variant) is usually sufficient and keeps per-window costs low. Set a 30-second timeout and a token limit appropriate for your batch size. If your trace volume is high, shard the input by tool name or time window and run parallel invocations. Log every invocation—input span count, output summary, validation status, and model latency—to your observability platform so you can detect when the monitoring prompt itself degrades. Finally, maintain a versioned prompt registry so that changes to the failure taxonomy or SLO targets are tracked alongside the prompt template, and run a weekly dry-run against a historical golden set of known-good and known-bad trace batches to catch regressions before they affect production alerting.