Tool-Call Failure Summary Prompt Template

This prompt is designed for platform engineers and AI reliability leads who need to convert raw, verbose tool-call failure traces into a structured, decision-ready report for product stakeholders. The core job-to-be-done is aggregation and translation: taking dozens of individual tool_call failure events, which are often scattered across multiple traces and sessions, and condensing them into meaningful categories, frequency trends, and a clear assessment of user-facing impact. The ideal user has access to production trace data but needs to communicate reliability without exposing the full complexity of the underlying logs. Required context includes a batch of failure traces, the tool definitions involved, and the time window under analysis.

Use this prompt when you are preparing a weekly reliability summary, a post-incident review, or a monthly stakeholder update focused specifically on tool-call health. It is effective for identifying systemic issues, such as a specific tool that consistently fails due to malformed arguments or an external API that has begun timing out. The prompt's value lies in its ability to enforce a consistent output schema that separates failure categorization (e.g., 'Timeout', 'Invalid Argument', 'Authorization Failure') from impact assessment (e.g., 'User could not complete checkout', 'Agent stalled and required human takeover'). This structured separation is what makes the output actionable for product managers who need to prioritize bug fixes.

Do not use this prompt for real-time alerting or single-trace debugging. If you are diagnosing why a single create_ticket call failed for user ID 42, you need a trace inspection prompt, not an aggregation prompt. Similarly, this prompt is not a substitute for a metrics dashboard; it is a narrative reporting tool. Avoid feeding it uncleaned traces that still contain sensitive user data—ensure PII redaction has already occurred upstream. The next step after reading this section is to prepare your trace data in the expected input format and review the prompt template to understand how to set the [FAILURE_TRACES], [TOOL_DEFINITIONS], and [REPORTING_PERIOD] placeholders.

This prompt template transforms raw tool-call failure data from your trace aggregation pipeline into a structured summary report. It is designed for platform engineers who need to communicate tool reliability to product stakeholders without exposing them to verbose trace logs. The template expects pre-aggregated failure data—do not send raw trace events directly to the model. Instead, run your aggregation queries first, then populate the placeholders with summarized counts, categorized failures, and trend data before invoking the model.

textCopy
You are an AI reliability analyst producing a tool-call failure summary for product stakeholders. Your audience is non-engineering decision-makers who need to understand tool reliability trends, business impact, and recommended actions without raw trace detail.

## INPUT DATA

Failure summary period: [REPORTING_PERIOD]
Total tool calls in period: [TOTAL_TOOL_CALLS]
Total failures: [TOTAL_FAILURES]
Overall failure rate: [FAILURE_RATE_PERCENT]%

Failure categories with counts:
[FAILURE_CATEGORIES_TABLE]

Top 5 failing tools with failure counts and rates:
[TOP_FAILING_TOOLS]

Failure trend vs previous period:
[TREND_COMPARISON]

Estimated user impact:
[USER_IMPACT_ESTIMATE]

Notable incident IDs for reference:
[INCIDENT_IDS]

## OUTPUT SCHEMA

Produce a structured report with exactly these sections:

1. **Executive Summary** (2-3 sentences): Overall reliability picture, primary concern, and trend direction.
2. **Failure Breakdown by Category** (table): Category name, failure count, percentage of total failures, trend arrow (↑↓→), and one-line explanation.
3. **Top Failing Tools** (table): Tool name, failure count, failure rate, primary failure category, and business impact level (Critical/High/Medium/Low).
4. **Trend Analysis** (2-3 sentences): Comparison to previous period with magnitude and direction.
5. **User Impact Assessment** (2-3 sentences): Affected features, estimated user exposure, and severity.
6. **Recommended Actions** (bulleted list, max 5 items): Concrete next steps prioritized by impact.
7. **Data Quality Notes** (1-2 sentences): Any caveats about the underlying trace data.

## CONSTRAINTS

- Do not fabricate failure categories or tools not present in the input data.
- If a trend comparison is unavailable, state "No prior period data available" rather than guessing.
- Classify business impact conservatively: only mark Critical if user-facing functionality is blocked.
- Keep the report under 500 words total.
- Use plain language suitable for a product manager or director.
- If total failures is zero, produce a brief positive summary with all sections noting no failures.
- Do not include raw trace IDs, stack traces, or internal error codes.

## RISK LEVEL: [RISK_LEVEL]

If RISK_LEVEL is HIGH, add a **Risk Escalation Note** section at the top explaining why immediate attention is required.

Adaptation guidance: Replace each square-bracket placeholder with data from your trace aggregation pipeline before sending to the model. The [FAILURE_CATEGORIES_TABLE] placeholder should contain pre-categorized counts (e.g., timeout, auth error, malformed arguments, rate limit). The [TOP_FAILING_TOOLS] placeholder expects a ranked list with counts and rates. If your observability platform does not support trend comparison, populate [TREND_COMPARISON] with "No prior period data available" rather than omitting it. The [RISK_LEVEL] placeholder accepts HIGH, MEDIUM, or LOW and controls whether the escalation note section appears. Validate the model's output against the schema before sharing with stakeholders—if any required section is missing or the word count exceeds 500, retry with stricter constraints or repair the output programmatically.

This prompt is designed to be integrated into an automated trace analysis pipeline, not run as a one-off chat. The typical harness ingests raw tool-call logs from your observability platform (e.g., LangSmith, Arize, Datadog LLM Observability), batches them by time window or feature, and feeds them into the prompt template. The output is a structured JSON report that can be stored in a reporting database, rendered in a dashboard, or attached to a recurring stakeholder email. The key integration points are: (1) a pre-processing step that extracts and normalizes tool-call failure events from raw traces, (2) the prompt execution itself with model selection and parameter tuning, (3) a post-processing validation layer that checks the output schema, severity classifications, and attribution accuracy, and (4) a routing step that determines whether the report goes directly to stakeholders or through a human review queue based on the failure severity and impact assessment.

For production use, implement a validation wrapper around the model response. Before the report reaches a stakeholder, validate that: the JSON schema matches the expected structure (required fields: failure_categories, frequency_trends, impact_assessment, top_failures), severity classifications use only the allowed enum values (CRITICAL, HIGH, MEDIUM, LOW), and each failure entry includes a trace_id reference for auditability. If validation fails, retry the prompt once with the validation errors appended as additional context. If the retry also fails, route the raw trace data and partial output to a human reviewer with a flag indicating the automation gap. For high-severity reports (any CRITICAL failure or impact assessment above a defined threshold), always require human approval before distribution. Log every prompt execution with the input trace window, model version, output, validation result, and reviewer decision for governance and future prompt improvement.

Model choice matters here. Use a model with strong JSON mode and long-context handling (e.g., GPT-4o, Claude 3.5 Sonnet) because trace windows can be large and the output schema is strict. Set temperature to 0 or near-zero to maximize consistency across reports. If your trace volume exceeds the context window, pre-aggregate failures by category and tool name before passing them to the prompt, or use a map-reduce pattern where the prompt processes one tool or service at a time and a second synthesis prompt combines the results. Avoid streaming for this use case—you need the complete, validated JSON object before any downstream step consumes it. For teams using RAG, do not retrieve external documents for this prompt; the only evidence should be the trace data provided in [TRACE_DATA]. Adding external context risks contaminating the failure attribution with irrelevant information.