Tool Failure Attribution Prompt for User Reports

This prompt is designed for platform engineers and AI SREs who need to attribute a user-reported failure to a specific tool call, argument error, or tool timeout by analyzing a production trace. The ideal user has access to raw trace data—including tool-call spans, arguments, responses, and timing metadata—and needs to move from a vague user complaint like 'the search didn't work' to a precise, actionable diagnosis. The prompt takes three inputs: the user's complaint, the associated trace data, and the tool definitions that were available to the model at runtime. It produces a structured tool-failure report identifying the exact failing tool call, the error context, and a recommended remediation.

Use this prompt when a user reports that the assistant couldn't complete a request, a tool-based action failed silently, or a response was incomplete due to a suspected tool issue. It is particularly valuable during incident triage, post-mortem analysis, and feedback-loop closure where support tickets must be mapped to engineering fixes. The prompt assumes you already have access to the trace data and tool definitions—it does not replace infrastructure monitoring or alerting systems. It is a diagnostic tool for post-hoc failure analysis, not a real-time guardrail. For high-severity incidents where the remediation involves a prompt change, retrieval pipeline update, or tool configuration modification, always require human review of the suggested fix before deployment.

Do not use this prompt when the failure is clearly a model reasoning error unrelated to tool use, when the trace data is incomplete or missing tool-call spans, or when the user complaint is too vague to map to a specific session. The prompt is also not suitable for aggregate trend analysis across many sessions—it is designed for single-session, single-complaint diagnosis. If you need to correlate multiple user reports to identify systemic tool failures, pair this prompt with a feedback-cluster analysis workflow and run it across a batch of traces with human sampling for false-attribution prevention. For regulated or high-risk domains where tool actions have compliance implications, ensure the output report is reviewed by a human before any remediation is applied to production systems.

This prompt template is designed to be pasted directly into your trace analysis workflow. It instructs the model to act as a platform engineer reviewing a user-reported failure against a provided trace log. The goal is to produce a structured tool-failure report that isolates the exact point of failure, rather than a general summary of the session. You must replace the square-bracket placeholders with actual data from your observability platform before execution.

textCopy
You are a platform engineer analyzing a production trace to attribute a user-reported failure to a specific tool call. Your analysis must be precise, evidence-based, and confined to the provided trace data.

[INPUT]
User Report: [USER_REPORT_DESCRIPTION]
Trace Data (JSON): [FULL_TRACE_JSON]

[CONSTRAINTS]
1. Identify the single most likely failing tool call. If multiple calls failed, identify the first one that caused the cascade.
2. For the identified tool call, extract the exact function name, input arguments, and the error message or timeout signal from the trace.
3. If the failure is due to a malformed argument, specify the exact argument and why it is invalid.
4. If the failure is a timeout, state the duration attempted and the timeout threshold if available in the trace.
5. Do not speculate about fixes outside the trace evidence. Your remediation recommendation must be directly traceable to the identified failure.
6. If no tool failure is evident in the trace, state that explicitly and suggest the failure may be in the model's reasoning, retrieval, or prompt instructions.

[OUTPUT_SCHEMA]
{
  "attribution_report": {
    "primary_failure_tool_call": {
      "function_name": "string | null",
      "span_id": "string | null",
      "input_arguments": {},
      "error_message": "string | null",
      "failure_type": "enum: [MALFORMED_ARGUMENT, TIMEOUT, EXECUTION_ERROR, AUTHORIZATION_ERROR, NOT_FOUND]"
    },
    "causal_chain": [
      {
        "step": "string",
        "trace_evidence": "string"
      }
    ],
    "root_cause_summary": "string",
    "recommended_remediation": "string",
    "confidence": "enum: [HIGH, MEDIUM, LOW]",
    "alternative_hypotheses": ["string"]
  }
}

[EXAMPLES]
User Report: "The assistant said it couldn't find my order but I provided the order ID."
Trace shows a call to `lookup_order` with argument `order_id: null`. The tool returned an error: "Missing required parameter: order_id".
Correct Attribution: The tool call `lookup_order` failed due to a MALFORMED_ARGUMENT. The `order_id` was null, likely because the model failed to extract it from the user's input.

User Report: "The report generation just hung and then said it failed."
Trace shows a call to `generate_report` with a start time and an end time 30 seconds later with a status of "TIMEOUT". The tool's timeout threshold is 30 seconds.
Correct Attribution: The tool call `generate_report` failed due to a TIMEOUT after 30 seconds, matching the tool's configured threshold.

After pasting this template, replace [USER_REPORT_DESCRIPTION] with the verbatim text from the user's support ticket or feedback channel. Replace [FULL_TRACE_JSON] with the complete trace object from your observability platform, ensuring it includes all spans, tool-call events, and error metadata. The output schema is designed to be machine-readable for automated ticketing systems, but the causal_chain and root_cause_summary fields provide the human-readable narrative needed for a postmortem. Before deploying this prompt into an automated pipeline, run it against a golden set of 20 known tool-failure traces to validate that primary_failure_tool_call.span_id and failure_type match your ground-truth labels with at least 90% accuracy.

The Tool Failure Attribution Prompt is designed to be integrated into an automated trace analysis pipeline, not used as a one-off chat interface. When a user report or support ticket is linked to a specific trace ID, the harness should fetch the full trace payload from your observability store (e.g., LangSmith, Arize, a custom OpenTelemetry collector), extract the relevant spans, and inject them into the prompt's [TRACE_JSON] placeholder. The [USER_REPORT] field should contain the verbatim user complaint, including any error messages they saw. The harness must also supply [TOOL_CATALOG] as a structured list of available tools with their expected argument schemas, so the model can distinguish between a tool timeout, an argument error, and a missing tool.

Before calling the LLM, validate the trace payload to ensure it contains at least one tool-call span. If the trace has no tool interactions, skip this prompt and route to a different diagnostic flow (e.g., a retrieval gap or model hallucination prompt). After receiving the model's output, parse the JSON response and validate that the attributed_tool_call_id field matches an actual span ID in the input trace. If the model attributes the failure to a tool call that does not exist in the trace, log a false-attribution event and retry with a stricter prompt variant that instructs the model to respond with "attributed_tool_call_id": null when no tool failure is found. This prevents the harness from acting on hallucinated attributions.

For high-severity incidents where the tool failure impacts billing, data integrity, or user safety, route the model's output to a human review queue before any automated remediation is triggered. The review interface should display the original trace visualization alongside the model's failure report, with the attributed span highlighted. Log every attribution result—including null attributions—to a dedicated tool_failure_attribution_log index with the trace ID, user report, model version, and attribution confidence score. This log becomes your ground-truth dataset for fine-tuning a smaller, faster attribution classifier later. Avoid wiring this prompt directly to automated rollback or tool-disable actions without human-in-the-loop approval for any action beyond a non-critical alert.