Tool-Call Error Correlation with Eval Failure Prompt

This prompt is designed for agent developers and AI reliability engineers who need to move beyond surface-level eval failures and identify the root cause in their tool-call layer. The core job-to-be-done is connecting a failed automated evaluation to a specific, upstream tool error within the same trace. Instead of manually inspecting logs, you provide structured tool-call logs and eval results, and the prompt produces a correlation matrix that maps tool errors to the eval dimensions they broke, complete with a severity score. The ideal user has access to trace-level observability data and is responsible for prioritizing engineering fixes in a complex agent pipeline.

Use this prompt when you have completed traces that contain both the tool-call execution details and the final automated evaluation scores. It is built for offline batch analysis, making it suitable for a weekly review of production failures or a post-deployment quality audit. The required context is strict: you must supply structured data for both [TOOL_CALL_LOG] and [EVAL_RESULTS] from the same trace. The prompt is not designed for real-time intervention on a live agent. Do not use it if your goal is to fix a single, in-flight conversation or if your eval data lacks trace-level attribution. It is also not a replacement for a real-time monitoring dashboard; its value is in systematic, batched root-cause analysis.

Before using this prompt, ensure your observability pipeline is capturing the necessary data. The tool-call log must include timestamps, function names, arguments, and error codes or success states. The eval results must be structured with named dimensions (e.g., 'faithfulness', 'completeness') and a clear pass/fail or numeric score. If your traces are incomplete or your evals are unstructured, the prompt will produce a low-confidence or incomplete matrix. The next step after running this analysis is to take the prioritized list of tool-eval correlations and use it to plan engineering work, such as hardening a specific API integration or adding argument validation before a tool call.

This prompt template is designed to be pasted directly into your observability pipeline, batch analysis script, or manual trace review workflow. It accepts structured tool-call logs and eval results from a single trace and produces a correlation matrix showing which tool errors propagated to which eval dimensions, along with severity scoring. The template uses square-bracket placeholders that you must replace with actual data before execution. All placeholders are mandatory unless marked optional in the adaptation notes below.

textCopy
You are an AI reliability engineer analyzing a production trace where an agent's tool calls may have caused downstream evaluation failures.

## INPUT DATA

### Tool-Call Log
[TOOL_CALL_LOG]

### Eval Results
[EVAL_RESULTS]

### Trace Metadata (Optional)
[TRACE_METADATA]

## OUTPUT SCHEMA

Return a JSON object with this exact structure:
{
  "correlation_matrix": [
    {
      "tool_call_id": "string",
      "tool_name": "string",
      "error_type": "string | null",
      "error_severity": "critical | high | medium | low | none",
      "affected_eval_dimensions": [
        {
          "dimension_name": "string",
          "impact_severity": "critical | high | medium | low",
          "causal_chain": "string explaining how the tool error caused the eval failure",
          "evidence_from_trace": "string citing specific trace events"
        }
      ],
      "recovery_possible": true | false,
      "recovery_suggestion": "string | null"
    }
  ],
  "unaffected_eval_failures": [
    {
      "dimension_name": "string",
      "failure_reason": "string explaining why this failure is NOT attributable to tool errors",
      "alternative_root_cause_hypothesis": "string"
    }
  ],
  "summary": {
    "total_tool_calls": number,
    "tool_calls_with_errors": number,
    "total_eval_failures": number,
    "eval_failures_explained_by_tool_errors": number,
    "dominant_failure_pattern": "string",
    "recommended_action": "string"
  }
}

## CONSTRAINTS
[CONSTRAINTS]

## ANALYSIS RULES

1. For each tool call in the log, determine if it produced an error (timeout, malformed arguments, unexpected response, missing required fields, permission denial, empty result).
2. For each tool error, trace forward through the trace to identify which eval dimensions were affected.
3. A causal chain must reference specific trace events (timestamps, step IDs, log lines) as evidence.
4. If a tool call succeeded but the eval still failed, classify those failures under "unaffected_eval_failures" with alternative hypotheses.
5. Severity scoring: "critical" means the eval failure makes the response unusable; "high" means significant quality degradation; "medium" means noticeable but not blocking; "low" means minor impact.
6. If [RISK_LEVEL] is "high", flag any correlation where tool errors affected safety, compliance, or factual accuracy dimensions for immediate human review.

## EXAMPLES
[EXAMPLES]

Adaptation notes: Replace [TOOL_CALL_LOG] with a structured JSON array of tool-call events including timestamps, tool names, arguments, responses, and error fields. Replace [EVAL_RESULTS] with a JSON array of eval dimension scores, thresholds, and failure flags. [TRACE_METADATA] is optional but should include trace ID, session ID, model version, and prompt version when available. [CONSTRAINTS] should specify any domain-specific rules, such as "ignore timeout errors under 200ms" or "treat citation failures as critical." [EXAMPLES] should include 1-2 few-shot examples of correct correlation matrices from your own production traces to improve output consistency. [RISK_LEVEL] should be set to "high" when analyzing traces from regulated domains, safety-critical systems, or customer-facing financial decisions. For high-risk traces, always route the output to a human reviewer before taking automated action such as prompt rollback or tool reconfiguration.

This prompt is designed to be called programmatically as part of an automated incident review or periodic quality audit workflow, not as a one-off chat interaction. The harness must supply structured tool-call logs and eval results from the same trace, formatted as the [TOOL_CALL_LOG] and [EVAL_RESULTS] inputs. These inputs should be extracted directly from your observability platform (e.g., LangSmith, Arize, Weights & Biases, or a custom trace store) and injected into the prompt template without manual reformatting. The prompt expects JSON arrays for both inputs, where each tool call includes tool_name, input_args, output_summary, error_code, and timestamp, and each eval result includes eval_name, score, threshold, pass_fail, and eval_dimension. If your trace store uses a different schema, build a thin transformation layer that normalizes the data before prompt assembly—do not ask the model to guess field mappings.

The implementation should include a validation wrapper around the model's JSON output. The expected output schema is a correlation matrix with tool_errors, eval_failures, correlations, and severity_summary objects. Before the output reaches any downstream dashboard or alerting system, validate that: (1) every tool_error_id in the correlations array exists in the tool_errors list, (2) every eval_failure_id exists in the eval_failures list, (3) severity_score values are integers between 1 and 5, and (4) the propagation_path field is a non-empty string. If validation fails, retry the prompt once with the validation errors appended as a [CONSTRAINTS] note. If the retry also fails, log the raw output and escalate for human review rather than silently ingesting malformed correlation data. For high-stakes production systems where eval failures trigger rollbacks or incident alerts, add a human approval step before the correlation matrix is used to generate automated remediation actions.

Model choice matters here. The prompt requires strong JSON-following behavior and the ability to reason across two structured datasets simultaneously. Use a model with proven structured output reliability (e.g., GPT-4o, Claude 3.5 Sonnet, or equivalent) and set temperature=0 to minimize variance in severity scoring. If you are processing high volumes of traces, batch multiple trace pairs into a single prompt call where each trace gets its own correlation analysis, but keep batch sizes small (3–5 traces) to avoid context-window pressure that degrades correlation accuracy. Log every prompt call with the trace ID, model version, input hashes, and output correlation matrix for auditability. This log becomes essential evidence when the correlation prompt itself is suspected of misattributing an eval failure to the wrong tool error.