Tool Call Regression Detection Suite Prompt Template

This prompt is a release gate for agent platform teams that need to validate tool-use behavior before shipping a new prompt version. It takes a set of test cases—each containing a user input and the expected tool call—runs them against the current system prompt and tool catalog, and produces a structured regression report. The primary job-to-be-done is answering one question: 'Did any previously passing tool-use scenario break with this prompt change?' The ideal user is an AI engineer or platform developer responsible for prompt versioning, CI/CD pipelines, or agent reliability. Required context includes a stable test suite of tool-use scenarios, the current system prompt under test, and the complete tool catalog with function schemas.

Use this prompt when you are cutting a prompt release candidate and need to prove that tool-call correctness hasn't regressed. It is designed for binary pass/fail evaluation per test case, with diff analysis that highlights exactly which tool calls changed between versions. The output is a structured regression report suitable for automated gating in a CI/CD pipeline—if any previously passing case now fails, the gate blocks the release. This prompt assumes you already have a golden dataset of expected tool calls. If you don't have that dataset, build it first using the Tool Selection Precision and Recall Evaluation Prompt or the Tool Call Pass/Fail Gate Criteria Prompt Template from this pillar.

Do not use this prompt for evaluating conversation quality, response fluency, single-turn text generation, or general-purpose model behavior. It is not a judge for tone, helpfulness, or safety policy adherence. It does not grade reasoning traces, argument quality, or whether the agent should have called a different tool that wasn't in the original test suite. For those concerns, use the Tool Selection Justification Trace Evaluation Prompt, the Function Argument Accuracy Evaluation Rubric Prompt, or the Tool Use Safety Policy Compliance Rubric Prompt. This prompt is a narrow, high-signal gate for tool-call correctness regression—nothing more. If you need to detect newly introduced hallucinations rather than regressions against a known baseline, pair this with the Tool Call Hallucination Rate Measurement Prompt Template.

This is the core prompt template for the Tool Call Regression Detection Suite. It is designed to be pasted into your evaluation harness, where you will replace the square-bracket placeholders with your specific test cases, baseline traces, and tool catalog. The prompt instructs the LLM judge to compare a new agent trace against a known-good baseline, identify differences, and produce a structured regression report with pass/fail decisions per test case. Use this template as the starting point for your CI/CD pipeline's evaluation stage before shipping a new prompt version or model upgrade.

textCopy
You are an expert evaluator for AI agent tool-use behavior. Your task is to run a regression detection suite by comparing a set of test case executions against a known baseline. You will receive a tool catalog, a list of test cases with their expected baseline tool calls, and the actual tool calls made by the agent under test. For each test case, determine if the agent's behavior has regressed from the baseline.

[TOOL_CATALOG]

[BASELINE_TRACES]

[ACTUAL_TRACES]

[REGRESSION_CRITERIA]

[OUTPUT_SCHEMA]

[CONSTRAINTS]

How to adapt this template: Replace [TOOL_CATALOG] with the full JSON Schema definitions of all available tools the agent can call. Replace [BASELINE_TRACES] with the known-good sequence of tool calls, arguments, and results for each test case from the previous approved version. Replace [ACTUAL_TRACES] with the new traces generated by the agent under test. [REGRESSION_CRITERIA] should define what constitutes a regression—such as a different tool being selected, a missing required argument, an out-of-order call, or a hallucinated parameter value. [OUTPUT_SCHEMA] must specify the exact JSON structure you expect back, including fields for test_case_id, pass, diff_summary, and severity. [CONSTRAINTS] can include rules like "ignore whitespace differences in string arguments" or "a timing change without a functional difference is not a regression." After adapting the template, run it through your evaluation harness and validate the output against your schema before accepting the results.

The Tool Call Regression Detection Suite prompt is designed to operate as a gate in a CI/CD pipeline for prompt version releases. It expects a structured input containing a baseline set of tool-call traces, a candidate set of traces from the new prompt version, and a test case manifest that defines expected behavior per scenario. The output is a machine-readable regression report with per-case pass/fail status and a diff analysis. To integrate this into an application, you should wrap the prompt in a harness that validates input schemas before submission, parses the JSON output, and routes failures to the appropriate logging and alerting channels.

Start by constructing the input payload as a JSON object with three required keys: baseline_traces, candidate_traces, and test_cases. Each trace should include the tool name, arguments, timestamp, and call sequence position. The test cases define the expected tool selection, argument constraints, and sequencing rules. Before sending to the model, validate that all required fields are present and that trace arrays are non-empty. Use a structured output mode (such as JSON mode or function calling with a strict schema) to enforce the output shape. The expected output schema includes a summary object with pass/fail counts, a cases array with per-test results, and a diffs array describing regressions. Implement a post-processing validator that checks for schema compliance and retries once with a clarification prompt if parsing fails.

Model choice matters here. Use a model with strong instruction-following and structured output capabilities, such as GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro. Avoid smaller or older models that may struggle with the multi-step comparison logic required for diff analysis. Set the temperature to 0 for deterministic results. For high-stakes release pipelines, route any case where the model reports a regression but confidence is below a threshold (e.g., confidence < 0.9) to a human review queue. Log every run with the prompt version, model, input hashes, and output report for auditability. If your test suite is large, consider batching test cases into groups of 20-30 to stay within context windows and reduce comparison errors.

Common failure modes include the model misinterpreting semantic equivalence in arguments (e.g., treating limit=50 and limit=50.0 as different), missing sequencing violations when call order is partially correct, and hallucinating diffs that don't exist. Mitigate these by providing clear examples of what constitutes a regression in your [EXAMPLES] placeholder, including edge cases around argument normalization. For production use, pair this prompt with a separate Argument Accuracy Evaluation Rubric to independently verify argument-level regressions. Never use this suite as the sole gate for safety-critical tool-use workflows—always supplement with human review for high-risk tool categories like write operations, financial transactions, or PII access.