Example Set Tool Call Accuracy Test Prompt

This prompt is built for agent developers and prompt QA engineers who maintain few-shot example sets that teach a model when and how to call tools. Before shipping an agent, you need to verify that every tool-use example in your prompt selects the correct function and constructs valid arguments. Manual review misses argument shape errors, type mismatches, and hallucinated parameter names. This prompt automates that audit by comparing each example's tool call against the actual function signatures you provide.

Feed this prompt your complete tool definitions (JSON Schema or function declarations) and your full example set. It returns a structured accuracy report scoring each example on two dimensions: tool selection correctness (did the example call the right function?) and argument schema compliance (do the supplied arguments match the expected types, required fields, and constraints?). The report flags specific violations—missing required parameters, incorrect types, hallucinated field names, and wrong function choices—so you can fix problematic examples before they corrupt agent behavior in production.

Use this prompt before any release that changes tool signatures, adds new tools, or modifies example sets. It is also valuable during initial agent development when you are iterating on tool definitions and need to keep examples synchronized. Run it as a pre-commit check or as part of your CI pipeline for prompt changes. Do not use this prompt for evaluating runtime tool calls from live traffic—it is designed for offline example validation only. Runtime tool call evaluation requires different infrastructure, including execution feedback loops, side-effect monitoring, and latency considerations that this static analysis prompt does not address. For production monitoring, pair this with runtime observability prompts from the Prompt Observability and Trace Analysis pillar.

This prompt template is designed to be pasted directly into your testing harness. It evaluates each example in your tool-use demonstration set against the expected tool signature and argument schema. Replace the square-bracket placeholders with your actual tool definitions, example set, and evaluation criteria before running the test. The prompt instructs the model to act as an automated validator, comparing the tool call implied by each example against the ground-truth specification.

textCopy
You are an automated tool-call accuracy validator. Your task is to evaluate a set of few-shot examples that demonstrate tool use. For each example, you will compare the tool call shown in the example against the expected tool signature and argument schema defined below.

## TOOL DEFINITIONS
[TOOLS]

## EXAMPLE SET TO EVALUATE
[EXAMPLES]

## EVALUATION CRITERIA
For each example in the set, perform the following checks:
1. **Tool Selection Correctness:** Does the example select the correct tool from the available definitions? Score as `pass` or `fail`.
2. **Argument Schema Compliance:** Do the arguments passed in the example match the required and optional parameters, types, and constraints defined in the tool schema? Score as `pass` or `fail`.
3. **Argument Value Correctness:** For the specific input shown in the example, are the argument values logically correct and appropriate? Score as `pass` or `fail`.
4. **Overall Example Accuracy:** An example passes overall only if all three checks above pass.

## OUTPUT FORMAT
Return a JSON object with the following structure:
{
  "overall_accuracy_score": <float between 0.0 and 1.0>,
  "total_examples": <integer>,
  "passed_examples": <integer>,
  "failed_examples": <integer>,
  "per_example_results": [
    {
      "example_index": <integer>,
      "tool_selection_correct": <"pass" or "fail">,
      "argument_schema_compliant": <"pass" or "fail">,
      "argument_values_correct": <"pass" or "fail">,
      "overall_pass": <true or false>,
      "failure_notes": <string explaining the failure, or null if passed>
    }
  ],
  "summary_findings": <string summarizing common failure patterns>
}

## CONSTRAINTS
- Evaluate every example in the set. Do not skip any.
- If an example is ambiguous about which tool to call, flag it as a failure and explain the ambiguity.
- If an argument value cannot be verified from the example input, note this in `failure_notes` but do not automatically fail the example unless the schema requires a value that is missing.
- Do not invent tool definitions or examples. Use only what is provided.

To adapt this template, replace [TOOLS] with your complete function definitions in JSON Schema format, including function names, descriptions, and parameter specifications. Replace [EXAMPLES] with your full few-shot demonstration set, preserving the exact format your application uses (typically a sequence of user messages and assistant tool-call responses). For high-risk agent workflows where incorrect tool calls could trigger destructive actions, add a [RISK_LEVEL] placeholder that gates whether failures require human review before proceeding. After running the test, feed the output into your regression testing dashboard and flag any example with an overall_pass of false for immediate correction or removal.

The Example Set Tool Call Accuracy Test Prompt is designed to be a gate in a continuous integration pipeline for agent systems. Its primary job is to programmatically validate that a set of few-shot examples—which teach the model when and how to use tools—actually produces the correct tool selection and argument construction. You should wire this prompt into a pre-release testing workflow that runs whenever the example set, tool definitions, or system prompt changes. The harness must treat each example as a test case: send the example's input to the model under test, capture the model's tool call, and compare it against the expected tool call defined in the example's output. A simple pass/fail report is insufficient; the harness must produce structured diagnostics that pinpoint whether the failure was a wrong tool selection, a missing required argument, a type mismatch, or a hallucinated parameter.

To implement this, construct a test runner that iterates over your example set. For each example, extract the [INPUT] and the [EXPECTED_TOOL_CALL]. Send the input to the model with the full system prompt and tool definitions loaded. Parse the model's response to extract the actual tool call. Then, run a two-stage validation: first, check that the tool_name matches exactly. Second, validate the arguments against the expected schema. For argument validation, use a JSON Schema validator to confirm that all required fields are present, no extra fields exist, and each field's type matches. Log every mismatch with the example ID, the expected value, the actual value, and the specific field that failed. This structured log is your accuracy report. For high-risk agent workflows—such as those triggering financial transactions or modifying production infrastructure—add a human approval gate that blocks the release if any critical tool call examples fail.

Integrate this harness into your CI system as a script that exits with a non-zero code if the accuracy falls below a configurable threshold (e.g., 100% for critical tools, 95% for non-critical). Use environment variables to control the model endpoint, API key, and the path to the example set file. Store the example set in a version-controlled JSON file where each entry has an id, input, expected_tool_name, and expected_arguments schema. The harness should also measure and log latency per example to detect regressions in tool-call speed. Avoid running this test against production models without a dry-run mode first; a misconfigured test could accidentally invoke real tools if your agent harness is not properly sandboxed. Always run tool-call accuracy tests in an isolated environment where tool execution is mocked or disabled, validating only the model's intent to call the tool, not the tool's side effects.