Tool Selection Precision and Recall Evaluation Prompt

This prompt is designed for agent platform teams and evaluation engineers who need to measure how accurately a model selects tools from a defined catalog. The job-to-be-done is regression testing: you have a golden dataset of inputs paired with correct tool choices, and you need to compute precision, recall, and confusion matrices across tool categories. Use this when your tool catalog has changed, your system prompt has been updated, or you're comparing model versions and need quantitative evidence that tool selection behavior hasn't degraded. The prompt expects a batch of test cases, each containing a user input, the available tool definitions, and the ground-truth tool name that should have been selected.

Do not use this prompt when you need to evaluate argument accuracy, call sequencing, or whether the model should have abstained. Those are separate evaluation surfaces covered by sibling prompts in this pillar. This prompt also assumes you already have a golden dataset with authoritative labels—if you're still building that dataset or need human annotators to establish ground truth, start there first. The prompt produces aggregate metrics, not per-case debugging traces; if you need to diagnose why a specific tool selection failed, pair this with the Tool Selection Justification Trace Evaluation Prompt or the Tool Use Error Categorization prompt. For high-risk domains where tool misuse could cause harm, always route low-precision tool categories to human review before accepting the metrics as a release gate.

The ideal reader is comfortable running batch evaluation pipelines, understands precision/recall trade-offs, and can interpret confusion matrices. You'll need a test harness that iterates over your golden dataset, sends each case through the model under test, collects the model's tool choice, and then feeds both the predictions and ground truth into this evaluation prompt. The prompt itself acts as the judge—it does not call tools. Wire it into your CI/CD pipeline so that every prompt or model change triggers a regression run. If your tool catalog is large or changes frequently, consider stratifying your test cases by tool category to catch category-specific drift before it affects aggregate numbers.

This prompt template evaluates an agent's tool selection decisions against a golden dataset of correct tool choices. It is designed to be called once per test case, receiving the agent's selected tool, the ground-truth correct tool, the full tool catalog available at decision time, and the user context that prompted the selection. The output is a structured JSON object containing per-tool precision, recall, and F1 scores, an aggregate confusion matrix, and a list of misclassifications with diagnostic notes. Use this prompt when you need to regression-test tool selection behavior after changing the tool catalog, system prompt, or model version.

textCopy
You are an evaluation harness for tool selection accuracy. Your job is to compare an agent's chosen tool against the ground-truth correct tool for a given user request and produce structured metrics.

## INPUTS
- Agent Selected Tool: [AGENT_SELECTED_TOOL]
- Ground Truth Correct Tool: [GROUND_TRUTH_TOOL]
- Available Tool Catalog (at decision time): [TOOL_CATALOG]
- User Request Context: [USER_CONTEXT]
- Previous Test Results (for aggregate computation): [PREVIOUS_RESULTS]

## OUTPUT SCHEMA
Return a single JSON object with this exact structure:
{
  "per_tool_metrics": [
    {
      "tool_name": "string",
      "true_positives": number,
      "false_positives": number,
      "false_negatives": number,
      "precision": number,
      "recall": number,
      "f1_score": number
    }
  ],
  "aggregate_confusion_matrix": {
    "labels": ["string"],
    "matrix": [[number]]
  },
  "misclassifications": [
    {
      "user_context": "string",
      "selected_tool": "string",
      "ground_truth_tool": "string",
      "error_type": "wrong_tool | hallucinated_tool | missed_call | over_call",
      "diagnostic_note": "string"
    }
  ],
  "aggregate_precision": number,
  "aggregate_recall": number,
  "aggregate_f1": number,
  "total_test_cases": number
}

## CONSTRAINTS
- If [AGENT_SELECTED_TOOL] is "NONE" or null, treat it as the agent choosing not to call any tool.
- If [GROUND_TRUTH_TOOL] is "NONE" or null, the correct decision was to abstain from calling a tool.
- A hallucinated tool is any tool name in [AGENT_SELECTED_TOOL] that does not appear in [TOOL_CATALOG].
- Update [PREVIOUS_RESULTS] incrementally: add this test case's counts to the running totals before computing metrics.
- If [PREVIOUS_RESULTS] is empty or null, initialize all counts to zero.
- Precision = true_positives / (true_positives + false_positives). If denominator is zero, precision is 1.0.
- Recall = true_positives / (true_positives + false_negatives). If denominator is zero, recall is 1.0.
- F1 = 2 * (precision * recall) / (precision + recall). If both are zero, F1 is 0.0.
- Do not fabricate tool names or metrics. Derive everything from the provided inputs.

## RISK LEVEL
[RISK_LEVEL]

## EXAMPLES
[EXAMPLES]

Adapt this template by replacing the square-bracket placeholders with values from your evaluation harness. For batch evaluation, call this prompt once per test case and accumulate results in [PREVIOUS_RESULTS]. The [TOOL_CATALOG] should be the exact tool definitions available at decision time—include tool names, descriptions, and parameter schemas so the evaluator can detect hallucinated tools. Set [RISK_LEVEL] to "HIGH" if tool misuse could cause data loss, financial impact, or safety incidents; this should trigger additional human review of misclassifications before accepting aggregate metrics. Provide at least three [EXAMPLES] covering correct selection, incorrect selection, and correct abstention to calibrate the evaluator's judgment.

This prompt is designed to operate as a batch evaluation step, not a real-time guard. Feed it a golden dataset of test cases where each record contains the user input, the available tool catalog at that moment, and the expected correct tool name. The prompt returns a structured JSON object with per-case predictions, confusion matrix entries, and aggregate precision/recall metrics. Wire this into a CI/CD pipeline that runs on every prompt or tool catalog change, comparing the new metrics against a stored baseline to detect regressions before deployment.

Construct the input payload by assembling the [TOOL_CATALOG] as a JSON array of tool definitions (name, description, parameters), the [TEST_CASES] as an array of objects with input, expected_tool, and optional context, and the [OUTPUT_SCHEMA] as the exact JSON structure you expect back. Use a model with strong JSON mode and low temperature (0.0–0.1) for deterministic scoring. Implement a retry wrapper: if the response fails to parse as valid JSON or is missing required fields (aggregate_metrics, per_case_results), retry up to 2 times with an error message appended to the prompt. After 3 total attempts, fail the evaluation run and alert the pipeline owner—do not silently pass a broken eval.

Post-process the output by extracting aggregate_metrics.precision, aggregate_metrics.recall, and aggregate_metrics.f1 for each tool. Store these alongside the confusion_matrix and per_case_results in your evaluation database. Compare against the previous run's metrics using a threshold (e.g., precision drop > 0.02 triggers a warning, > 0.05 blocks deployment). For high-stakes tool-use workflows where a wrong tool call could trigger destructive actions, add a human review gate on any case where predicted_tool differs from expected_tool and the expected_tool is classified as high-risk in your tool catalog metadata. Log every evaluation run with the prompt version, model version, tool catalog hash, and test suite hash for full traceability.