Rubric for Tool Selection Correctness Prompt Template

This prompt is for agent platform teams who need to evaluate tool-use decisions at scale without relying solely on manual spot checks. Use it when you have an agent that selects and calls external tools—APIs, databases, MCP servers, browser actions—and you need a repeatable, model-graded rubric to score whether those tool choices were correct. The primary job-to-be-done is turning a subjective review of an agent's tool-call trace into a structured, automatable evaluation that can run in CI/CD, pre-release gates, or production monitoring pipelines. The ideal user is an AI engineer, evaluation lead, or platform developer responsible for agent reliability who already has access to the agent's tool call trace, the available tool definitions, and the user's original task context.

The prompt produces a structured evaluation with dimension-level scores, misuse severity classification, and downstream impact assessment. It assumes you have captured the complete tool-call sequence, including the tool names, arguments, timestamps, and any error responses. You must also provide the full tool schema or function definitions that were available to the agent at decision time, because the rubric judges correctness against what the agent could have known, not against tools added later. The user's original task or query is required as ground truth for intent. Without these three inputs—trace, tool definitions, and task context—the rubric cannot produce reliable scores. For high-risk domains such as healthcare, finance, or infrastructure control, the rubric output should feed into a human review queue rather than serving as the sole gate for deployment.

Do not use this prompt for evaluating final answer quality, conversation tone, or non-tool-related agent behavior. It is specifically designed for tool selection correctness, argument accuracy, call sequencing, and abstention decisions. If you need to evaluate whether the agent's final response was helpful, use a separate answer-quality rubric. If you need to evaluate multi-turn conversation coherence, use a conversation evaluation prompt. This rubric also does not assess the performance or correctness of the tools themselves—a perfectly selected tool that returns incorrect data is a tool failure, not a selection failure. The next step after reading this section is to gather your tool-call traces and tool definitions, then proceed to the prompt template to begin scoring.

The following prompt template is designed to be copied directly into your evaluation harness. It instructs an LLM judge to assess an agent's tool-use decisions against a structured rubric, producing a consistent, numeric score and a detailed justification. The template uses square-bracket placeholders for all dynamic inputs, ensuring you can adapt it to different agent architectures, tool libraries, and risk profiles without modifying the core evaluation logic.

codeCopy
You are an expert evaluator grading an AI agent's tool-use decisions. Your task is to assess the agent's tool selection, argument construction, and call sequencing against the provided context and expected behavior.

# EVALUATION CONTEXT
[AGENT_TASK_DESCRIPTION]

# AVAILABLE TOOLS AND SCHEMAS
[TOOL_DEFINITIONS]

# AGENT'S TOOL CALL SEQUENCE
[AGENT_TOOL_CALLS]

# EXPECTED TOOL CALL SEQUENCE (GROUND TRUTH)
[EXPECTED_TOOL_CALLS]

# EVALUATION RUBRIC
Score the agent's tool-use on a 1-5 scale for each dimension below. Provide a specific justification for each score, citing the agent's actual calls and arguments.

## Dimension 1: Tool Selection Correctness (1-5)
- 5: All tool choices are optimal. The agent selected the most appropriate tool for every step, avoiding unnecessary or redundant calls.
- 4: Tool choices are correct, with minor suboptimal selections that do not affect the outcome.
- 3: One clearly incorrect tool choice, or multiple minor suboptimal selections that could lead to inefficiency.
- 2: Multiple incorrect tool choices that would prevent task completion or produce incorrect results.
- 1: The agent selected tools that are fundamentally inappropriate for the task, or failed to call required tools.

## Dimension 2: Argument Accuracy (1-5)
- 5: All arguments are syntactically valid, correctly typed, and semantically appropriate for the task context.
- 4: Arguments are correct with minor issues (e.g., suboptimal parameter values) that do not cause failure.
- 3: One argument error that would cause a tool call to fail or return incorrect data.
- 2: Multiple argument errors across different tool calls.
- 1: Arguments are missing, malformed, or nonsensical for the majority of calls.

## Dimension 3: Call Sequencing and Dependency Handling (1-5)
- 5: The sequence of calls respects all data dependencies. Outputs from prior calls are correctly used as inputs to subsequent calls.
- 4: Correct sequencing with minor inefficiencies (e.g., a call that could have been parallelized).
- 3: One sequencing error where a call is made before its required input is available.
- 2: Multiple sequencing errors that would break the workflow.
- 1: The call sequence is random or shows no understanding of task dependencies.

## Dimension 4: Abstention Correctness (1-5)
- 5: The agent correctly abstained from calling tools when clarification was needed, the task was impossible, or the required tool was unavailable. It did not hallucinate tool calls.
- 4: Correct abstention with a minor issue, such as slightly verbose clarification request.
- 3: Failed to abstain when it should have, making one inappropriate call, or abstained unnecessarily for a straightforward step.
- 2: Made multiple inappropriate calls when abstention was required, or abstained on critical steps.
- 1: Hallucinated tool calls for non-existent tools or fabricated arguments when abstention was the only correct action.

# MISUSE SEVERITY CLASSIFICATION
Classify the most severe tool misuse observed:
- CRITICAL: Tool misuse that would cause data loss, security violation, or irreversible state change.
- HIGH: Tool misuse that would produce incorrect results or break the primary workflow.
- MEDIUM: Tool misuse that causes inefficiency, redundant work, or requires a retry.
- LOW: Cosmetic or stylistic issues with no functional impact.
- NONE: No misuse detected.

# DOWNSTREAM IMPACT ASSESSMENT
Describe the likely impact of the observed tool-use errors on the downstream system or user, if the calls were executed as-is.

# OUTPUT FORMAT
Return a valid JSON object with the following schema:
{
  "tool_selection_score": <int 1-5>,
  "argument_accuracy_score": <int 1-5>,
  "call_sequencing_score": <int 1-5>,
  "abstention_score": <int 1-5>,
  "overall_score": <float, average of four dimension scores>,
  "misuse_severity": "<CRITICAL|HIGH|MEDIUM|LOW|NONE>",
  "downstream_impact": "<string description>",
  "justification": "<string with per-dimension reasoning>"
}

To adapt this template, replace the placeholders with your specific evaluation context. [AGENT_TASK_DESCRIPTION] should contain the user's original request and any system-level instructions. [TOOL_DEFINITIONS] must include the full JSON schema or function signatures of every tool available to the agent. [AGENT_TOOL_CALLS] is the sequence of calls the agent actually made, and [EXPECTED_TOOL_CALLS] is your ground-truth reference for what should have happened. If you lack a ground truth, remove Dimension 1's reference to expected calls and rely on the judge's reasoning about tool appropriateness given the task. For high-stakes domains, always include a human review step before accepting the judge's severity classification, especially for CRITICAL or HIGH misuse findings.

This rubric prompt is designed to run inside an automated evaluation harness, not as a one-off manual review. The typical integration point is after an agent has produced a tool-use trace—a sequence of tool calls, arguments, and responses—and you need a structured, repeatable quality score before that trace is accepted for training data, regression testing, or agent performance dashboards. The harness should treat the rubric prompt as a stateless function: input the trace and the tool definitions, output a scored rubric JSON, and then validate that output before it touches any downstream system.

Wire the prompt into a pipeline that first extracts the agent's tool-use trace and the available tool schemas into the [TOOL_USE_TRACE] and [TOOL_DEFINITIONS] placeholders. After the LLM judge returns its JSON, run a strict schema validator against the expected output shape—check that overall_score is a number in the defined range, that each tool_call_evaluation entry has a non-null tool_name and score, and that misuse_severity_classification is one of the allowed enum values. If validation fails, retry once with the validation error appended to the prompt as additional context. If the second attempt also fails, log the raw output and route the trace to a human review queue rather than silently accepting a malformed score. For high-risk domains where tool misuse carries operational or compliance impact, always require human sign-off on any trace classified as CRITICAL severity before it enters a training dataset or triggers an automated rollback.

Choose a model for the judge that is strong at following structured output instructions and has a context window large enough to hold your longest tool-use traces plus the full rubric. For most teams, a frontier model with JSON mode enabled is the right starting point. Log every evaluation—input trace, raw judge output, validated score, and any retry attempts—so you can detect judge drift, calibration shifts, or rubric ambiguity over time. The next step after implementing this harness is to run a calibration pass: have human reviewers score a sample of traces using the same rubric, compare against the LLM judge scores, and adjust the rubric's anchor descriptions or severity definitions until inter-rater agreement reaches your target threshold.