Tool Selection Justification Trace Evaluation Prompt

This prompt is designed for AI platform engineers and evaluation leads who need to programmatically assess the quality of reasoning traces generated by tool-using agents. The core job-to-be-done is debugging tool selection failures at scale: when an agent picks the wrong tool, you need to know whether its reasoning was incomplete, irrelevant to the task, or misaligned with the actual capabilities of the tools in its catalog. The ideal user is someone running an evaluation harness against agent traces, typically after a test suite or production log review has surfaced unexpected tool choices. They have access to the agent's full reasoning trace, the tool catalog that was available at decision time, and the user's original task context.

You should use this prompt when you need structured, repeatable scores for justification quality across hundreds or thousands of traces. It is particularly valuable during model migration, prompt refactoring, or tool catalog changes—any scenario where previously reliable tool-selection behavior might silently degrade. The prompt produces three scores (completeness, relevance, and capability alignment) plus a consolidated finding, making it suitable for automated regression gates. Do not use this prompt for real-time agent decision-making; it is an offline evaluation tool. Do not use it to grade the correctness of the tool choice itself—that requires a separate Tool Selection Correctness Grading Prompt. This prompt evaluates the explanation, not the outcome.

Before using this prompt, ensure you have the full reasoning trace available, not just the final tool call. The trace must include the agent's stated rationale for selecting the tool. If your agent does not produce explicit reasoning traces, this prompt cannot function—you will need to instrument your agent to emit chain-of-thought before tool calls. Also verify that your tool catalog descriptions accurately reflect what each tool actually does; the prompt judges alignment against the provided descriptions, so stale or misleading catalog entries will produce misleading evaluation scores. For high-stakes debugging where a single misclassification could block a release, pair this automated evaluation with human spot-checking of a sample of traces to calibrate the judge's behavior against your team's standards.

This section provides the core prompt template for grading tool selection justification traces. The prompt is designed to evaluate whether an agent's reasoning for choosing a specific tool is complete, relevant to the task, and aligned with the actual capabilities of the selected tool. Use this template as the foundation for automated evaluation pipelines, debugging tool selection failures, or comparing reasoning quality across model versions. The template uses square-bracket placeholders that you must replace with your specific tool catalog, task context, and evaluation criteria before use.

textCopy
You are an expert evaluator grading the quality of reasoning traces that justify tool selection decisions. Your job is to assess whether the provided justification correctly explains why a specific tool was chosen for a given task.

## INPUTS

**Task Description:**
[TASK_DESCRIPTION]

**Available Tools (with capabilities):**
[TOOL_CATALOG]

**Selected Tool:**
[SELECTED_TOOL_NAME]

**Justification Trace:**
[JUSTIFICATION_TRACE]

**Actual Tool Capabilities (ground truth):**
[TOOL_CAPABILITIES_REFERENCE]

## EVALUATION CRITERIA

Evaluate the justification trace across three dimensions, each scored 1-5:

1. **Completeness (1-5):** Does the justification address why this tool was chosen over alternatives? Does it mention relevant capabilities and constraints? Are there gaps in reasoning?
   - 1: No meaningful justification provided
   - 3: Mentions some relevant factors but misses key alternatives or constraints
   - 5: Thoroughly addresses tool selection rationale, compares alternatives, and acknowledges limitations

2. **Relevance to Task (1-5):** Does the justification connect tool capabilities to specific task requirements? Is the reasoning task-grounded rather than generic?
   - 1: Justification is unrelated to the actual task
   - 3: Makes some task-relevant points but includes irrelevant or generic statements
   - 5: Every point directly connects tool capabilities to specific task requirements

3. **Alignment with Actual Capabilities (1-5):** Does the justification accurately describe what the selected tool can do? Are there hallucinated capabilities or omissions?
   - 1: Claims capabilities the tool does not have or fundamentally misunderstands the tool
   - 3: Mostly accurate but contains minor errors or omissions about tool capabilities
   - 5: Perfectly accurate representation of tool capabilities with no hallucinations or omissions

## OUTPUT FORMAT

Return a JSON object with this exact structure:
{
  "completeness_score": <integer 1-5>,
  "relevance_score": <integer 1-5>,
  "alignment_score": <integer 1-5>,
  "aggregate_score": <float 1.0-5.0>,
  "justification_summary": "<2-3 sentence summary of the overall quality>",
  "strengths": ["<specific strength 1>", "<specific strength 2>"],
  "weaknesses": ["<specific weakness 1>", "<specific weakness 2>"],
  "hallucinated_capabilities": ["<capability claimed but not present>"],
  "missing_capabilities": ["<relevant capability not mentioned>"],
  "improvement_suggestions": "<actionable suggestion for better justification>"
}

## CONSTRAINTS

- Base scores strictly on the justification trace content, not on whether the tool selection itself was correct
- If the justification mentions capabilities not in the reference, flag them as hallucinated
- If the justification omits clearly relevant capabilities from the reference, flag them as missing
- The aggregate score is the mean of the three dimension scores
- If the justification trace is empty or missing, score all dimensions as 1

Adaptation guidance: Replace [TASK_DESCRIPTION] with the specific task the agent was trying to accomplish. Populate [TOOL_CATALOG] with your full tool list including descriptions—this provides context for what alternatives were available. [SELECTED_TOOL_NAME] should be the tool the agent actually chose, and [JUSTIFICATION_TRACE] is the raw reasoning text you want to evaluate. The [TOOL_CAPABILITIES_REFERENCE] is critical: provide the authoritative description of what the selected tool actually does, drawn from your tool documentation or schema. This ground truth enables hallucination detection. For batch evaluation, wrap this prompt in a harness that iterates over test cases and collects structured outputs for analysis.

Next steps: After adapting the template, run it against a sample of 20-50 justification traces and manually review the scores for calibration. Pay special attention to the hallucinated_capabilities field—false positives here indicate the evaluator is too strict, while false negatives mean it's missing fabricated claims. Adjust the scoring rubric descriptions if needed before scaling to full evaluation pipelines. For production use, pair this prompt with a JSON schema validator to catch malformed outputs before they enter your metrics pipeline.

This prompt is designed to operate as a scoring step inside an automated evaluation harness, not as a one-off manual review tool. The typical integration pattern is: a test suite executes an agent against a set of scenarios, the agent produces tool calls and reasoning traces, and this prompt grades the quality of the justification trace for each tool selection decision. The harness should feed the prompt a structured input containing the task context, the available tool catalog with descriptions, the tool that was selected, and the reasoning trace the agent produced. The output is a structured score object that downstream systems can log, aggregate, and alert on.

Validation and retry logic is critical because the prompt produces structured JSON output. Implement a JSON schema validator that checks for required fields (justification_completeness_score, task_relevance_score, capability_alignment_score, overall_score, failure_flags) and enforces score ranges (e.g., 1–5 for individual dimensions). If validation fails, retry up to two times with the same input and an added instruction to strictly follow the output schema. After three failures, log the raw response and escalate for manual review. For high-stakes evaluation pipelines (e.g., pre-deployment gates), route all overall_score values below 3 to a human review queue with the original trace and the model's critique attached.

Model choice and temperature settings matter for judge consistency. Use a model with strong instruction-following and structured output capabilities (e.g., GPT-4o, Claude 3.5 Sonnet) at temperature 0 or very low temperature (0.0–0.1) to maximize scoring reproducibility. Avoid using the same model family as the agent under test for evaluation, as self-preference bias can inflate scores. Log the judge model version alongside every score so that score drift from judge model updates can be detected. Store evaluation results in a structured format (JSON columns in a database or Parquet files) with fields for test_case_id, agent_model_version, judge_model_version, timestamp, and all score dimensions. This enables longitudinal tracking of justification quality across agent iterations.

Integration with CI/CD and alerting turns this prompt from a diagnostic tool into a production gate. Wire the evaluation harness into your agent's pull request pipeline: run the full test suite, compute aggregate metrics (mean overall score, distribution of failure flags), and fail the build if the mean score drops below a defined threshold or if any critical failure flag (e.g., hallucinated_capability) appears in more than 1% of test cases. For ongoing monitoring, run a subset of evaluation scenarios daily against production agent traces and trigger an alert if justification quality degrades. The failure flags field is particularly actionable—use it to route specific failure categories (e.g., missing_capability_reference, circular_reasoning) to the engineering team responsible for the agent's tool-use prompts.