Tool Selection Degradation

A sustained increase in 4xx client errors—particularly 400 Bad Request and 404 Not Found—indicates the agent is formulating malformed API calls or targeting non-existent endpoints. Unlike transient network issues, this pattern reflects a degradation in the agent's understanding of tool schemas.

• Key metric: Error rate by status code, segmented per tool
• Example: An agent repeatedly sending JSON payloads to an endpoint expecting XML, or calling a deprecated /v1/ path after a migration to /v2/
• Threshold: A >5% week-over-week increase in 4xx errors for a stable tool warrants investigation

When an agent's tool selection probability distribution flattens, it indicates growing uncertainty. A healthy agent assigns high probability to the correct tool; a degrading agent spreads probability across multiple candidates, often selecting suboptimal or irrelevant tools.

• Key metric: Shannon entropy of the tool selection softmax distribution per query
• Example: For a 'send email' task, the agent assigns 30% probability to email_api, 25% to slack_api, 20% to calendar_api, and 25% to file_system—indicating confusion about which tool serves the communication goal
• Diagnostic: Compare current entropy distributions against a golden baseline from the model's initial deployment

Degraded tool selection often manifests as increased end-to-end latency driven by internal retry mechanisms. The agent selects an incorrect tool, receives an error or unexpected response, and enters a corrective loop—consuming additional inference cycles and API calls.

• Key metric: P95 and P99 latency for tool-invoking agent steps
• Example: A task that historically completed in 1.2 seconds now takes 4.7 seconds, with telemetry revealing 3 sequential failed tool calls before a successful one
• Cost implication: Each retry loop multiplies token consumption and API invocation costs, directly impacting infrastructure budgets

A specific form of malformed invocation where the agent generates parameters that violate the target tool's JSON Schema or OpenAPI specification. This includes missing required fields, incorrect data types, or hallucinated parameter names.

• Key metric: Percentage of tool calls that fail schema validation at the gateway layer
• Example: Calling a search_database function with a query parameter when the schema requires sql_query, or passing a string where an integer limit is expected
• Root cause: Often indicates context window pollution where stale or conflicting tool descriptions have overwritten the correct schema in the agent's working memory

A measurable shift in the frequency distribution of tool usage across the agent's available toolkit. An agent may develop an over-reliance on a familiar tool while neglecting newer, more appropriate ones—or begin invoking tools irrelevant to its assigned domain.

• Key metric: KL divergence between current tool usage distribution and the expected baseline
• Example: A customer support agent increasingly routes queries to a refund_processor tool when a knowledge_base_search would suffice, indicating a planning shortcut that bypasses information retrieval
• Detection: Track per-tool invocation counts over rolling 24-hour windows and flag statistically significant deviations

The agent systematically confuses tools with overlapping functional signatures—tools that share similar names, descriptions, or parameter structures but serve distinct purposes. This indicates the agent's semantic understanding of tool boundaries is eroding.

• Key metric: Confusion matrix tracking which tools are selected versus which tools should have been selected (requires labeled evaluation data)
• Example: Confusing create_ticket (IT support) with create_case (CRM), or get_user_by_id (internal directory) with get_customer_by_id (billing system)
• Mitigation signal: A spike in this metric suggests tool descriptions need disambiguation or the agent's system prompt requires reinforcement of tool selection rules