Tool Misuse Intent Detection Prompt

This prompt is designed for agent platform engineers who need a reliable, pre-execution safety gate. Its job is to classify whether a user's request intends to misuse the tools available to an agent, focusing on three high-risk categories: data exfiltration (using legitimate tools to read and leak sensitive data), unauthorized actions (performing destructive or privileged operations), and policy circumvention (chaining tool calls to bypass safety rules). The ideal user is an engineer integrating this classifier into an agent's request path, after user input is received and before the agent's planning or tool-calling step begins. It is not a general-purpose harm classifier, not a replacement for tool-level authorization or identity access management, and not designed for post-hoc audit log analysis.

Deploy this prompt when the cost of a single misused tool call is high—for example, an agent with access to a production database, a customer data API, or a deployment pipeline. The prompt expects a structured input containing the user's request and a description of the available tools, including their capabilities and risk profiles. It returns a classification label (EXFILTRATION, UNAUTHORIZED_ACTION, POLICY_CIRCUMVENTION, or BENIGN) along with a confidence score and a concise evidence statement. This output should be used as a gating decision: if the risk level exceeds your configured threshold, the agent must refuse the request and log the incident before any tool is called. Do not use this prompt for content moderation of agent responses or for classifying harm in static text that has no tool-use context.

Before integrating this prompt, ensure you have mapped out your agent's tool surface and defined clear risk levels for each tool. A tool that reads public documentation has a different risk profile than one that queries customer PII. The prompt's effectiveness depends on the quality of the [TOOLS] description you provide—vague tool descriptions lead to poor classifications. After deployment, monitor the false-positive rate closely; over-refusal on benign multi-step requests is a common failure mode. Pair this prompt with a human-review queue for low-confidence classifications and a session-level risk tracker to detect multi-turn probing attacks that a single-turn classifier will miss.

The prompt below is designed to sit between a user's request and the execution layer of an agentic system. Its sole job is to classify whether the request intends to misuse available tools for data exfiltration, unauthorized actions, or policy circumvention. It is not a conversational response prompt; it is a structured classification gate. Copy this template, replace the bracketed placeholders with your specific tool definitions and policies, and integrate it as a pre-execution check. The output is strict JSON, making it trivial to parse in application code for a blocking or flagging decision.

textCopy
System: You are a security classification engine. Your only output is valid JSON. Do not wrap the JSON in markdown or add commentary.

Classify the user's request for tool misuse intent. Analyze the request against the available tools, their intended purposes, and the organization's security policy.

Available Tools and Intended Use:
[TOOL_DEFINITIONS]

Security Policy:
[SECURITY_POLICY]

User Request:
[USER_REQUEST]

Output a JSON object with the following schema:
{
  "classification": "BENIGN" | "MISUSE" | "AMBIGUOUS",
  "confidence_score": 0.0-1.0,
  "misuse_category": ["DATA_EXFILTRATION", "UNAUTHORIZED_ACTION", "POLICY_CIRCUMVENTION", null],
  "rationale": "A brief, specific explanation of the classification, referencing the policy or tool definitions.",
  "requires_human_review": true | false
}

Classification Guidelines:
- BENIGN: The request aligns with the intended use of the tools and does not violate the security policy.
- MISUSE: The request clearly intends to use tools for a prohibited purpose.
- AMBIGUOUS: The request is unclear, could be interpreted in multiple ways, or requires more context to classify safely. Always set requires_human_review to true for AMBIGUOUS classifications.
- Set requires_human_review to true for any MISUSE classification with a confidence_score below 0.9.

To adapt this template, you must provide concrete definitions for [TOOL_DEFINITIONS] and [SECURITY_POLICY]. The [TOOL_DEFINITIONS] should be a structured list describing each tool's name, its legitimate purpose, and its data access scope (e.g., read_only, write_user_data, admin). The [SECURITY_POLICY] should be a plain-language summary of the rules, such as 'Do not aggregate user data for export' or 'Do not use admin tools to modify another user's account.' The precision of these definitions directly determines the classifier's accuracy. For high-risk deployments, you should build an evaluation harness that tests this prompt against a golden dataset of known benign and malicious tool-use requests, measuring precision and recall for the MISUSE and AMBIGUOUS classes before production rollout.

After integrating this prompt, your application code should parse the JSON output and act on it. A BENIGN classification with a high confidence score can proceed automatically. An AMBIGUOUS or MISUSE classification should halt execution and, based on the requires_human_review flag, either queue the interaction for a human analyst or return a canned refusal message to the user. Do not pass the raw classification rationale back to the user, as it may reveal policy details useful to an attacker. Log the full classification payload for audit and monitoring purposes.

This prompt is designed to sit as a pre-execution gate in an agent loop, not as a post-hoc audit tool. Before the agent invokes any tool—especially those with write, delete, or data-access capabilities—the user's request and the proposed tool call should be passed through this classifier. The output is a structured decision (ALLOW, BLOCK, FLAG) with a confidence score and a specific misuse category. The implementation harness must enforce that a BLOCK decision prevents tool execution and returns a safe refusal to the user, while a FLAG decision queues the interaction for human review but may optionally allow a read-only or sandboxed version of the tool to proceed depending on your risk tolerance.

Wire the prompt into your agent's tool-selection middleware. After the planner selects a tool and generates arguments, intercept the (user_request, tool_name, tool_arguments, tool_description, session_history_summary) tuple and format it into the prompt's [USER_REQUEST], [TOOL_NAME], [TOOL_ARGUMENTS], [TOOL_DESCRIPTION], and [SESSION_CONTEXT] placeholders. The model should return a strict JSON object with fields decision, confidence, misuse_category, and rationale. Implement a post-processing validator that checks: (1) decision is one of the allowed enum values, (2) confidence is a float between 0.0 and 1.0, and (3) misuse_category is not null when decision is BLOCK or FLAG. If validation fails, retry once with the validation error injected into the [CONSTRAINTS] field. If the retry also fails, default to BLOCK with maximum confidence and log the failure as a critical safety-system error.

For high-risk deployments, never rely solely on this prompt's output to block tool execution without additional safeguards. Implement a secondary, deterministic rule engine that checks for known-dangerous argument patterns (e.g., rm -rf, SQL DROP, API calls to external domains not on an allowlist) as a hard stop. Log every classification decision—including the full prompt, raw model output, validated output, and final gating action—to an immutable audit store. For FLAG decisions, route to a human review queue with a strict SLA; if the SLA expires, treat the interaction as BLOCK. When choosing a model, prefer a fast, instruction-tuned model (e.g., GPT-4o-mini, Claude 3.5 Haiku) to keep latency under 500ms, as this gate sits on the critical path of every tool call. Run a weekly regression suite of known tool-abuse patterns and benign tool-use edge cases to detect drift in classification behavior.