Hallucination Detection in Tool-Use Arguments Prompt Template

This prompt is designed for a single, high-stakes job: acting as a pre-execution guardrail that inspects the arguments of a planned tool call and verifies they are grounded in the conversation context or retrieved evidence. The ideal user is an agent or function-calling system builder who needs to prevent cascading errors where a model fabricates an entity ID, invents a parameter value, or assumes a capability that doesn't exist. This is not a general factuality check for conversational text; it is a targeted validation step that sits between the model's tool selection and the actual API, database, or function execution.

You should use this prompt when the cost of a bad tool call is high—writing to a database with a hallucinated primary key, sending an email to a fabricated address, or calling an API with unsupported parameters. The required context includes the full conversation history, the proposed tool name and its schema, and the complete set of arguments the model generated. The prompt works best when it has access to the same evidence the model used, such as retrieved documents or prior tool outputs. It is not a replacement for schema validation or type-checking in your application code; those checks should still run. Instead, this prompt catches semantic hallucinations that pass structural validation but are factually wrong.

Do not use this prompt for low-risk, read-only operations where a hallucinated argument would only cause a harmless error message. It is also not suitable for real-time streaming tool-use scenarios where the added latency of a full verification step would break the user experience. For those cases, consider a lightweight, rules-based check or an asynchronous audit. After running this prompt, the output should be a structured verdict that your harness can act on: block the call, flag for human review, or allow execution. The next step is to wire this verdict into your tool-execution middleware so that no hallucinated argument ever reaches an external system.

This prompt template is designed to sit between your agent's tool-call generation step and the actual execution of that call. It acts as a guardrail, receiving the proposed tool name and arguments alongside the conversation history and any retrieved evidence. The model's job is to flag parameters that are fabricated, unsupported, or contradictory to the provided context. Use this when the cost of executing a hallucinated tool call—such as writing to a database, sending an email, or making a purchase—is higher than the latency cost of a verification step. Do not use this for low-risk, read-only tool calls where fabrication would be immediately obvious and harmless.

textCopy
You are a tool-call argument auditor. Your task is to verify that every argument in a proposed tool call is strictly grounded in the provided conversation context and retrieved evidence. You must flag any hallucinated, fabricated, or unsupported parameters before the tool executes.

## INPUT

### Proposed Tool Call
- Tool Name: [TOOL_NAME]
- Arguments: [TOOL_ARGUMENTS_JSON]

### Conversation Context
[CONVERSATION_HISTORY]

### Retrieved Evidence (if available)
[RETRIEVED_EVIDENCE]

## CONSTRAINTS
- Every argument value must be traceable to a specific span in the Conversation Context or Retrieved Evidence.
- Do not infer, assume, or generate values that are not explicitly present in the provided context.
- Treat entity IDs, database keys, email addresses, and numerical quantities with the highest scrutiny.
- If an argument is a user-provided input repeated verbatim, it is considered grounded.
- If an argument is derived from a tool's previous output present in the conversation, it is grounded only if the exact value appears.

## OUTPUT SCHEMA
Return a valid JSON object with the following structure:
{
  "tool_name": "string",
  "overall_verdict": "PASS" | "FLAG" | "UNCERTAIN",
  "arguments_audit": [
    {
      "parameter": "string",
      "provided_value": "string | number | boolean | null",
      "verdict": "GROUNDED" | "HALLUCINATED" | "UNCERTAIN",
      "evidence_span": "string or null",
      "explanation": "string"
    }
  ],
  "hallucinated_parameters": ["string"],
  "recommendation": "PROCEED" | "BLOCK" | "REQUEST_CLARIFICATION"
}

## EXAMPLES

### Example 1: Grounded Call
Proposed Tool Call:
- Tool Name: send_email
- Arguments: {"to": "alice@example.com", "subject": "Q3 Report"}

Conversation Context:
User: "Please send the Q3 report to alice@example.com"

Output:
{
  "tool_name": "send_email",
  "overall_verdict": "PASS",
  "arguments_audit": [
    {"parameter": "to", "provided_value": "alice@example.com", "verdict": "GROUNDED", "evidence_span": "alice@example.com", "explanation": "Email explicitly provided by user."},
    {"parameter": "subject", "provided_value": "Q3 Report", "verdict": "GROUNDED", "evidence_span": "Q3 report", "explanation": "Subject derived directly from user request."}
  ],
  "hallucinated_parameters": [],
  "recommendation": "PROCEED"
}

### Example 2: Hallucinated ID
Proposed Tool Call:
- Tool Name: update_order_status
- Arguments: {"order_id": "ORD-99887", "status": "shipped"}

Conversation Context:
User: "Please ship my recent order."
Assistant: "I found your order #ORD-12345 for a widget. Shall I mark it as shipped?"
User: "Yes, please."

Output:
{
  "tool_name": "update_order_status",
  "overall_verdict": "FLAG",
  "arguments_audit": [
    {"parameter": "order_id", "provided_value": "ORD-99887", "verdict": "HALLUCINATED", "evidence_span": null, "explanation": "The order ID ORD-99887 does not appear in the conversation. The only order ID present is ORD-12345."},
    {"parameter": "status", "provided_value": "shipped", "verdict": "GROUNDED", "evidence_span": "ship my recent order", "explanation": "Status derived from user intent."}
  ],
  "hallucinated_parameters": ["order_id"],
  "recommendation": "BLOCK"
}

## INSTRUCTIONS
1. Parse the proposed tool call and extract every argument.
2. For each argument, search the Conversation Context and Retrieved Evidence for the exact value or a direct, unambiguous derivation.
3. If a value is not found, mark it HALLUCINATED.
4. If the evidence is ambiguous or the value could be a reasonable inference, mark it UNCERTAIN and recommend REQUEST_CLARIFICATION.
5. Return the audit JSON. Do not include any text outside the JSON object.

To adapt this template for your own agent, replace the placeholder variables with your actual runtime data. The [TOOL_ARGUMENTS_JSON] should be the serialized arguments object your model generated, not a prettified version. For [CONVERSATION_HISTORY], include the full turn history with role labels so the auditor can distinguish user-provided values from assistant-generated ones. The [RETRIEVED_EVIDENCE] field is optional but critical for RAG-augmented agents—if your agent retrieved documents before selecting a tool, include them here. If your agent uses multiple tools in sequence, include the outputs of previously executed tools in the conversation context so the auditor can trace derived values. For high-risk domains like finance or healthcare, consider adding a [RISK_LEVEL] parameter that adjusts the strictness of the grounding requirement: at HIGH risk, even UNCERTAIN parameters should trigger a BLOCK recommendation and require human review before execution.

This prompt operates as a pre-execution guard in agent and function-calling systems. Its job is to inspect tool-call arguments generated by a model and flag any parameter values that are hallucinated—fabricated IDs, unsupported enum values, invented entity references, or values that cannot be traced to the conversation context or retrieved evidence. The harness must intercept the model's tool call before the tool is invoked, route the arguments through this detection prompt, and decide whether to block, repair, or allow execution based on the structured hallucination report.

Integration pattern: Place this prompt inside a middleware layer that sits between the model output parser and the tool execution engine. When the model emits a function call, extract the tool_name and arguments payload. Construct the prompt by populating [TOOL_SCHEMA] with the JSON Schema definition of the expected parameters, [CONVERSATION_CONTEXT] with the last N turns of the dialogue or task state, and [RETRIEVED_EVIDENCE] with any documents, database records, or API responses the model was supposed to ground its arguments in. The [TOOL_CALL_ARGUMENTS] placeholder receives the raw arguments the model generated. The model's response should conform to a strict JSON schema containing a hallucination_flags array, each entry specifying the parameter_path, claimed_value, flag_type (e.g., fabricated_id, unsupported_enum, no_evidence), severity, and explanation. Parse this output and apply a decision rule: if any flag has severity critical or high, block execution and either request re-generation with the flag report as feedback, escalate to a human reviewer, or fall back to a safe default. For medium or low severity flags, log the incident and proceed with execution if the risk tolerance for the domain permits.

Validation and retry logic: After receiving the hallucination report, validate that the output is parseable JSON matching the expected schema. If parsing fails, retry the prompt once with a stricter format instruction appended. If the retry also fails, treat it as a blocking event—do not execute the tool call with unvalidated arguments. For high-risk domains such as healthcare, legal, or financial tool execution, always require human review when any flag is raised, regardless of severity score. Logging: Capture the full prompt, the model's hallucination report, the tool name, the original arguments, and the final execution decision (blocked, repaired, allowed) in an observability store. This trace is essential for debugging agent loops, identifying prompt drift, and demonstrating audit readiness. Model choice: Use a model with strong instruction-following and structured output capabilities. Avoid models known to hallucinate in structured extraction tasks. If latency is critical, consider a smaller fine-tuned model specifically trained on argument-grounding verification for your tool schemas.

What to avoid: Do not use this prompt as a post-execution audit—by then, the hallucinated tool call has already produced side effects (database writes, API mutations, user-visible actions). Do not skip validation when the tool call is read-only; hallucinated read arguments can still return misleading results that cascade into downstream agent decisions. Do not rely solely on this prompt without eval calibration: run it against a labeled dataset of known hallucinated and grounded tool-call arguments to measure precision and recall before production deployment. Finally, ensure the [TOOL_SCHEMA] you provide includes enum constraints, required fields, and type definitions—the prompt's detection accuracy depends on having a complete schema to compare against.