Hallucination Detection in Agent Planning Steps Prompt Template

This prompt is designed for a specific, high-risk moment in an agent loop: the transition between planning and execution. After an agent has decomposed a task into sub-goals and selected a sequence of tool calls, but before it commits to executing them, this prompt acts as a structured verification gate. Its primary job is to detect and flag hallucinated capabilities, fabricated observations, and unsupported planning premises that the agent may have invented during its reasoning process. The ideal user is an AI safety engineer, an agent framework developer, or a platform engineer responsible for preventing cascading failures in multi-step autonomous workflows.

You should not use this prompt as a general-purpose hallucination detector for final user-facing text. It is specifically tuned for the structural artifacts of agent planning: action schemas, tool arguments, state variables, and dependency graphs. It is also not a replacement for proper tool definition and API contracts. If your agent is hallucinating tool names because the tool registry is poorly defined, fix the registry first. This prompt is most effective when inserted between a planner module and an executor module, receiving the original task context, the agent's proposed plan, and any tool outputs from prior steps as inputs. In high-risk domains such as healthcare or finance, the output of this verification step should be logged and, for critical actions, routed to a human reviewer before execution proceeds.

To integrate this prompt into your agent architecture, treat its output as a binary gate with a structured report. A pass result allows execution to continue. A fail result should halt the agent, log the flagged hallucinations, and either escalate to a human or trigger a replanning step with corrected context. Common failure modes include the verifier itself hallucinating when the plan is long and complex, so break large plans into smaller verification chunks. Also, ensure that the [TOOL_REGISTRY] and [OBSERVATION_LOG] placeholders are populated with the exact, unsummarized outputs from previous tool calls—summarization before verification often hides the very discrepancies this prompt is designed to catch.

This template is designed to sit between an agent's planning step and its execution step. It takes the agent's proposed plan, the original task context, and any tool outputs the agent claims to have observed, then systematically checks whether each planned action, sub-goal, and state assumption is grounded in actual evidence. The prompt flags three categories of hallucination: fabricated capabilities (the agent assumes it has tools or permissions it doesn't), fabricated observations (the agent references tool outputs or state that never occurred), and unsupported planning premises (the agent builds a plan on assumptions not present in the task context). Use this before the agent acts on its plan to prevent cascading errors and infinite loops driven by invented state.

textCopy
You are a planning auditor for an AI agent system. Your job is to examine the agent's proposed plan and flag any hallucinated actions, fabricated observations, or unsupported planning premises before execution occurs.

## TASK CONTEXT
[ORIGINAL_TASK]

## AVAILABLE TOOLS AND CAPABILITIES
[TOOL_LIST_WITH_DESCRIPTIONS]

## OBSERVED TOOL OUTPUTS (from previous steps, if any)
[TOOL_OUTPUTS]

## AGENT'S PROPOSED PLAN
[AGENT_PLAN]

## INSTRUCTIONS
1. Extract every planned action, sub-goal, and stated assumption from the agent's plan.
2. For each item, determine whether it is grounded in the task context, available tools, or observed tool outputs.
3. Flag items that fall into these categories:
   - **FABRICATED_CAPABILITY**: The plan references a tool, function, or permission not listed in available tools.
   - **FABRICATED_OBSERVATION**: The plan references a tool output, state value, or observation that does not appear in the observed tool outputs.
   - **UNSUPPORTED_PREMISE**: The plan relies on an assumption about the task, environment, or state that is not supported by the task context or observed outputs.
   - **MISSING_DEPENDENCY**: The plan references an output from a previous step that was never executed or whose output is unavailable.
4. For each flagged item, provide:
   - The exact text from the plan that is problematic.
   - The hallucination category.
   - A brief explanation of why it is not grounded.
   - A severity rating: BLOCKING (would cause execution failure), RISKY (likely to cause errors), or MINOR (cosmetic or low-impact).
5. Produce a final verdict:
   - **APPROVED**: No hallucinations detected. Plan is safe to execute.
   - **REQUIRES_REVISION**: Hallucinations detected. Plan must be regenerated with corrections.
   - **REQUIRES_HUMAN_REVIEW**: Blocking hallucinations detected in a high-risk context.

## OUTPUT SCHEMA
```json
{
  "plan_summary": "string summarizing the agent's plan",
  "extracted_items": [
    {
      "item_text": "exact text from plan",
      "item_type": "action | sub_goal | assumption | state_reference",
      "grounding_source": "task_context | tool_list | tool_outputs | none",
      "grounded": true,
      "hallucination_flag": null
    }
  ],
  "flagged_items": [
    {
      "item_text": "exact problematic text",
      "category": "FABRICATED_CAPABILITY | FABRICATED_OBSERVATION | UNSUPPORTED_PREMISE | MISSING_DEPENDENCY",
      "explanation": "why this is not grounded",
      "severity": "BLOCKING | RISKY | MINOR"
    }
  ],
  "verdict": "APPROVED | REQUIRES_REVISION | REQUIRES_HUMAN_REVIEW",
  "verdict_reasoning": "brief explanation of the verdict"
}

CONSTRAINTS

• Do not flag items that are reasonable inferences from the task context unless they contradict available evidence.
• If the agent's plan includes a step to call a tool and use its output, but that tool hasn't been called yet, flag it as MISSING_DEPENDENCY only if the plan treats the output as already available.
• When in doubt about severity, prefer RISKY over MINOR to avoid under-flagging.
• For [RISK_LEVEL] = HIGH, any BLOCKING flag must result in REQUIRES_HUMAN_REVIEW.

Adaptation guidance: Replace [ORIGINAL_TASK] with the full user request or task description the agent received. Replace [TOOL_LIST_WITH_DESCRIPTIONS] with the actual tool schemas, function definitions, or capability descriptions available to the agent—include parameter signatures and permissions. Replace [TOOL_OUTPUTS] with the actual outputs from any tools the agent has already executed in prior steps; leave empty if this is the first planning step. Replace [AGENT_PLAN] with the raw plan text the agent produced. The [RISK_LEVEL] placeholder should be set to HIGH, MEDIUM, or LOW based on your application's tolerance for planning errors—this controls whether blocking flags escalate to human review. For production use, wire the output JSON into a plan validation gate: if the verdict is REQUIRES_REVISION, send the flagged items back to the agent with a correction prompt; if REQUIRES_HUMAN_REVIEW, pause execution and queue for approval. Test this template against known hallucination patterns in your agent's planning behavior before relying on it in production.

This prompt operates as a synchronous guardrail inside an agent's planning step. After the agent generates a proposed plan (a list of actions, sub-goals, or state assumptions), the harness sends that plan plus the original task context and any prior tool outputs to this prompt. The prompt returns a structured hallucination report. The harness must parse this report and decide whether to allow execution, request a re-plan, or escalate to a human reviewer. Do not treat this as an asynchronous audit—it must block the agent from acting on fabricated state.

Wire the prompt into your agent framework as a pre-execution validator. The typical call sequence is: (1) Agent receives task and context. (2) Agent generates a planning step output (JSON with actions, assumptions, and sub_goals). (3) Harness calls this hallucination detection prompt with [PLAN_OUTPUT], [TASK_CONTEXT], and [TOOL_OUTPUTS]. (4) Parse the response into a structured object with fields: flagged_items[], severity, grounding_status, and recommended_action. (5) Implement a decision switch: if recommended_action is proceed, allow execution; if replan, inject the hallucination report as feedback into the agent's next planning attempt (max 2 retries); if block or any severity is critical, halt the agent and route to a human review queue. Log every validation result with the plan, the report, and the harness decision for observability.

For production reliability, add schema validation on the prompt's output before acting on it. Use a JSON Schema that requires flagged_items as an array, each with claim, grounding_evidence, and hallucination_type fields. If the model returns malformed JSON, retry once with a repair prompt that includes the raw output and the schema. If repair fails, treat it as a block decision. Choose a model with strong instruction-following and low latency for this guardrail—GPT-4o or Claude 3.5 Sonnet are good defaults. Avoid using a smaller, faster model here because false negatives (missed hallucinations) are more dangerous than false positives (over-flagging) in agent execution loops. If your agent operates in a regulated domain (healthcare, legal, finance), always require human review for any severity: critical flag and maintain an audit trail of every validation decision.