Agent Planning Step Failure Eval Prompt

This prompt is designed for agent developers and AI reliability engineers who need to isolate the root cause of a production agent failure when the automated evaluation score is below the threshold and the primary suspect is a planning error. The core job-to-be-done is to move beyond the surface-level eval score and pinpoint the exact planning step where the agent's internal task decomposition diverged from the required workflow. You should use this prompt when you have access to a structured execution trace that includes the agent's initial plan, its step-by-step actions, and the final output that was evaluated against a known rubric.

To use this effectively, you must provide the full evaluation rubric that defined the failure, the agent's complete execution trace (including its initial plan and any plan revisions), and the failing output. The prompt works by forcing a step-by-step comparison between the 'required plan' implied by the rubric and the 'executed plan' captured in the trace. It will produce a structured analysis that identifies the first point of divergence, explains the causal chain from that divergence to the final output failure, and recommends a specific corrective action, such as modifying a planning instruction in the system prompt or adding a verification step. Do not use this prompt if the execution trace is unavailable or incomplete, as the analysis will be speculative. It is also the wrong tool if the failure is clearly a simple tool-call argument error or if you suspect the LLM judge itself is misconfigured; for those scenarios, use the Tool-Call Error Correlation with Eval Failure Prompt or the LLM Judge Score Inconsistency Investigation Prompt instead.

Before running this analysis, verify that the trace contains step-level detail. A trace that only shows final input and output is insufficient. The prompt requires the agent's stated plan or reasoning before each action. If your agent framework does not log this, you will need to add that observability before this playbook becomes useful. The output of this prompt is a diagnostic artifact, not an automatic fix. The recommended corrective action should be treated as a strong hypothesis to be tested in a staging environment with a regression test suite before being deployed to production. For high-stakes agent workflows, always require a human to review the trace analysis and approve the corrective action before any prompt or code change is merged.

This prompt template is designed to be pasted directly into your investigation workflow. It instructs the model to act as a diagnostic engine, comparing the agent's intended plan against its actual execution trace to pinpoint the root cause of an evaluation failure. The goal is to produce a structured, evidence-backed analysis that connects a planning divergence to a specific eval rubric violation, moving beyond surface-level output critique.

textCopy
You are an AI reliability engineer diagnosing an agent evaluation failure. Your task is to analyze the provided trace data and determine why the agent's planning step caused the final output to fail its evaluation.

[CONTEXT]
- Task Description: [TASK_DESCRIPTION]
- Eval Rubric: [EVAL_RUBRIC]
- Eval Failure Reason: [EVAL_FAILURE_REASON]

[TRACE_DATA]
- Agent's Initial Plan (from planning step): [INITIAL_PLAN]
- Agent's Execution Trace (tool calls, observations, plan revisions): [EXECUTION_TRACE]
- Agent's Final Output: [FINAL_OUTPUT]

[OUTPUT_SCHEMA]
Produce a JSON object with the following structure:
{
  "plan_vs_execution_summary": "A one-sentence summary of the core divergence.",
  "root_cause_analysis": {
    "divergence_point": "The exact step in the execution trace where the agent deviated from the required plan.",
    "missing_or_incorrect_step": "The specific planned step that was skipped, hallucinated, or executed incorrectly.",
    "causal_link_to_eval_failure": "A clear explanation of how this divergence directly caused the eval failure described in the context.",
    "evidence_from_trace": "A direct quote or log line from the execution trace that proves the divergence."
  },
  "severity_assessment": "Critical | High | Medium | Low",
  "recommended_remediation": "A concrete suggestion for a prompt, tool, or policy change to prevent this failure in the future."
}

[CONSTRAINTS]
- Base your analysis strictly on the provided trace data. Do not invent facts.
- If the trace data is insufficient to determine a root cause, set the root_cause_analysis to null and explain why in the plan_vs_execution_summary.
- The causal_link_to_eval_failure must explicitly reference the eval rubric and failure reason.

To adapt this template, replace the square-bracket placeholders with data from your production trace and evaluation systems. The [EVAL_RUBRIC] and [EVAL_FAILURE_REASON] fields are critical for grounding the analysis. If your trace format differs, adjust the [TRACE_DATA] section to match your logging schema. For high-stakes applications, always pair this automated analysis with a human review of the root_cause_analysis before implementing the recommended_remediation.

This prompt is designed to be the analytical core of an automated eval failure investigation workflow, not a one-off debugging tool. The harness should receive a structured input payload containing the failed eval rubric, the agent's final output, and the full execution trace. The prompt's output—a plan-vs-execution divergence report—should be parsed as structured JSON and fed into downstream systems: ticketing, alerting, or a human review queue. Because this analysis is used to decide whether to roll back a prompt version or update agent planning logic, the harness must enforce strict validation on both the input and output schemas before any action is taken.

Wire the prompt into an application by building a dedicated investigation service. This service should: (1) Accept a webhook or queue message containing an eval failure event with a trace ID. (2) Fetch the full trace, eval rubric, and model output from your observability store. (3) Assemble the prompt with the [TRACE_JSON], [EVAL_RUBRIC], [FINAL_OUTPUT], and [PLANNING_INSTRUCTIONS] placeholders. (4) Call a capable reasoning model (e.g., Claude 3.5 Sonnet or GPT-4o) with response_format set to a JSON schema matching the expected output. (5) Validate the returned JSON against the schema. If validation fails, retry once with a repair prompt that includes the validation error. If it fails again, escalate to a human analyst with the raw trace. (6) On success, write the structured divergence report to your investigation database and check the divergence_severity field: critical or high findings should automatically create a ticket and notify the agent development team. medium findings can be batched for weekly review. low or none findings should be logged for trend analysis but require no immediate action.

The highest-risk failure mode in this harness is acting on an incorrect analysis. A hallucinated divergence report could trigger an unnecessary prompt rollback or, worse, mask a real planning defect. Mitigate this by requiring human review for any critical severity finding before a rollback is executed. Additionally, log the raw prompt, model response, and validation result for every investigation. This audit trail is essential for debugging the investigation pipeline itself and for demonstrating due diligence during incident postmortems. Do not use a cheap or fast model for this task; the reasoning required to correlate plan steps with trace events demands a frontier model. Finally, ensure your trace data includes step-level timestamps and tool-call arguments—without this granularity, the prompt cannot produce a reliable divergence analysis.