Planning Success Rate

Planning Success Rate is the percentage of times an autonomous agent successfully decomposes a high-level goal into a valid, executable sequence of sub-tasks or actions. It is a leading indicator of system health, as a failure in planning typically prevents successful task execution.

• Calculation: (Number of successful plans generated / Total number of planning attempts) * 100.
• Validity Criteria: A plan is considered successful if it is logically coherent, respects all defined constraints (safety, resources), and is composed of actions the agent is authorized and able to execute.
• Example: An agent tasked with 'generate a monthly sales report' must plan steps like: 1) query CRM API, 2) aggregate data, 3) format into presentation. Failure to create this sequence counts against the rate.

This SLI measures the quality of the agent's reasoning output, not the runtime success of the actions themselves. It is distinct from related downstream SLIs.

• vs. Action Success Ratio: Planning Success Rate evaluates the blueprint; Action Success Ratio measures the success of individual tool calls during execution.
• vs. Task Completion Rate: A high Planning Success Rate is necessary but not sufficient for a high Task Completion Rate. A valid plan can still fail due to external API errors or unexpected data.
• Primary Use: Isolates failures in the cognitive architecture (e.g., prompt logic, context window limits, reasoning model capability) from failures in the execution environment.

The metric's value is highly dependent on the scope and complexity of the task assigned to the agent. Effective monitoring requires segmentation.

• Task Complexity Tiers: Baseline rates should be established for different task types (e.g., simple data lookup vs. multi-document analysis with synthesis).
• Context Sensitivity: Success rate can vary dramatically based on the available context (e.g., few-shot examples in the prompt, state from agentic memory). A drop may indicate context window overflow or degraded retrieval quality.
• Domain Specificity: An agent fine-tuned for legal contract review will have a higher planning success rate on legal tasks than on unrelated medical diagnostics.

A low or declining Planning Success Rate signals specific issues within the agent's reasoning stack. Common failure modes include:

• Goal Ambiguity: The user instruction is too vague for the agent to parse into discrete steps.
• Constraint Violation: The agent's proposed plan breaches a safety guardrail or operational policy (e.g., attempts an unauthorized API call).
• Resource Unawareness: The agent plans steps that require unavailable tools, data sources, or exceed computational budgets.
• Reasoning Model Hallucination: The underlying LLM generates an illogical, inconsistent, or physically impossible sequence of actions.
• Context Window Exhaustion: The agent cannot hold all necessary information in its working memory to formulate a complete plan.

As a core SLI, Planning Success Rate is used to define Service Level Objectives (SLOs) and manage error budgets for autonomous systems.

• SLO Example: 'The agent's Planning Success Rate must be >= 99.5% over a 30-day rolling window.'
• Error Budget Consumption: Every planning failure consumes the error budget. A rapid burn rate indicates systemic issues requiring immediate engineering attention, potentially halting feature deployments.
• Composite SLI Input: Planning Success Rate is often a key component of a Composite SLI for overall agent efficacy or a Resiliency Score, combined with metrics like Self-Correction Success Rate.

Accurate measurement requires instrumentation within the agent's planning loop and integration into telemetry pipelines.

• Instrumentation Point: The metric must be captured after the planning phase outputs a plan and before execution begins. This often requires a validation step.
• Required Telemetry: Each planning attempt must emit a structured log event containing the task goal, the generated plan, a success/failure flag, and the failure reason code.
• Dashboards and Alerts: Real-time dashboards should track the rate segmented by agent version, task type, and user. Alerting rules should trigger on sustained breaches of the SLO threshold or anomalous drops.
• Link to Traces: Planning failures should be linked to distributed traces and agent reasoning traces for deep debugging.

An Agentic SLI is a quantitative measure of a specific aspect of an autonomous agent's performance. Unlike traditional SLIs, they are tailored to the unique behaviors of AI agents.

• Examples: Planning Success Rate, Task Completion Rate, Action Success Ratio.
• Purpose: Provides the raw, measurable data used to assess operational health and drive SLOs.
• Key Consideration: Must be directly observable from the agent's telemetry, such as execution logs or reasoning traces.

An Agentic SLO is a target value or range for an Agentic SLI, defining the acceptable performance level for a system over a specified period.

• Example: "Planning Success Rate ≥ 99.5% over a 30-day rolling window."
• Function: Converts an SLI into a business or operational goal. The gap between the SLI and the SLO determines the error budget.
• Critical for: Setting reliability expectations, prioritizing engineering work, and managing risk in production deployments.

Task Completion Rate measures the percentage of assigned high-level tasks an autonomous agent successfully finishes within defined constraints (time, cost, correctness).

• Relation to Planning: A successful plan is a prerequisite for task completion. This metric evaluates the end-to-end success of the agent's entire workflow.
• Contrast with Planning Success Rate: Planning Success Rate measures the validity of the plan itself, while Task Completion Rate measures the successful execution of the planned actions and achievement of the final goal.

Self-Correction Success Rate measures the effectiveness of an agent's recursive error correction loops in identifying and remediating its own failures without human intervention.

• Direct Dependency: A low Planning Success Rate may trigger self-correction cycles where the agent replans.
• Key Mechanism: This SLI quantifies the agent's resilience and ability to recover from planning or execution failures autonomously.
• Engineering Insight: A high Self-Correction Success Rate can compensate for a lower initial Planning Success Rate, maintaining overall system reliability.

Redundant Action Ratio measures the proportion of steps or tool calls within an agent's execution plan that are unnecessary or duplicative.

• Diagnostic for Planning Quality: A high ratio indicates inefficiency in the planning phase, such as poor state tracking or failure to prune irrelevant actions.
• Impact: Increases latency, cost, and potential for error without contributing to task success.
• Optimization Target: Improving planning algorithms directly aims to reduce this ratio, making execution leaner and more deterministic.

An Error Budget is the allowable amount of time an autonomous agent system can fail to meet its SLOs within a defined compliance period.

• Calculation: Derived from the SLO. For example, a 99.5% monthly SLO allows for ~3.6 hours of failure.
• Strategic Tool: Balances reliability with the pace of innovation. Consuming the budget on planned changes (like deploying a new planner) is acceptable; unplanned consumption signals operational issues.
• Governance: Ties the technical metric (Planning Success Rate) directly to business decisions about risk and deployment velocity.