Task Success Rate

The foundation of Task Success Rate is a precisely defined goal. This is not a vague instruction but a deterministic success criterion that the agent's final output must satisfy. Measurement involves verifying that the agent's actions or generated content fully address the user's original intent.

• Example: For an agent tasked with booking a flight, success is not just generating a response, but producing a valid, confirmed booking reference for a flight matching the specified date, budget, and destination constraints.
• Failure Modes: Include the agent misunderstanding the request, providing incomplete information, or solving a related but incorrect problem.

Success requires both completeness (the agent executed all necessary steps) and correctness (each step was performed accurately). This is often assessed via ground truth comparison or validation rules.

• Structured Validation: For tasks with defined outputs (e.g., API calls, data extraction), success can be automatically validated against a schema or by checking the state change in an external system.
• Unstructured Evaluation: For open-ended tasks (e.g., writing a summary), evaluation may require LLM-as-a-judge scoring or human review against a rubric assessing factual accuracy, relevance, and coherence.
• Partial Credit: Some frameworks assign weighted scores for partially correct outcomes, providing a more nuanced metric than a binary pass/fail.

A 'task' must be scoped as an atomic unit of work within a defined operational session. This prevents conflation of multi-step processes and ensures consistent measurement.

• Session Definition: A session is a bounded interaction, from initial user prompt to final agent response, which may include internal planning and tool use loops.
• Atomicity: A task like 'analyze this quarterly report and email a summary to the team' contains two atomic sub-tasks: analysis and email dispatch. Success rate can be measured for each sub-task individually and for the composite task.
• Context Management: The agent must maintain necessary context (via episodic memory or conversation history) throughout the session to achieve the goal.

Reliable measurement depends on comprehensive telemetry pipelines. The agent system must be instrumented to emit events for critical actions, decisions, and final outputs.

• Key Signals: Events for session start/end, tool calls, final answer submission, and internal validation checks.
• Trace Collection: Distributed tracing links all events within a session, enabling reconstruction of the agent's reasoning path for failure analysis.
• Integration with Evaluation Harness: Telemetry feeds into an evaluation harness that automatically scores outcomes against predefined success criteria, calculating the aggregate success rate.

A raw Task Success Rate percentage is only meaningful when compared to a performance baseline. This involves running the agent against a benchmark suite of representative tasks.

• Establishing Baselines: A baseline success rate is established for a given agent version and task domain (e.g., 87% on customer support ticket resolution).
• Detecting Regressions: After model updates or prompt changes, the benchmark is re-run. A statistically significant drop in success rate indicates a performance regression.
• A/B Testing: Success Rate is a primary metric for A/B tests comparing different agent architectures, model providers, or prompting strategies.

Real-world measurement must account for ambiguous scenarios and edge cases that complicate binary success/failure labeling.

• User-Ambiguous Requests: If a user request is inherently ambiguous, multiple valid outcomes may exist. Success criteria may need to accommodate a set of acceptable answers.
• Resource Failures: If a task fails due to an external API being down (a tool call failure), this may be counted differently than a failure due to the agent's own reasoning error.
• Degrees of Success: Some frameworks incorporate quality scores (e.g., 1-5 scale) alongside the binary success flag to capture the nuance of partially successful or suboptimal completions.

Latency is the total time delay between the initiation of a request to an AI agent and the completion of its response. It is a critical user experience metric, especially for interactive agents.

• Components: Includes network transmission, queuing, model inference (Time to First Token, Time per Output Token), and external tool/API call durations.
• Trade-off with Success Rate: Aggressive latency SLOs (e.g., < 2 seconds) may force agents to time out before completing complex tasks, artificially lowering the measured Task Success Rate.
• Measurement: Typically tracked as average, median, and tail latencies (P95, P99).

Hallucination Rate quantifies the frequency with which an AI agent generates confident but factually incorrect or nonsensical output not grounded in its source data or tools. It is a direct antagonist to Task Success Rate.

• Impact on Success: A high hallucination rate on factual tasks (e.g., data lookup, code generation) directly causes task failure, as the output is unusable.
• Measurement: Requires comparison to a verifiable ground truth or use of Retrieval-Augmented Generation (RAG) faithfulness metrics.
• Mitigation: Techniques like prompt engineering, chain-of-thought verification, and improved retrieval precision aim to reduce this rate.

Cost Per Task is the total computational and financial expenditure required for an agent to attempt a single task. It provides the economic context for Task Success Rate.

• Calculation: Sum of costs for input/output tokens, external API calls, compute runtime, and vector database queries.
• Business Interpretation: A 95% success rate is less valuable if the cost per successful task is prohibitively high. The optimal metric is often Cost per Successful Task.
• Optimization: Engineers balance success rate against cost by adjusting agent complexity (e.g., limiting reflection cycles, choosing cheaper models for sub-tasks).

Tool Call Success Rate is the percentage of times an agent correctly executes an external API or software function call, yielding a usable result. It is a leading indicator for overall Task Success Rate in tool-using agents.

• Granular Metric: Breaks down overall agent failure into specific tooling failures (e.g., authentication errors, malformed requests, API timeouts).
• Instrumentation: Requires specific observability hooks in the Tool Calling layer to capture input, output, status codes, and errors.
• Dependency: A task requiring three sequential tool calls, each with a 95% success rate, has a theoretical maximum task success rate of ~85.7% (0.95^3).

Step Success Rate measures the accuracy of an agent's individual reasoning or action steps within a multi-step plan. It is a finer-grained diagnostic than final Task Success Rate.

• Relation to Overall Success: Low step success rate often cascades, causing overall task failure. Monitoring it helps identify where in a planning loop an agent fails.
• Use in Evaluation: Used in Evaluation Harnesses to score sub-tasks (e.g., 'correctly parsed user query', 'chose appropriate tool', 'validated tool output').
• Improvement Lever: Improving step success via better prompts, few-shot examples, or model selection directly lifts the final task metric.

A Service Level Objective (SLO) is a target for a Service Level Indicator (SLI) like Task Success Rate. The Error Budget is the allowable deviation from that target.

• Example SLO: 'Task Success Rate will be ≥ 98% over a 30-day rolling window.'
• Error Budget Calculation: If the SLO is 98%, the error budget is 2% of failures. Exhausting this budget triggers a freeze on risky changes.
• Operational Use: Defines the reliability contract for an agentic service. Guides trade-off decisions between launching new features (which may reduce success rate) and stabilizing existing ones.