Task Completion Rate

The unambiguous specification of the work unit assigned to the agent. This is the foundational component, as the rate cannot be measured without a clear definition of the task's start condition, end condition, and boundaries. For example:

• Start: User submits a natural language request: "Summarize the Q3 sales report."
• End: Agent returns a 200-word summary to the user interface.
• Boundaries: The task scope is limited to the 'Q3_sales.pdf' document; analyzing other files constitutes a different task. Poorly scoped tasks lead to inaccurate SLI calculations.

The explicit, measurable conditions that must be met for a task to be counted as 'completed.' These are multi-dimensional constraints beyond mere termination. Key criteria include:

• Functional Correctness: Does the output satisfy the user's intent? (e.g., the summary accurately reflects the report's key figures).
• Operational Constraints: Was the task finished within allowed time (SLO for latency), cost (token/API budget), and resource limits?
• Policy Compliance: Did the agent's actions and final output adhere to all safety guardrails, ethical guidelines, and data handling policies? A task that finishes quickly but violates a guardrail is a failure.

The logic for categorizing task outcomes, which directly feeds the numerator and denominator of the rate calculation. Standard classifications are:

• Success: All success criteria are met.
• Failure: The agent cannot produce a valid output (e.g., gets stuck in a loop, times out, produces a critically incorrect result).
• Partial Success / Degraded: The agent produces a useful output but violates a non-critical constraint (e.g., slightly exceeds the time budget, uses a fallback method). This may be tracked separately or counted as a failure depending on SLO strictness.
• Invalid: The task itself was malformed or unsupported; often excluded from the rate calculation to avoid skewing the metric.

The timeframe over which the rate is calculated and the method for rolling up individual task outcomes. This determines the SLI's sensitivity and stability.

• Window: Typically a rolling window (e.g., last 24 hours, last 7 days) or a calendar-aligned period (e.g., per hour, per day).
• Aggregation Formula: The standard calculation is (Successful Tasks / Total Eligible Tasks) * 100.
• Weighting: Some implementations apply weights, such as prioritizing the completion rate of high-criticality tasks over low-priority ones. The aggregation method must be consistent to enable trend analysis and SLO tracking.

Recognition that an agent's ability to complete a task is often contingent on external systems and environmental state. This component ensures the SLI reflects agent performance, not external failures.

• Dependency Health: Was a required external API, database, or tool unavailable? Sophisticated measurement may segment the rate by dependency status.
• Context Validity: Did the agent have access to the necessary input data, user permissions, and session context? A task failure due to missing context may be classified differently than a failure due to flawed agent reasoning.
• Fallback Handling: If the agent successfully invokes a contingency plan, does that count as a success for the original task? This must be defined in the success criteria.

The system—automated or human-in-the-loop—that applies the success criteria to classify each task outcome. This is the operational engine of the SLI.

• Automated Evaluators: Rule-based checks (e.g., output schema validation, latency threshold), model-based scorers, or ground-truth comparison.
• Human Evaluation: For complex or subjective tasks, a percentage of outcomes may be sampled for human review, with the results used to calibrate automated systems.
• Trace Analysis: The evaluator consumes the agent's reasoning trace, tool call logs, and final output to make a deterministic classification. The reliability of the Task Completion Rate depends entirely on the accuracy and consistency of this mechanism.

An Agentic SLI is a quantitative measure of a specific aspect of an autonomous agent's performance. It is the fundamental building block for observability, providing a direct signal of health and effectiveness.

• Examples: Task Completion Rate, Planning Success Rate, End-to-End Task Latency.
• Purpose: To answer the question, "How is the service performing?" from a user's perspective.
• Key Property: Must be a measurable, user-centric attribute, not an internal system metric like CPU usage.

An Agentic SLO is a target value or range for an Agentic Service Level Indicator (SLI). It defines the acceptable level of performance for an autonomous agent system over a specified period.

• Structure: "SLI X must be ≥ Y% over rolling 30-day window."
• Example: "Task Completion Rate SLO: ≥ 99.5% over 30 days."
• Purpose: Creates a clear, measurable reliability target for engineering and business stakeholders. SLOs are used to manage error budgets and guide prioritization.

An Error Budget is the allowable amount of time an autonomous agent system can fail to meet its Service Level Objectives (SLOs) within a defined compliance period. It is calculated as (100% - SLO%) * period.

• Function: Balances reliability with innovation pace. If the budget is consumed, focus shifts to stability; if budget remains, new features can be deployed.
• Example: With a 99.5% monthly SLO, the error budget is 0.5% of the month (~3.6 hours).
• Management: Burn rate alerts trigger when the budget is being consumed too quickly.

Planning Success Rate is an Agentic SLI that measures the percentage of times an autonomous agent successfully decomposes a high-level goal into a valid, executable sequence of sub-tasks or actions.

• Relation to Task Completion: A prerequisite SLI. A failed plan often leads to a failed task.
• Measurement: Evaluates the agent's reasoning and decomposition capability before execution begins.
• Example: An agent tasked with "book a business trip" must first generate a valid plan: [1. Check calendar, 2. Search flights, 3. Book hotel]. This SLI tracks how often that initial plan is sound.

Action Success Ratio is an Agentic SLI that measures the proportion of individual tool calls or API executions performed by an autonomous agent that complete successfully without error.

• Granularity: A more granular metric than Task Completion Rate. A single task may involve multiple actions.
• Purpose: Identifies unreliable tools or integration points. A low ratio indicates issues with external APIs, authentication, or the agent's understanding of tool usage.
• Calculation: (Successful Tool Executions) / (Total Tool Executions).

End-to-End Task Latency is an Agentic SLI that measures the total time elapsed from when an autonomous agent receives a task to when it delivers a final, validated result.

• User-Centric: Captures the total wait time experienced by the user or calling system.
• Components: Includes planning time, execution time (serial or parallel actions), and any internal reasoning/validation loops.
• SLO Context: Often has a p95 or p99 latency SLO (e.g., "95% of tasks complete within 30 seconds"). This works in tandem with Task Completion Rate to define service quality.