Action Success Ratio

The Action Success Ratio measures success at the level of individual tool calls or API executions, not the overall task. This provides high-resolution insight into where failures occur.

• Example: An agent completing a 5-step workflow where 4 API calls succeed and 1 fails has an Action Success Ratio of 80% (4/5), even if the overall task fails.
• This granularity is essential for debugging specific integration points and identifying unreliable external services.

Each action is evaluated on a binary pass/fail basis, typically defined by the HTTP status code or structured error response from the called tool or API.

• Success Criteria: A 2xx HTTP status code and the return of expected, parsable data.
• Failure Criteria: A 4xx or 5xx HTTP status, a timeout, or the return of an error message in the tool's defined output schema.
• This objective criteria eliminates ambiguity, making the SLI computable and automatable for real-time dashboards.

This SLI serves as a leading indicator for the stability of an agent's connections to external systems and data sources. A declining ratio directly signals integration issues.

• It monitors the operational reliability of third-party APIs, databases, and internal microservices the agent depends on.
• Sudden drops can pinpoint service outages, schema changes, or authentication problems before they critically impact broader business workflows.

The raw Action Success Ratio is used to define Service Level Objectives (SLOs) that formalize reliability targets for agent operations.

• Example SLO: "99.5% of agent tool calls shall succeed over a 30-day rolling window."
• Violations of this SLO consume the system's Error Budget, guiding decisions on stability investments vs. feature development.
• It is a key input for Composite SLIs that measure overall agent efficiency or cost-effectiveness.

Action Success Ratio must be analyzed alongside task-level SLIs like Task Completion Rate to provide a complete performance picture.

• A high Action Success Ratio with a low Task Completion Rate may indicate successful individual steps but flawed agent logic or planning that sequences them incorrectly.
• Conversely, a low Action Success Ratio often directly causes a low Task Completion Rate, pointing to integration failures as the root cause.

Accurate measurement requires instrumenting the agent's execution layer with lightweight wrappers or decorators around every tool invocation.

• These wrappers intercept the call, record start time, parse the response, and emit a success/failure metric with relevant tags (e.g., tool_name, error_type).
• The data is sent to an observability backend (e.g., Prometheus, Datadog) for aggregation and alerting.
• This implementation is part of Tool Call Instrumentation, a core practice in agentic observability.

A customer service agent uses the Action Success Ratio to monitor its reliability in executing backend operations. For example, when a user requests an order status update, the agent must call the get_order_details API. A low ASR here directly impacts customer satisfaction.

• Primary Tool Calls: Lookup customer profile, fetch order history, update support ticket.
• Common Failures: API timeouts, authentication errors, or invalid data formats returned by legacy systems.
• SLO Target: Engineering teams might set an SLO of 99.5% for this ratio, meaning no more than 0.5% of tool calls can fail during a billing cycle.

An autonomous supply chain agent that coordinates logistics uses ASR to ensure its instructions to external systems are executed. A failure to call a warehouse inventory API could halt a fulfillment pipeline.

• Key Actions: Reserve inventory, generate shipping labels, update tracking databases, adjust demand forecasts.
• Impact Analysis: A dip in ASR for the allocate_inventory tool directly correlates with delayed shipments. Monitoring this SLI allows teams to pinpoint unreliable third-party APIs.
• Composite SLI Integration: ASR is often combined with End-to-End Task Latency and Task Completion Rate to form a Composite SLI for overall orchestration health.

In algorithmic trading, an agent's action success is paramount. A failed execute_order call to a brokerage API could result in significant financial loss. The ASR SLI is monitored in real-time.

• High-Stakes Tools: Market data queries, pre-trade compliance checks, order execution, portfolio rebalancing.
• Anomaly Detection: A sudden drop in ASR triggers an immediate Alerting Rule, potentially causing the agent to enter a safe fallback mode or halt trading.
• Relationship to Cost: Failed actions often still incur costs. A low ASR will negatively impact the Cost Per Successful Task SLI.

An agent that automates cloud infrastructure management relies on a high ASR for its tool calls to providers like AWS or Azure. A failed deploy_server action can cause service outages.

• Infrastructure Tools: Provision resources, scale clusters, apply security patches, run database backups.
• SLO Burn Rate Connection: A sustained low ASR will rapidly consume the Error Budget for system reliability. Teams track the SLO Burn Rate to understand how quickly they are deviating from targets.
• Health Correlation: ASR is a leading indicator for the Health Check Success Rate of the services the agent manages.

A clinical agent that retrieves patient data from Electronic Health Record (EHR) systems uses ASR to ensure compliance and accuracy. A failed retrieve_lab_results call could impact diagnostic support.

• Sensitive Operations: Query patient records, schedule appointments, submit insurance codes, anonymize data for research.
• Guardrail Compliance: Each tool call must also adhere to privacy policies. The Guardrail Compliance Rate SLI works in tandem with ASR to measure both success and safety.
• Root Cause Analysis: A low ASR prompts a formal Root Cause Analysis (RCA) to determine if failures are due to network issues, API changes, or incorrect parameter formatting by the agent.

In a multi-agent system for research, a "Searcher" agent might call web search, academic database, and internal wiki APIs. The ASR for each agent's tool calls determines the system's overall information retrieval reliability.

• Coordinated Actions: One agent's successful API call provides context for another's subsequent tool use. A failure breaks the chain.
• System-Wide View: The Multi-Agent Observability platform aggregates individual agent ASR metrics to calculate a system-wide Workflow Completion Rate.
• Redundancy Planning: A low ASR for a critical tool may lead to engineering a fallback where a different agent with an alternative tool attempts the same action.

An Agentic SLI is a quantitative measure of a specific aspect of an autonomous agent's performance. It is the fundamental building block for observability, providing a direct signal of health and effectiveness.

• Examples: Action Success Ratio, Planning Success Rate, End-to-End Task Latency.
• Purpose: To objectively answer questions like "How often do our agent's tool calls succeed?" or "How fast does the agent complete tasks?"
• Implementation: Typically measured as a ratio, rate, average, or percentile over a defined time window.

An Agentic SLO is a target value or range for an Agentic Service Level Indicator. It defines the acceptable level of performance for an autonomous agent system, creating a contract for reliability.

• Structure: "Action Success Ratio SHALL BE ≥ 99.5% over a 30-day rolling window."
• Function: SLOs turn raw SLI measurements into actionable business goals. They are used to manage error budgets, guide development priorities, and trigger alerts.
• Key Concept: An SLO is not a performance maximum; it is the minimum acceptable threshold for users.

An Error Budget is the allowable amount of time an autonomous agent system can fail to meet its Service Level Objectives within a defined compliance period. It is calculated directly from the SLO.

• Calculation: If your SLO is 99.5% monthly availability, your error budget is 0.5% of the month (~3.6 hours).
• Purpose: To explicitly balance reliability with innovation. Spending the budget on risky changes is acceptable; exhausting it mandates a focus on stability.
• Management: Error budget burn rate is a critical metric for SRE and DevOps teams managing agent deployments.

Task Completion Rate is an Agentic SLI that measures the percentage of assigned, high-level tasks an autonomous agent successfully finishes within defined operational constraints (time, cost, correctness).

• Scope vs. Action Success Ratio: This is a higher-level metric. A single task may involve many individual actions. A high Action Success Ratio is necessary but not sufficient for a high Task Completion Rate.
• Example: An agent's task is "book a flight and hotel." Task Completion Rate measures if the entire itinerary was successfully booked, while Action Success Ratio measures if each individual API call to the flight or hotel service succeeded.

Self-Correction Success Rate is an Agentic SLI that measures the effectiveness of an autonomous agent's recursive error correction loops. It quantifies how often an agent can identify and remediate its own failures without human intervention.

• Relation to Action Success Ratio: This metric often acts on failures detected by a low Action Success Ratio for a specific step. A high Self-Correction Success Rate can salvage a task despite initial action failures.
• Mechanism: Involves the agent detecting an error (e.g., a failed API call), reasoning about the cause, formulating a new plan (e.g., retry with different parameters, use an alternative tool), and executing it successfully.

A Composite SLI is a Service Level Indicator derived from the mathematical combination of two or more underlying Agentic SLIs. It provides a unified score for a complex aspect of agent performance.

• Purpose: To create a single, high-level metric for concepts like "overall agent efficiency" or "safety score" that cannot be captured by one raw SLI.
• Example: A Weighted Efficiency Score could combine Action Success Ratio (weight: 40%), Redundant Action Ratio (weight: -30%), and End-to-End Task Latency (weight: 30%) into one value.
• Use Case: Often used for executive dashboards or as the primary metric for automated canary analysis.