Success Rate

Success Rate is defined as the ratio of successful tool or API invocations to the total number of invocations over a specified time window. The formula is:

• Success Rate = (Successful Invocations / Total Invocations) * 100%

A 'successful' invocation is typically defined by the absence of a terminal error. This includes:

• HTTP status codes in the 2xx range.
• The absence of a thrown exception or timeout.
• A response that is structurally valid and usable by the agent.

It is the inverse of Error Rate, where Error Rate = 1 - Success Rate.

Success Rate is most actionable when measured at multiple levels of granularity and tagged with Cost Attribution Tags. Effective monitoring breaks it down by:

• Per-Tool/API Endpoint: Identifies which specific external dependency is failing.
• Per-Agent or Task Type: Reveals if issues are specific to certain workflows.
• Per-User, Team, or Project: Enables cost and reliability accountability.
• Per-Deployment/Version: Critical for Canary Deployment analysis, comparing success rates between old and new agent logic.

This segmentation transforms a single metric into a diagnostic tool, pinpointing failure domains.

Success Rate is not static; it must be evaluated as a time-series metric. Key patterns include:

• Degradation Trends: A gradual decline may indicate resource exhaustion or growing data corruption in a downstream service.
• Sharp Drops: Often correlate with external API deployments, network outages, or credential issues.
• Error Budget Burn Rate: When linked to an SLO (e.g., 'Success Rate >= 99.9%'), the rate at which the Error Budget is consumed dictates operational urgency. A rapid burn rate triggers incident response and may halt feature deployments. Monitoring these dynamics is essential for proactive reliability engineering.

A measured Success Rate directly informs the configuration and triggers of system resilience mechanisms:

• Circuit Breaker Pattern: A sustained low Success Rate for a specific endpoint should 'trip' the circuit breaker, failing fast and preventing cascading failures.
• Retry Policies & Exponential Backoff: Transient errors (e.g., HTTP 429, 500) are retried. The Success Rate after retries is the user-visible metric. Aggressive retries on persistently failing endpoints waste resources.
• Dead Letter Queues (DLQ): Invocations that fail after all retries can be sent to a DLQ. The DLQ size relative to total volume is a lagging indicator of Success Rate problems requiring manual intervention.

Success Rate alone provides an incomplete picture of tool call health. It must be analyzed alongside:

• Tool Call Latency / P95 Latency: A call can succeed but be unusably slow. High latency often precedes failures.
• Payload Size: Unexpectedly large request/response sizes can lead to timeouts, affecting success.
• Rate Limit Telemetry: Successes may drop because quotas are exhausted, not due to endpoint failure.
• Synthetic Transaction results: Proactively measure success from outside the network.

A 'successful' call that returns incorrect data (e.g., '200 OK' with wrong information) is a silent failure not captured by this metric, requiring business logic validation.

Success Rate is implemented by instrumenting tool calls with observability standards like OpenTelemetry. Key practices include:

• Creating a Span for each tool call with Span Attributes for the tool name, endpoint, and HTTP status code.
• Recording a Span Event for failures or retries.
• Using a Span Exporter to send data to a backend (e.g., Prometheus, Datadog) where Success Rate is calculated as a metric from the count of spans with error statuses vs. total spans.
• Ensuring Trace Correlation links the tool call span to the broader agent Trace for root cause analysis. This standardized instrumentation ensures consistent, vendor-agnostic measurement.

Error Rate is the inverse of Success Rate, representing the proportion of tool or API invocations that result in a failure over a given period. It is typically measured by counting calls that return non-successful HTTP status codes (e.g., 4xx, 5xx) or result in thrown exceptions.

• Direct Counterpart: A high Error Rate directly indicates a low Success Rate.
• Granular Analysis: Errors are often categorized (e.g., client errors, server errors, timeouts) to diagnose root causes.
• SLO Impact: Error Rate is a primary input for calculating Error Budget consumption against reliability objectives.

A Service Level Indicator (SLI) is a quantitative measure of a service's performance from the user's (or agent's) perspective. For tool call instrumentation, Success Rate is a fundamental SLI.

• User-Centric Metric: An SLI measures what the consumer of the service experiences.
• Examples: Tool call latency, availability, and throughput are other common SLIs.
• Foundation for SLOs: SLIs provide the raw data against which Service Level Objectives (SLOs) are defined.

A Service Level Objective (SLO) is a target value or range for an SLI. It defines the reliability contract for a service. For tool dependencies, an SLO might be 'Success Rate must be ≥ 99.9% over a 30-day rolling window.'

• Reliability Target: Converts the SLI (Success Rate) into an explicit goal.
• Basis for Error Budgets: The difference between the SLO and actual performance defines the Error Budget.
• Drives Engineering Decisions: Breaching an SLO triggers investment in reliability improvements.

The Circuit Breaker Pattern is a resilience design pattern that prevents an agent from repeatedly calling a failing tool. It programmatically fails fast, allowing the dependency time to recover.

• Three States: Closed (normal operation), Open (failing fast), Half-Open (testing for recovery).
• Protects Success Rate: Stops cascading failures and wasted calls on unhealthy endpoints.
• Observability Integration: State changes (Open/Closed) are critical events logged as Span Events.

A Synthetic Transaction is a scripted test that proactively simulates an agent's tool calls from outside the production environment. It is used to monitor availability and validate Success Rate before real users or agents are impacted.

• Proactive Monitoring: Detects outages or degradations before they affect production traffic.
• Tests Full Stack: Validates the entire call path, including network, authentication, and API logic.
• Baselining: Establishes expected performance and Success Rate for comparison with real user monitoring (RUM) data.

A Canary Deployment is a release strategy where a new version of an agent or its tool-calling logic is deployed to a small subset of traffic. Instrumentation is used to compare its Success Rate and other metrics against the stable version.

• Risk Mitigation: Limits the impact of a bad release that could degrade Success Rate.
• A/B Testing for Reliability: Directly compares the performance of two code paths.
• Automated Rollback: If the canary's Success Rate falls below a threshold, the deployment can be automatically rolled back.