Error Rate

Error Rate is formally defined as the ratio of failed invocations to the total number of invocations over a specified time window. It is typically expressed as a percentage.

• Formula: (Number of Failed Calls / Total Calls) * 100
• Failure Criteria: A call is generally counted as a failure if it returns a non-successful HTTP status code (e.g., 4xx or 5xx), throws an unhandled exception, or exceeds a configured timeout threshold without a response.
• Time Window: Calculated over rolling periods (e.g., last 1 minute, 5 minutes) to provide real-time and historical views of system health.

Error Rate aggregates several distinct types of failures, each with different root causes and implications for system design and resilience patterns.

• Client Errors (4xx): Failures like 400 Bad Request or 429 Too Many Requests often indicate issues with the agent's request formulation, authentication, or adherence to rate limit telemetry.
• Server Errors (5xx): Errors like 500 Internal Server Error or 503 Service Unavailable signal problems within the external dependency itself.
• Network Failures: Timeouts, connection resets, and DNS failures occur at the transport layer, often requiring retry policies with exponential backoff.
• Business Logic Errors: An API may return a 200 OK with an error payload, which requires parsing response content to accurately classify the call.

Error Rate does not exist in isolation; it is a core Service Level Indicator (SLI) that interacts with other key performance indicators to define overall system reliability.

• Success Rate: The inverse of Error Rate (Success Rate = 100% - Error Rate). Both are used to define Service Level Objectives (SLOs).
• Latency: High error rates can correlate with elevated P95 latency as systems spend time on failing requests or retries.
• Error Budget: The allowable amount of unreliability derived from an SLO. A sustained high Error Rate consumes the error budget, triggering operational reviews and freezing risky deployments.
• Dependency Tracking: Error Rates are tracked per external service, enabling targeted remediation of problematic dependencies.

Accurate Error Rate measurement requires comprehensive tool call instrumentation to capture the full context of each failure.

• Span Attributes: Failed calls should have spans tagged with error status (error=true) and detailed attributes like http.status_code, error.type, and error.message.
• Span Events: Log significant failure moments (e.g., retry.attempted, circuit_breaker.opened) as span events to understand the failure lifecycle.
• Metric Generation: Emit a counter metric (e.g., tool.calls.errors) with dimensions for tool name, error type, and team via cost attribution tags.
• Trace Correlation: Use distributed tracing to see how a single failing tool call impacts the broader agent reasoning trace.

Monitoring Error Rate drives critical engineering decisions around system design, deployment, and incident response.

• Resilience Patterns: High error rates necessitate implementing the circuit breaker pattern to fail fast and prevent cascading failures.
• Deployment Safety: Canary deployments of new agent logic or tool integrations closely monitor Error Rate for regressions before full rollout.
• Alerting and SLOs: Error Rate is a primary signal for alerting when it breaches SLO thresholds, prompting immediate investigation.
• Capacity Planning: Persistent errors from a dependency may indicate the need for alternative providers, client-side caching, or queueing via a dead letter queue (DLQ) for later replay.

A spike in Error Rate is a starting point for investigation, requiring drill-down into specific failure signatures and traces.

• Error Grouping: Aggregate errors by tool, endpoint, status code, and execution context ID to identify patterns.
• Root Cause Analysis: Use correlated traces to examine the exact parameters, timing, and sequence of events leading to a failure.
• Temporal Analysis: Compare Error Rate with other metrics like call volume and payload size to identify correlations (e.g., errors spike with larger requests).
• Proactive Monitoring: Use synthetic transactions to probe critical tool dependencies from outside the production network, detecting errors before users or agents are impacted.

Success Rate is the inverse of Error Rate, representing the ratio of successful tool or API invocations to total invocations. It is a direct Service Level Indicator (SLI) for reliability.

• Calculated as: (Successful Calls / Total Calls) * 100%.
• A 99.9% success rate implies a 0.1% error rate.
• Often used in Service Level Objectives (SLOs) to define reliability targets for external dependencies.

A Service Level Objective (SLO) is a target value or range for a Service Level Indicator (SLI), such as Error Rate or Success Rate. It forms a reliability contract for tool-calling systems.

• Example: "Tool call success rate must be ≥ 99.5% over a 30-day rolling window."
• Error Budgets are derived from SLOs, defining the allowable amount of unreliability.
• Breaching an SLO triggers operational focus on improving resilience or dependency health.

The Circuit Breaker Pattern is a resilience design pattern that prevents cascading failures by programmatically failing fast when calls to a tool or service are likely to fail, based on recent error rates.

• Monitors failure counts or error rate thresholds.
• Trips from CLOSED (normal operation) to OPEN (failing fast), then to HALF-OPEN (testing for recovery).
• Protects the agent from waiting on timeouts and conserves system resources during dependency outages.

A Retry Policy defines rules for automatically re-attempting failed tool calls. Exponential Backoff is a common strategy where wait times between retries increase exponentially.

• Policies specify conditions for retry (e.g., on HTTP 5xx errors, timeouts), max attempts, and backoff logic.
• Exponential Backoff (e.g., 1s, 2s, 4s, 8s) reduces load on a failing service.
• Must be combined with idempotency keys for APIs where retries could cause duplicate side effects.

Span Events are structured, timestamped log records attached to a tracing Span. They are used to record significant moments during a tool call's execution, including errors.

• Critical for debugging: Events can log exception.thrown, retry.initiated, or circuit.breaker.opened.
• Carry structured attributes like error message, stack trace, and HTTP status code.
• Provide a detailed, chronological narrative within the span's duration for root cause analysis.

Anomaly Detection in tool call instrumentation uses statistical or machine learning models to identify unexpected deviations in metrics like Error Rate, signaling potential issues before they breach SLOs.

• Models establish a baseline of normal error rate patterns (e.g., time-of-day variations).
• Flags statistically significant spikes or trend changes in failures.
• Enables proactive alerting on emerging dependency problems or novel failure modes.