Service Level Indicator (SLI)

An SLI must be a numerical value derived from observable data, not a subjective opinion. It is calculated from raw telemetry signals like latency histograms, HTTP status codes, or error logs.

Examples include:

• Tool Call Latency: Measured in milliseconds from request initiation to final byte received.
• Success Rate: Calculated as (Successful Calls / Total Calls) * 100.
• Error Rate: The inverse of success rate, focusing on 4xx/5xx HTTP responses or thrown exceptions.

Without a precise, automated measurement, you cannot define a meaningful Service Level Objective (SLO).

An effective SLI measures what the end-user (or the agent acting on the user's behalf) actually experiences. It focuses on the external behavior of the service, not its internal health.

For tool calls, this means:

• Measuring latency from the agent's point of view, including network time.
• Defining 'success' based on the agent receiving a usable, correct response, not just a TCP handshake.
• Avoiding internal metrics like CPU utilization or queue depth, which are leading indicators but not direct measures of user experience.

The core question is: 'Was the tool call fast and successful for the agent executing the task?'

The chosen SLI should correlate with user satisfaction and business outcomes. Monitoring an irrelevant metric provides no actionable signal for reliability engineering.

Key considerations:

• Latency SLIs directly impact agent task completion time and user perceived performance.
• Success Rate SLIs determine whether an agent can complete its intended function or fails mid-execution.
• Poor SLI selection example: Measuring 'API calls per second' when what matters is whether those calls succeed and return correct data.

SLIs should answer the question: 'What matters most to the users of this agentic system?'

An SLI is not a single measurement but an aggregated value over a specific population and time window. The aggregation method must be explicit to avoid ambiguity.

Critical aggregation parameters:

• Time Window: 'Over the last 5 minutes', 'Daily', 'Weekly'.
• Population: 'All POST requests to the /execute endpoint', 'Tool calls from the DataAnalysisAgent'.
• Aggregation Function: 'Average latency', '95th percentile (P95) latency', 'Proportion of successful requests'.

For example: 'The 95th percentile latency for all get_weather tool calls measured over a 1-hour rolling window.'

An SLI should be scoped to a discrete, logical service operation that a user or agent triggers. In tool call instrumentation, this typically maps to a single API endpoint or tool function.

Implementation guidance:

• One SLI per logical operation: calculate_invoice, fetch_customer_record, submit_order.
• Avoid overly broad SLIs: 'Database latency' is too vague; 'Query latency for the transactions table' is actionable.
• Use Span names and attributes from distributed tracing (e.g., OpenTelemetry) to naturally define these operational boundaries.

This scoping allows for precise alerting and debugging when the SLI breaches its target.

The data required to compute the SLI must be technically feasible to collect with high fidelity and minimal performance overhead. If you cannot measure it, it cannot be an SLI.

Requirements for tool calls:

• Automatic Instrumentation: Using frameworks like OpenTelemetry to decorate tool calls with start/end timestamps and result status.
• Low Overhead: Collection must not significantly impact the performance it's trying to measure.
• Reliable Export: Telemetry data must be reliably shipped to a backend system (e.g., Prometheus, Datadog) for aggregation.

Common collection methods include client-side SDKs, service mesh sidecars, or API gateway logs.

An Error Budget quantifies the acceptable amount of unreliability, derived from an SLO. It is calculated as (100% - SLO%) * time_window. If an SLO is 99.9% monthly availability, the error budget is 0.1% downtime (~43 minutes/month). Consuming this budget on failed tool calls or high latency allows for risk-taking in feature development. Exhausting the budget triggers a focus on stability and remediation.

A Service Level Agreement (SLA) is a formal, often commercial, contract between a service provider and customer that includes consequences (like financial penalties) for failing to meet specified SLOs. While an SLO is an internal reliability target, an SLA is an external promise. In agentic observability, SLAs might govern the performance guarantees of a third-party LLM API or tool service that an agent depends upon.

Distributed Tracing is a method for observing requests as they propagate through a distributed system. It is the primary technical mechanism for measuring SLIs like latency across complex agent workflows. A Trace composed of Spans provides the end-to-end context needed to attribute latency or failure to specific tool calls, internal reasoning steps, or external API dependencies.

Golden Signals are four key metrics for monitoring any service: Latency, Traffic, Errors, and Saturation. They provide a foundational set of potential SLIs.

• Latency: Time to serve a request (e.g., tool call).
• Traffic: Demand (e.g., requests per second).
• Errors: Rate of failed requests.
• Saturation: How 'full' a resource is (e.g., queue depth, CPU). For agents, these signals are measured per-tool and per-workflow.

Synthetic Monitoring uses scripted, automated tests (synthetic transactions) to probe a system from the outside, simulating user or agent behavior. It is critical for measuring proactive SLIs like availability and correctness before real users are impacted. For tool call instrumentation, synthetic tests can regularly execute key agent workflows to validate that all external dependencies are responding correctly and within SLO thresholds.