Service Level Indicator (SLI)

An effective SLI must be a quantifiable metric derived from observable system data, not an opinion or estimate. It is calculated from raw telemetry like logs, metrics, or tracing data.

• Examples: Model inference latency (p95), successful request rate, token generation throughput.
• Non-Examples: "User happiness," "system health," or "model seems accurate."

The measurement must be automatable and reproducible, forming an objective basis for evaluating an SLO.

The SLI should measure an aspect of the service that directly impacts the end-user's experience or business outcome. It focuses on the external behavior of the service, not internal operational details.

• User-Facing: Measures what the user perceives (e.g., end-to-end API latency for a chat completion).

• Journey-Based: Often aligns with a Critical User Journey (CUJ), such as "time to receive a complete answer from the RAG system."

Internal metrics like GPU utilization are important for diagnostics but are poor SLIs, as users are unaffected by high utilization if latency remains low.

The precise method of calculation and the time period of evaluation must be unambiguous. This defines what is being measured and over what duration.

• Aggregation: Specify the statistical method (e.g., average, 95th percentile, ratio of counts). For latency, percentiles (p95, p99) are crucial to track tail performance.

• Time Window: Define the rolling window for calculation (e.g., "the ratio of successful requests over the past 30 days"). This aligns with the SLO's compliance period and enables burn rate analysis.

Example: p95(model_inference_latency) over a 1-minute rolling window.

For AI-powered services, SLIs must capture the unique quality and performance dimensions of machine learning inference and data pipelines, beyond traditional HTTP metrics.

• Quality SLIs: Hallucination rate, answer faithfulness score, retrieval precision@K.

• Performance SLIs: Time To First Token (TTFT), Time Per Output Token (TPOT), end-to-end RAG pipeline latency.

• Data SLIs: Input data drift magnitude, training-serving skew detection.

These specialized indicators are essential for defining meaningful SLOs for AI systems.

The engineering team must have clear levers to influence the SLI. If the metric degrades, there should be known procedures or system changes that can improve it.

• Actionable Example: High p99 latency can be addressed by optimizing model inference (e.g., enabling continuous batching), scaling resources, or simplifying a feature.

• Non-Actionable Example: An SLI based on global internet latency is not controllable by the service team.

This characteristic ensures SLIs drive meaningful engineering and operational improvements.

An SLI is not defined in isolation; it is the measurable component of a Service Level Objective (SLO). The SLO sets the target for the SLI (e.g., "p95 latency < 500ms for 99.9% of requests this quarter").

• Error Budget Derivation: The SLI's measurement directly calculates the error budget consumption.

• Business Correlation: The best SLIs have a demonstrable link to business metrics like user retention, conversion rate, or revenue. For instance, high latency (an SLI) may correlate with cart abandonment (a business KPI).

This alignment turns technical metrics into drivers of business reliability.

A Service Level Objective (SLO) is the quantitative target set for a Service Level Indicator (SLI). It defines the acceptable level of service, typically expressed as a percentage over a rolling time window (e.g., "99.9% of requests must have latency < 100ms over 30 days"). For AI services, SLOs apply to metrics like inference latency, error rate, or output quality. An SLO is an internal goal that balances reliability with innovation velocity, creating a formal error budget for the engineering team.

An error budget is the explicit, allowable amount of unreliability a service can incur without violating its Service Level Objective (SLO). It is calculated as 100% - SLO. For example, a 99.9% SLO creates a 0.1% error budget. This budget quantifies risk, enabling teams to make data-driven decisions about deploying new features, taking on technical debt, or performing risky migrations. Once the budget is exhausted, the focus must shift to improving reliability before further feature development.

A Service Level Agreement (SLA) is a formal contract with external customers that includes one or more Service Level Objectives (SLOs) and defines the business consequences—such as service credits or financial penalties—if those SLOs are not met. While an SLO is an internal engineering target, an SLA is an external promise. SLAs for AI services must be carefully scoped, as factors like unpredictable model behavior or upstream data provider issues can increase the risk of breach.

A golden signal is one of four high-level metrics used in Site Reliability Engineering (SRE) to comprehensively assess a service's health. They are:

• Latency: Time to serve a request.
• Traffic: Demand on the system (e.g., queries per second).
• Errors: Rate of failed requests.
• Saturation: How "full" a resource is (e.g., GPU memory utilization). For AI services, these signals are foundational SLIs. Monitoring them provides a quick, holistic view of system performance and is the first step toward defining precise SLOs.

A Critical User Journey (CUJ) is a specific, high-value sequence of user interactions that is essential to the core value of a service (e.g., "user uploads a document, asks a question, and receives a summarized answer"). SLIs and SLOs should be derived from CUJs to ensure reliability targets align with user experience, not just backend system metrics. For AI services, this means measuring end-to-end latency and success rate for the complete journey, which may involve multiple model calls and retrieval steps.

Burn rate is the speed at which a service consumes its error budget. It is calculated as the percentage of the budget used per unit of time (e.g., 20% of the monthly budget burned per hour). Monitoring burn rate is crucial for multi-window alerting. A high burn rate over a short window (e.g., 5 minutes) signals a potential incident, while a sustained moderate burn rate over a longer window (e.g., 6 hours) indicates a chronic issue that will violate the SLO if unaddressed.