Service Level Indicator (SLI)

An effective SLI quantifies an aspect of the service as perceived by the end-user or client application. It should measure the outcome of a request, not internal system states.

Examples for Inference:

• Request Latency: Time from request submission to receipt of final response.
• Time to First Token (TTFT): Critical for streaming responses.
• Success Rate: Percentage of requests that complete without error.

Avoid: Internal metrics like GPU utilization or cache hit rate, which are diagnostic but not direct user experience indicators.

The SLI must be expressed as a numerical value derived from well-defined, instrumented data. It should be calculable from request logs or telemetry without ambiguity.

Key Properties:

• Precise Definition: e.g., "Latency is measured from the moment the last byte of the request is received by the load balancer to the moment the first byte of the response is sent."
• Aggregation Method: Clearly defined, such as a percentile (P50, P90, P99) or a ratio (successful requests / total requests).
• Measurable in Production: Requires robust performance telemetry instrumentation across the service stack.

The chosen SLI should map to a critical business outcome or user need. For inference services, this often relates to responsiveness, reliability, or cost-efficiency.

Inference-Specific Alignments:

• Latency SLIs align with user satisfaction for interactive applications (e.g., chatbots).
• Throughput (QPS/TPS) SLIs align with batch processing capacity and infrastructure cost control.
• Error Rate SLIs align with service reliability and trust.

The SLI provides the empirical basis for defining a Service Level Objective (SLO), which is the target threshold for the SLI.

A good SLI acts as a leading indicator that reacts predictably to changes in the system, whether from deployments, load increases, or infrastructure issues.

Characteristics:

• Detects Regressions: A performance regression from a model update or configuration change should be visible in the SLI trend (e.g., increased P99 latency).
• Responds to Load: The SLI should reflect system behavior under different load conditions, as visualized on a throughput-latency curve.
• Avoids Noise: While sensitive, it should not be overly volatile due to insignificant fluctuations, requiring careful selection of measurement windows and aggregation.

The SLI should be easily comprehensible by developers, operators, and business stakeholders. Complexity hinders adoption, debugging, and decision-making.

Best Practices:

• Limit Quantity: Focus on 2-4 key SLIs per critical service component (e.g., one for latency, one for availability).
• Clear Naming: Use intuitive names like inference_latency_seconds or request_success_rate.
• Avoid Composite Metrics: While powerful, ratios like "availability" (success rate) are acceptable. Avoid creating complex, weighted formulas that obscure root causes.

The ultimate purpose of an SLI is to evaluate compliance with a Service Level Objective (SLO). An effective SLI makes setting and monitoring SLOs practical.

The SLO Relationship:

• The SLI is the measured metric (e.g., latency distribution).
• The SLO is the target threshold for that metric (e.g., P99 latency < 500ms over a 30-day window).
• This pairing drives operational decisions: breaching the SLO triggers alerting, error budget consumption, and investigation via bottleneck analysis and performance profiling.

A Service Level Agreement is a formal contract between a service provider and a customer that specifies the guaranteed level of service, often with associated business consequences like financial credits for violations. Key components include:

• The SLOs that define the service guarantees.
• The measurement methods and reporting periods.
• The remedies and penalties for breaching the SLOs. While SLIs and SLOs are internal engineering tools, the SLA is the external, business-facing commitment derived from them.

An Error Budget is the explicit, quantified amount of unreliability a service can tolerate within an SLO measurement period before violating its SLO. It is calculated as 1 - SLO. For example, a 99.9% availability SLO over a month allows for an error budget of 0.1% downtime, or approximately 43 minutes and 12 seconds. This concept:

• Transforms SLOs from opaque targets into a consumable resource for balancing reliability with innovation.
• Guides decision-making: Teams can spend the budget on risky deployments or must halt changes if the budget is exhausted.

Percentile Latency is a statistical measure of request latency distribution, crucial for defining SLIs that reflect user experience. It is not a single number but a spectrum:

• P50 (Median): The latency at which 50% of requests are faster and 50% are slower. Represents the typical experience.
• P90: The latency at the 90th percentile. 90% of requests are faster than this value.
• P99: The latency at the 99th percentile, representing the tail latency experienced by the slowest 1% of requests. SLOs for user-facing services are almost always defined using high percentiles (P95, P99) to ensure quality for all users, not just the average case.

Throughput is the rate at which a system processes work, serving as a critical SLI for capacity and scalability. Common metrics include:

• Queries Per Second (QPS): The number of complete inference requests processed per second.
• Tokens Per Second (TPS): For language models, the total number of output tokens generated per second across all requests. Throughput SLIs are used to define SLOs for system capacity (e.g., "maintain 1000 QPS under peak load") and are intrinsically linked to latency—as throughput approaches a system's saturation point, latency typically increases.

Availability measures the proportion of time a service is operational and able to handle requests, while Error Rate measures the proportion of requests that fail. They are fundamental SLIs for service health.

• Availability is often calculated as (Successful Requests / Total Requests) * 100% over a time window.
• Error Rate typically includes HTTP 5xx server errors, timeouts, and application-level failures. An SLO might state: "The service will have 99.95% availability (error rate ≤ 0.05%) over a rolling 30-day period." These SLIs are directly tied to user-visible service uptime.