Service Level Objective (SLO)

The Service Level Indicator (SLI) is the quantitative measure of a specific aspect of service performance that the SLO is based on. It is the raw metric that is tracked.

For inference systems, common SLIs include:

• Latency: P50, P90, P99, or P99.9 request latency.
• Availability: The proportion of successful requests (e.g., HTTP 200 responses).
• Throughput: Queries Per Second (QPS) or Tokens Per Second (TPS) maintained.
• Quality: For generative models, this could be a score from an evaluation model or a human rating system.

An SLI must be precisely defined, including how it is measured, aggregated, and any relevant conditions (e.g., 'P99 latency for requests under 1000 input tokens').

The target is the specific value or threshold the SLI must meet to satisfy the SLO. It defines the acceptable level of service.

Targets are typically expressed as:

• A percentage over a window: '99.9% of requests must have latency < 200ms.'
• A statistical bound: 'P99 latency must be < 500ms.'

Key considerations for setting targets:

• Realism: The target must be achievable given system capabilities and cost constraints.
• User impact: It should align with user experience (e.g., tail latency P99 is often more critical than median P50).
• Business requirements: It should support product or contractual needs. A target that is too loose fails to guarantee quality; one that is too tight leads to excessive cost and operational burden.

The measurement window defines the period over which compliance with the SLO is evaluated. The compliance is the calculated result of whether the SLI met its target within that window.

Common windows:

• Rolling windows: e.g., 'over any 30-day rolling period.'
• Calendar-aligned windows: e.g., 'per calendar month.'

Calculating Compliance (Error Budget): SLOs are often managed via an error budget—the allowable amount of service degradation. If the SLO is '99.9% availability over 30 days,' the error budget is 0.1% of that time: 43.2 minutes (0.001 * 30 days * 24 hours * 60 minutes).

• Consumed budget: Time spent outside the SLO target.
• Remaining budget: The error budget minus consumed budget. This quantifies risk and guides operational priorities (e.g., launching risky changes vs. focusing on stability).

For inference systems, latency SLOs are critical for user experience. They require careful definition of the SLI to account for different phases of generation.

Key Latency SLIs:

• Time to First Token (TTFT) SLO: 'P95 TTFT < 1.5 seconds for prompts under 4k tokens.' Governs perceived responsiveness.
• Time per Output Token (TPOT) SLO: 'P99 TPOT < 75 ms.' Governs the smoothness of streaming outputs.
• End-to-End Latency SLO: 'P90 total request time < 10 seconds for requests generating up to 500 tokens.' A composite measure.

Implementation Challenges:

• Variable Load: Latency degrades under load; SLOs must be defined for expected load ranges.
• Request Heterogeneity: Different input/output lengths cause wide latency distributions. SLOs may need to be segmented (e.g., one SLO for short conversations, another for long document analysis).

While often secondary to latency, throughput and cost SLOs are essential for managing infrastructure efficiency and economics.

Throughput SLO Examples:

• Minimum Sustained Throughput: 'The service shall maintain at least 500 QPS under a sustained load with P99 latency < 300ms.'
• Cost-per-Request SLO: 'The average compute cost per 1k output tokens shall not exceed $0.002.'

Relating to Other Metrics: Throughput and cost SLOs exist in tension with latency SLOs. The throughput-latency curve illustrates this trade-off. An SLO might define an operating point: 'Achieve 1000 TPS while maintaining P99 latency < 200ms.'

Resource Utilization as a Proxy: SLOs may indirectly target hardware utilization (e.g., 'GPU utilization shall be >40% during peak hours') to ensure cost-effective resource usage without violating latency targets.

An SLO operates within a hierarchy of service level definitions and is a tool for engineering decision-making.

The Hierarchy:

1. Service Level Agreement (SLA): A formal contract with users containing consequences (e.g., credits) for breach. It is based on one or more SLOs.
2. Service Level Objective (SLO): The internal, engineering-focused target.
3. Service Level Indicator (SLI): The measured metric.

Using Error Budgets for Management: The error budget derived from an SLO (e.g., 43 minutes of downtime per month) becomes a central management tool.

• Burn Rate: How quickly the error budget is being consumed. A high burn rate triggers alerts.
• Decision Framework: If the budget is healthy, teams can deploy new features or take risks. If the budget is nearly exhausted, the focus must shift to stability and remediation.
• This creates a data-driven feedback loop between reliability, product velocity, and user experience.

A Service Level Indicator is the quantitative measurement of a specific aspect of service performance that forms the basis for an SLO. It is the raw metric, while the SLO is the target value for that metric.

• Examples: Average request latency, error rate (e.g., 5xx HTTP errors), throughput (queries per second), availability (uptime percentage).
• Relationship to SLO: An SLO is a target for an SLI. For instance, an SLI might be "P99 latency," and the corresponding SLO is "P99 latency < 200ms."

Tail latency refers to the high-percentile latencies in a request distribution, representing the slowest requests. It is critical for SLOs because user experience is often defined by the worst-case performance, not the average.

• P99 Latency: The latency value at the 99th percentile; 99% of requests are faster than this value. A common SLO target (e.g., P99 latency < 300ms).
• Importance: Averages can mask bad experiences. A fast P50 (median) but high P99 latency means 1% of users have a poor experience, which may violate SLOs.
• Measurement: Requires detailed instrumentation and histogram-based metrics, not just averages.

An error budget is the explicit, quantified amount of unreliability a service can tolerate over a defined period while still meeting its SLOs. It operationalizes SLOs by framing them as a consumable resource.

• Calculation: If an SLO is "99.9% availability per month," the error budget is 0.1% downtime, or approximately 43.2 minutes per month.
• Use Case: Guides deployment velocity and risk-taking. Consuming the budget triggers a freeze on risky changes to focus on stability.
• Philosophy: Shifts the discussion from "perfectly reliable" to "how reliable do we need to be," enabling data-driven engineering trade-offs.

Throughput is the rate at which a system processes work, measured in Queries Per Second (QPS) or Tokens Per Second (TPS) for LLMs. It is a key capacity metric often linked to latency SLOs via the throughput-latency curve.

• QPS (Queries Per Second): Measures complete requests processed per second.
• TPS (Tokens Per Second): For generative models, measures output tokens generated per second.
• Relationship to Latency SLOs: As throughput increases toward system capacity, latency typically increases. An SLO defines the maximum acceptable latency at a target throughput level.

A Service Level Agreement is a formal contract between a service provider and a customer that includes consequences (usually financial penalties) for failing to meet the specified Service Level Objectives.

• Key Difference from SLO: An SLO is an internal, aspirational engineering target. An SLA is an external, contractual promise with business ramifications.
• Typical Structure: An SLA will reference one or more SLOs (e.g., "Monthly Uptime Percentage of 99.9%") and define the remedies (e.g., service credits) if they are not met.
• Internal SLAs: Used between internal teams to formalize dependencies and expectations.

A performance baseline is a set of established reference metrics for a system under controlled conditions. Performance regression is an undesirable deviation from this baseline, directly threatening SLO compliance.

• Establishing a Baseline: Done via controlled load testing, measuring key SLIs (latency, throughput) before major changes.
• Detecting Regression: Continuous performance testing and canary deployments compare new builds against the baseline to catch SLO violations before they reach production.
• Root Cause: Regressions can be caused by code changes, model updates, configuration drift, or underlying infrastructure changes.