Service Level Objective (SLO)

The Service Level Indicator (SLI) is the precise, measurable metric that quantifies a specific aspect of service health. It is the raw measurement upon which an SLO is based. For LLM inference, common SLIs include:

• Availability: The proportion of successful requests (e.g., HTTP 200 responses) over total requests.
• Latency: The time taken to return a complete response (p95 or p99 tail latency is critical for user experience).
• Throughput: The number of requests processed per second, often measured in Tokens Per Second (TPS).
• Output Quality: Measured via metrics like correctness score or hallucination rate, though these require specialized evaluation pipelines.

The SLO Target is the numerical goal for the SLI, expressed as a threshold or percentage (e.g., '99.9% availability' or 'p95 latency < 2 seconds'). The Error Budget is derived directly from this target; it is the allowable amount of unreliability, calculated as 1 - SLO Target. For a 99.9% monthly availability SLO, the error budget is 0.1%, or approximately 43.2 minutes of downtime per month. This budget is a crucial resource for balancing reliability with innovation—teams can spend it on risky deployments or must halt releases when it's depleted.

The Measurement Window defines the period over which the SLI is evaluated against the SLO target (e.g., rolling 28 days, calendar month). This choice impacts the stability and responsiveness of SLO compliance signals. Aggregation refers to how raw measurements are rolled up into the single SLI value for the window. Common methods include:

• Average: Simple mean (can mask tail problems).
• Percentile (p95, p99): Focuses on the worst-case user experiences, essential for latency SLOs.
• Binomial: For availability, a simple success/failure ratio. Consistent, automated aggregation is non-negotiable for valid SLOs.

SLOs for LLM serving must account for unique operational characteristics beyond standard web services:

• Token-Based Latency: Latency is not request-based but token-based. SLOs must consider Time To First Token (TTFT) and Time Per Output Token (TPOT) separately.
• Non-Deterministic Failures: A model can return an HTTP 200 but produce a hallucination or unsafe content. Quality SLOs require integrating output validation systems.
• GPU-Specific Metrics: SLIs like GPU Memory Utilization and KV Cache Hit Rate are leading indicators for latency and availability problems.
• Cost-Per-Request: While not a traditional SLO, establishing a cost efficiency target (e.g., cost per 1k tokens) is critical for financial governance.

Effective SLOs drive alerting on burnout, not on individual errors. Burn Rate measures how quickly the error budget is being consumed. A fast burn alert (e.g., consuming 10% of the budget in 1 hour) triggers an immediate incident response. A slow burn alert (e.g., consuming 2% of the budget over a day) triggers a lower-priority review. This approach, aligned with Multi-Window, Multi-Burn-Rate alerting strategies, reduces alert fatigue by focusing on threats to the SLO agreement, not transient blips.

An SLO is not a standalone document; it must be integrated into the engineering workflow. This involves:

• Canary Deployment & SLO Validation: Using SLO compliance as a gating metric for automated canary analysis before full rollout.
• Observability Dashboards: Real-time dashboards showing SLI measurements, error budget remaining, and burn rates.
• Incident Post-Mortems: Analyzing error budget expenditure to understand root causes.
• Capacity Planning: Using historical SLO trends to forecast infrastructure needs. Tools like Prometheus for metrics and Sloth for SLO generation are common in this ecosystem.

A Service Level Indicator (SLI) is a quantitative measure of a specific aspect of a service's performance, forming the raw data from which SLOs are derived. It is a carefully defined metric that directly reflects user experience.

• Examples: Request latency (p99), error rate (successful requests / total requests), availability (uptime), and throughput (requests per second).
• Key Principle: An SLI should measure what users actually care about, not just internal system health. For an LLM API, critical SLIs include token generation latency, successful completion rate of prompts, and the rate of valid, non-hallucinatory responses.

A Service Level Agreement (SLA) is a formal contract between a service provider and a customer that stipulates the consequences, typically financial penalties or service credits, if the provider fails to meet the committed Service Level Objectives (SLOs).

• Relationship to SLO: An SLO is an internal, aspirational target. An SLA is the external, contractual promise derived from one or more SLOs, often with a slightly less aggressive target to provide a safety buffer.
• Enforcement: SLAs are legally binding and define remediation procedures. For a paid LLM API, an SLA might guarantee 99.9% monthly uptime, with service credits issued for any violation.

An Error Budget is the explicit, quantified amount of unreliability a service can tolerate within a defined period (e.g., a month) before violating its SLO. It is calculated as 1 - SLO.

• Purpose: It operationalizes SLOs by creating a shared resource for balancing reliability against feature velocity. Teams can "spend" the budget on deployments and changes, but must halt feature work to focus on stability if the budget is exhausted.
• Example: A 99.9% monthly availability SLO for an LLM endpoint translates to an error budget of 0.1% downtime, or approximately 43 minutes and 48 seconds per month. This budget can be consumed by failed deployments, infrastructure outages, or traffic spikes.

SLO Burn Rate measures how quickly a service is consuming its error budget. It is a critical alerting metric that indicates the severity and urgency of an ongoing reliability issue.

• Calculation: It expresses the error budget consumption rate, often as a multiple of the budget that would be consumed if the issue continued for the full SLO period. A burn rate of 1.0 consumes the budget evenly over the period; a burn rate of 10.0 consumes it ten times faster.
• Alerting Strategy: High burn rates over short windows trigger urgent, page-worthy alerts. Lower burn rates over longer windows trigger lower-priority tickets. This prevents alert fatigue while ensuring critical SLO breaches are caught early.

Multi-Window, Multi-Burn-Rate Alerting is a sophisticated SLO-based alerting strategy that triggers alerts based on the error budget burn rate across different time windows (e.g., 1 hour, 6 hours, 3 days).

• Rationale: A short, severe outage and a long, slow degradation are both threats to an SLO but require different responses. This strategy captures both.
• Implementation: It defines alerting rules like:
  • Page Alert: Burn rate > 14.4 for 1 hour (budget exhausted in 5 minutes).
  • Ticket Alert: Burn rate > 6 for 6 hours (budget exhausted in 1 day).
  • Ticket Alert: Burn rate > 1 for 3 days (budget exhausted in 3 days). This approach ensures the right level of operational response for each incident profile.

Latency SLOs for LLMs are Service Level Objectives specifically targeting the time taken to generate a response, which is a paramount user experience metric for interactive applications. Defining them requires careful statistical consideration.

• Challenge: LLM latency has high variance due to variable output length, computational complexity, and system load. Using simple averages is misleading.
• Best Practice: Define SLOs using high-percentile measurements (e.g., p95 or p99 latency). For example: "95% of all chat completion requests for outputs up to 500 tokens must return within 2 seconds."
• Components: LLM latency SLOs often segment Time to First Token (TTFT) for perceived responsiveness and Time Per Output Token (TPOT) for generation speed, each with its own target.