Service Level Indicator (SLI)

An effective SLI must be expressed as a numerical quantity derived from observable system data. It cannot be a subjective judgment. For LLMs, this typically involves:

• Latency metrics: Time to First Token (TTFT), Inter-Token Latency, end-to-end request duration.
• Throughput metrics: Tokens per Second (TPS), successful requests per minute.
• Quality metrics: Perplexity scores, hallucination rates (though these require careful measurement).
• Availability metrics: Uptime percentage, error rate (4xx/5xx responses). The measurement must be automatable via telemetry systems like OpenTelemetry or Prometheus.

The SLI should measure an aspect of the service that directly impacts the end-user's experience or the business outcome. Avoid proxy metrics that are easy to measure but not felt by users.

Good Examples:

• P99 latency for chat completions (users feel slow responses).
• Error rate for API requests (users get failed interactions).
• Token throughput for a summarization feature (users wait for the result).

Poor Examples:

• GPU utilization percentage (infrastructure concern, not user-facing).
• Cache hit rate (an internal optimization, not a user outcome).

The performance of the SLI should be primarily influenced by engineering decisions and system changes within the team's purview. If an SLI is affected by external factors the team cannot mitigate, it fails as a useful indicator.

For LLM services, this means:

• Model serving infrastructure choices (batching, hardware) affect latency (TTFT, TPS).
• Application code and prompt engineering affect error rates and output validity.
• System architecture (caching, load balancing) affects availability.

An SLI like "third-party API latency" is not controllable if the dependency is external.

An SLI is meaningless without a target. It must have a corresponding Service Level Objective (SLO)—a target value or range that defines acceptable performance. The SLO provides the context for whether the SLI's current value is good or bad.

Example Pairing:

• SLI: End-to-end latency for a text generation endpoint.
• SLO: 95% of requests complete within 2 seconds over a 28-day window.

The SLO, derived from the SLI's measurement, creates an error budget that guides deployment velocity and prioritization of reliability work.

SLI measurement must be consistent and comparable over time. This requires:

• A stable collection methodology (e.g., always measured at the API gateway).
• A defined aggregation window (e.g., rolling 28 days, daily) for assessment against SLOs.
• Clear aggregation rules (e.g., is the SLI a percentile, a mean, a ratio?).

For LLM latency, this often means tracking latency percentiles (P50, P90, P99) over a rolling window to understand both typical and tail performance. Inconsistency in measurement invalidates trend analysis and SLO compliance tracking.

A service should have a small, focused set of SLIs (typically 2-5) that capture its core reliability promises. Too many SLIs create noise and dilute focus. The goal is to identify the vital few metrics that truly indicate service health.

For a core LLM inference API, essential SLIs might be:

1. Availability: Successful request ratio.
2. Latency: P99 request duration.
3. Throughput: Sustained Tokens per Second (for cost/performance).

Additional quality SLIs (e.g., for hallucination rate) may be added for specific, high-stakes use cases but should not overwhelm the core set.

An Error Budget quantifies the allowable unreliability for a service over a specific period, derived directly from its SLO. It is calculated as 1 - SLO.

• Example: A 99.9% monthly availability SLO permits an error budget of 0.1% downtime, or approximately 43.2 minutes per month.
• This budget governs the pace of innovation and risk-taking; spending it on planned releases is acceptable, but unplanned incidents consume it.
• Exhausting the budget typically triggers a blameless postmortem and a freeze on new feature deployments.

A Service Level Agreement is a formal contract between a service provider and its customers that includes one or more SLOs, along with consequences (e.g., financial penalties) for failing to meet them.

• Key Distinction: An SLA is an external, commercial contract. An SLO is an internal, engineering target. An SLI is the measured metric.
• SLAs are typically less aggressive than internal SLOs to provide a safety buffer.
• For internal LLM platforms, the "customer" may be another engineering team, governed by an Internal SLA.

A Golden Dataset is a curated, high-quality, and versioned set of input-output pairs used as a reference standard for evaluating LLM performance and detecting regressions.

• It serves as the ground truth for monitoring Output Drift and Concept Drift.
• Used in canary and shadow deployments to compare new model versions against a baseline.
• Should be representative of critical production traffic and include edge cases. Maintaining its relevance over time is a key challenge.

Statistical Process Control is a method of quality control that uses statistical tools, primarily control charts, to monitor a process and detect abnormal variation.

• Applied to SLIs to distinguish between common-cause variation (normal noise) and special-cause variation (indicative of a problem).
• Control limits (e.g., 3-sigma) are calculated from historical SLI data. Data points outside these limits trigger alerts for Anomaly Detection.
• This provides a more robust alerting mechanism than simple static thresholds.

Mean Time to Recovery is a key reliability metric measuring the average time taken to restore a service to normal operation after a failure or significant degradation is detected.

• Components of MTTR: Detection Time + Diagnosis Time + Mitigation Time + Restoration Time.
• A primary goal of SRE is to minimize MTTR through effective monitoring (SLIs), automation, and clear runbooks.
• While not an SLI itself, MTTR is often tracked as a supporting metric for the overall health of an operations team.