Service Level Objective (SLO) for Latency

The core of a latency SLO is a target percentile (e.g., P95, P99, P99.9) paired with a maximum time value. This defines the performance guarantee for the vast majority of requests, while acknowledging that some outliers will exist.

• Example: "P99 latency < 200ms."
• Rationale: Focusing on high percentiles like P99 ensures a good experience for nearly all users and protects against worst-case scenarios that impact system stability.
• Trade-off: Stricter percentiles (P99.9) are more expensive to meet and monitor than lower ones (P95).

The SLO must define the time period over which compliance is evaluated. This window determines how quickly the system can react to breaches and how much historical data is considered.

• Common Windows: 28 or 30 days are standard for monthly reporting cycles.
• Rolling vs. Calendar: A rolling 30-day window provides a continuously updated view, while a calendar month is simpler for reporting but can mask end-of-month issues.
• Implication: A shorter measurement window (e.g., 1 day) makes the SLO more sensitive to brief incidents but may be too noisy. A longer window provides stability but delays awareness of chronic degradation.

The error budget is the calculated, permissible amount of time a service can violate its SLO within the measurement window. It quantifies reliability risk and drives prioritization decisions.

• Calculation: For a 99.9% monthly SLO, the error budget is 0.1% of the window: 30 days * 24 hours * 0.001 = 43.2 minutes of allowed bad latency per month.
• Management: Exhausting the budget triggers a blameless postmortem and a freeze on new feature releases until reliability is restored.
• Function: It transforms SLOs from abstract goals into a concrete resource for managing the trade-off between innovation velocity and system stability.

The Service Level Indicator (SLI) is the specific, measured metric that feeds into the SLO. For latency, it must be precisely defined to ensure consistent, automated measurement.

• Definition: "The proportion of successful inference requests with an end-to-end latency less than 200ms, measured at the load balancer."
• Key Specifications:
  • Measurement Point: Where is latency measured? (Client-side, load balancer, application server).
  • Request Success Criteria: What constitutes a 'valid' request for the SLI? (Excludes client-canceled requests, includes 4XX errors?).
  • Aggregation Method: How is the percentile calculated? (A true histogram is required for accuracy, not a simple average).

The SLO must explicitly define the scope of the service it covers. This clarifies what components and code paths are included in the latency measurement, preventing ambiguity.

• In-Scope: Core model inference path, including pre/post-processing within the defined service.
• Out-of-Scope: Upstream dependencies (e.g., database calls, external API calls, authentication services) unless they are part of a composite end-to-end SLO.
• Example: An SLO for "Text Completion API v2" applies only to requests routed to that specific API endpoint and model version, not to the health check endpoint or admin APIs.

Burn rate is the speed at which the error budget is being consumed. Configuring alerts based on burn rate, rather than single-point breaches, provides more actionable and timely warnings.

• Fast Burn Alert: Triggers when the budget is being consumed rapidly (e.g., 10% in 1 hour). This signals a severe, ongoing incident requiring immediate attention.
• Slow Burn Alert: Triggers when budget is being consumed steadily over a longer period (e.g., 5% per day). This signals chronic degradation that requires engineering investigation.
• Advantage: This method alerts on the impact (budget loss) rather than the symptom (high latency), reducing alert fatigue and focusing on what matters for SLO compliance.

An Error Budget is the calculated, allowable amount of service unreliability, derived from an SLO. It is defined as 1 - SLO. If the SLO is 99.9% latency compliance, the error budget is 0.1% of requests that can be "slow" without breaching the agreement.

• Purpose: Provides a clear, quantified risk envelope for making engineering trade-offs (e.g., deploying new features vs. maintaining stability).
• Management: Spending the budget triggers a focus on reliability work; conserving it allows for innovation.

A Service Level Agreement (SLA) is a formal, often contractual, commitment between a service provider and a customer that includes consequences (e.g., financial penalties) for failing to meet specified SLOs. The SLO is the internal, engineering-focused target; the SLA is the external, business-facing promise.

• Relationship: SLOs are set more aggressively than SLA targets to provide a safety margin.
• Example: An internal SLO may be P99 latency < 200ms, while the customer-facing SLA promises P99 < 250ms.

Tail Latency refers to the high-percentile response times in a distribution, such as the 95th (P95) or 99th (P99) percentile. While average latency is important, tail latency defines the worst-case user experience and is critical for SLOs, as a few slow requests can breach a target.

• Importance: Averages can mask poor performance; SLOs are almost always defined on tail metrics.
• Cause: Often caused by system noise, garbage collection, queueing delays, or cold starts.

A Throughput-Latency Curve is a graph that plots the relationship between a system's request throughput (e.g., Queries Per Second) and its corresponding latency (often average or tail). It is used to identify the optimal operating point before queuing theory dictates that latency increases exponentially.

• Use Case: Essential for capacity planning and setting realistic SLOs. It shows the maximum sustainable throughput for a given latency target.
• Trade-off: Increasing throughput beyond the "knee" of the curve causes latency to spike, violating SLOs.

Canary Analysis is a deployment and evaluation strategy where a new model version or configuration is released to a small, controlled subset of production traffic. Its latency SLIs are compared in real-time against the stable baseline version to detect regressions before a full rollout.

• Purpose: Proactively validate that changes do not violate latency SLOs.
• Process: If the canary's performance metrics (P99 latency) remain within the error budget, the deployment proceeds.