Service Level Objective (SLO)

An SLO for model inference latency defines the maximum acceptable time for a model to return a prediction from the moment a request is received. This is critical for real-time applications like fraud detection or autonomous systems.

• Example SLO: "99% of inference requests for the recommendation model must complete within 100 milliseconds over a 30-day window."
• Measurement: P95 or P99 latency percentiles are common metrics.
• Impact: Breaching this SLO directly degrades user experience and can cause cascading failures in downstream services.

This SLO measures the proportion of time the model inference endpoint is operational and able to serve requests. It encompasses the health of the inference server, network, and dependent services.

• Example SLO: "The vision model API must have 99.9% availability per calendar quarter."
• Error Budget: A 99.9% SLO allows for ~43.2 minutes of downtime per month. This budget can be spent on deployments, experiments, or unexpected outages.
• Dependencies: This SLO is often tied to the availability of underlying hardware (NPUs/GPUs), model registries, and feature stores.

For continuous training or retraining systems, an SLO can define the reliability of the ML pipeline itself. This measures the success rate of automated training jobs from data ingestion to model registration.

• Example SLO: "95% of nightly retraining pipelines must complete successfully over a rolling 28-day period."
• Scope: Includes data validation, feature engineering, model training, evaluation, and registry upload stages.
• Importance: A low success rate indicates fragile data pipelines or unstable training code, risking model staleness and operational toil.

While traditional SLOs focus on service metrics, ML systems require SLOs for predictive performance. These can be defined as thresholds for model accuracy, precision/recall, or metrics tracking data drift.

• Example SLO: "The F1-score of the production classifier, measured on a daily sampled holdback dataset, must not degrade by more than 5 percentage points from the champion model's baseline."
• Example SLO for Drift: "The population stability index (PSI) for the primary input feature distribution must remain below 0.1 on a weekly basis."
• Challenges: Requires robust ML observability and monitoring to measure these metrics in production.

In real-time inference systems, predictions depend on online feature stores. An SLO for feature freshness defines the maximum acceptable delay between a real-world event and its feature value being available for inference.

• Example SLO: "99% of user-activity features must be available for lookup within 2 seconds of the activity event."
• Critical for: Recommendation systems, dynamic pricing, and fraud detection where context is time-sensitive.
• Architecture Impact: This SLO dictates choices between batch, streaming, or lambda architecture for feature computation.

A Service Level Agreement (SLA) is a formal contract between a service provider and a customer that defines the expected level of service. It is the business and legal wrapper that contains one or more Service Level Objectives (SLOs).

• SLOs are the measurable targets within an SLA (e.g., "99.9% availability").
• The SLA defines the consequences (e.g., service credits, penalties) for breaching the SLOs.
• An SLA is externally focused on the customer relationship, while SLOs are internally focused engineering targets.

A Service Level Indicator (SLI) is the specific, raw measurement of a service's behavior that an SLO targets. It is the foundational metric from which SLO compliance is calculated.

• Examples: Request latency (p99), error rate (successful requests / total requests), availability (uptime / total time), throughput.
• An SLI must be a quantifiable measure that accurately reflects user experience.
• The process is: Define an SLI (measure), set an SLO (target), and enforce an SLA (agreement).

An Error Budget is the calculated amount of unreliability a service can tolerate over a defined period while still meeting its SLO. It is derived directly from the SLO.

• Calculation: If an SLO is "99.9% availability per month," the error budget is 0.1% of unavailability, or approximately 43.2 minutes per month.
• This budget creates a shared, objective framework for balancing reliability work (spending the budget to fix issues) against feature velocity (spending the budget to launch new, potentially risky changes).
• Exhausting the error budget should trigger a freeze on new feature launches to focus on stability.

Quality of Service (QoS) is a broader, more general term for the overall performance characteristics of a service or network. While SLOs are specific, measurable targets for reliability, QoS encompasses the techniques and policies used to manage resources to achieve desired performance levels.

• QoS is often about resource management: Prioritizing traffic, bandwidth reservation, and packet scheduling to meet latency/throughput goals.
• SLOs are the goals, while QoS mechanisms (like traffic shaping) are one of the tools used to achieve them.
• In networking, QoS is critical for meeting SLOs for latency-sensitive applications like VoIP or real-time inference.

Observability is the property of a system that allows its internal state to be inferred from its external outputs (logs, metrics, traces). Telemetry is the automated collection and transmission of this data. Both are absolute prerequisites for defining and monitoring SLOs.

• You cannot define a meaningful SLO for latency (SLI) without robust distributed tracing and metric collection (Telemetry).
• You cannot investigate an SLO breach without high-cardinality, queryable logs and traces (Observability).
• Effective SLOs drive the requirement for what telemetry must be collected.

Site Reliability Engineering (SRE) is a discipline that applies software engineering principles to operations and infrastructure problems. The concept of SLOs and Error Budgets is a foundational pillar of the SRE methodology.

• SREs treat operations as a software problem.
• SLOs provide the engineering-grade, data-driven replacement for vague "it must be fast and reliable" requirements.
• The Error Budget is the key SRE mechanism for balancing innovation and stability, moving teams away from a purely reactive "break-fix" model.