Golden Signal

Latency measures the time it takes to service a request. For AI services, this is critical and often broken down into distinct phases:

• Model Inference Latency: Total time from input submission to output generation.
• Time To First Token (TTFT): For streaming LLM responses, the delay until the first token is emitted.
• Time Per Output Token (TPOT): The throughput for generating subsequent tokens. Monitoring percentile latency (p50, p95, p99) is essential, as p99 (tail latency) often dictates user-perceived performance. High latency directly violates user-centric SLOs.

Traffic quantifies the demand placed on your service. For AI systems, this is more nuanced than simple request counts.

• Queries Per Second (QPS): The raw volume of inference requests.
• Concurrent Users/Sessions: Number of simultaneous active interactions.
• Input/Output Volume: Size of prompts and generated completions, which impacts computational load. Understanding traffic patterns is required for capacity planning, auto-scaling, and correlating load with other signals like latency and error rates.

Errors measure the rate of requests that fail. In AI services, failures extend beyond HTTP 5xx codes to include model-specific failures.

• Service Errors: Failed API calls, timeouts, and infrastructure failures.
• Model Errors: Structured output validation failures, context window overflows, or resource exhaustion.
• Quality Errors: Outputs that violate SLOs for hallucination rate or answer faithfulness in RAG systems. Tracking error rate against an error budget is fundamental to SRE's risk-management approach.

Saturation measures how "full" your service is, indicating resource exhaustion before errors or latency spikes occur. It's a measure of system utilization.

• Hardware Metrics: GPU/CPU utilization, memory pressure, and I/O bandwidth.
• Service-Specific Limits: Queue lengths, continuous batching efficiency in LLM servers, or token generation buffer saturation.
• Derived Metrics: Scaling factor (demand/capacity). Monitoring saturation provides the leading indicator needed for proactive scaling, preventing tail latency amplification and cascading failures.

AI services require adapting the golden signals to model-specific behaviors. Key considerations include:

• Defining SLIs/SLOs: An SLO for model inference latency or an SLO for hallucination rate translates golden signals into actionable reliability targets.
• Agentic Systems: For autonomous agents, traffic may be tasks/hour, errors could be agent task success rate, and saturation might involve orchestration engine queue depth.
• Observability Integration: These signals must feed into multi-window alerting based on burn rate to protect SLOs.

While the four signals are sufficient for most services, AI systems often require supplemental signals for full observability:

• Quality & Correctness: Metrics like Retrieval Precision@K for RAG or instruction following accuracy.
• Data Health: Data drift detection to monitor input distribution shifts.
• Business Impact: SLO for business metric correlation linking latency to user conversion.
• Cost: SLO for cost efficiency (e.g., cost per inference) to balance performance with expenditure. These extensions ensure monitoring captures both the operational and functional health of AI systems.

A Service Level Objective (SLO) is a quantitative target for the reliability, performance, or quality of a service. It is derived from one or more Service Level Indicators (SLIs) and defines the acceptable level of service over a specific time window.

• Core Function: Translates raw metrics (SLIs) into a business-agreed target (e.g., "99.9% of requests must have latency < 200ms this quarter").
• AI Application: For AI services, SLOs can target model inference latency, hallucination rate, or retrieval precision, moving beyond traditional infrastructure metrics.
• Key Property: SLOs define "how good is good enough" and create an error budget for managing risk.

A Service Level Indicator (SLI) is a directly measurable metric that quantifies a specific aspect of a service's performance or reliability. SLIs are the raw measurements that feed into Service Level Objectives (SLOs).

• Golden Signal Examples: Latency (p95, p99), error rate (5xx responses), traffic (queries per second), saturation (GPU utilization %).
• AI-Specific SLIs: Include Time To First Token (TTFT), Time Per Output Token (TPOT), and answer faithfulness score.
• Measurement: SLIs must be measured from the user's perspective wherever possible to reflect true experience.

An error budget is the explicit, quantified amount of unreliability a service team is allowed within a Service Level Objective's (SLO) time window. It is calculated as 100% - SLO Target.

• Purpose: Serves as a shared resource for balancing velocity and stability. Spending the budget on launches is acceptable; exhausting it triggers a focus on reliability.
• Management: Teams track burn rate—the speed at which the error budget is consumed—to alert on sustained degradation versus brief spikes.
• Cultural Tool: Transforms reliability from an abstract goal into a concrete, spendable resource for product and engineering decisions.

A Service Level Agreement (SLA) is a formal contract between a service provider and a customer that defines the minimum expected level of service, often including financial penalties or remedies if commitments are not met.

• Relationship to SLOs: An SLA is typically an external business contract, while SLOs are internal engineering targets. SLOs are often set stricter than the SLA to provide a safety margin.
• Enforceability: SLAs involve legal and commercial consequences (e.g., service credits), whereas missed SLOs trigger internal operational reviews.
• AI Context: For AI-as-a-Service offerings, SLAs may cover uptime, throughput, and specific quality metrics like maximum hallucination rates.

Burn rate quantifies the speed at which a service consumes its error budget. It is calculated as the percentage of the total error budget consumed per unit of time (e.g., 10% per hour).

• Alerting Strategy: Forms the basis for multi-window alerting. A high burn rate for a short period may indicate a transient spike, while a moderate burn rate over a long period signals sustained degradation likely to violate the SLO.
• Proactive Management: Allows teams to respond to reliability issues before the error budget is fully exhausted and the SLO is breached.
• Visualization: Often plotted on a graph showing budget remaining over time, with trend lines predicting time until exhaustion.

A Critical User Journey (CUJ) is a specific, high-value sequence of user interactions that is essential to the user's success with a service. SLOs and SLIs should be defined to protect these journeys.

• Foundation for SLOs: Instead of monitoring every API endpoint, teams identify 5-10 key CUJs (e.g., "User submits a query and receives a summarized answer") and instrument their SLIs.
• AI Example: For a RAG system, a CUJ could be: User asks a question → System retrieves 3 relevant documents → LLM synthesizes a grounded answer. Latency and error SLIs would track this entire flow.
• User-Centric Design: Ensures monitoring focuses on what matters to the business and end-user, not just internal system health.