Golden Signals

Latency measures the time required to service a request. It is the primary indicator of user-perceived performance.

• Focus on tail latency: While average latency is useful, the 95th or 99th percentile (p95, p99) is critical for understanding worst-case user experience.
• Distinguish success vs. failure: Track latency for successful requests separately from failed ones, as errors often return quickly, skewing the metric.
• Example: An API with a p99 latency of 2 seconds means 99% of requests complete within 2 seconds; the remaining 1% are slower, potentially causing user frustration.

Traffic quantifies the demand placed on your service, typically measured as requests per second (RPS/QPS), network I/O, or concurrent sessions.

• Service-specific metrics: For a web server, it's HTTP requests/sec. For a database, it could be transactions/sec. For a streaming service, it's network bytes/sec.
• Correlates with other signals: A spike in traffic often correlates with increased latency and errors. Understanding baseline traffic patterns is essential for capacity planning.
• Use for scaling: Traffic is the primary input for autoscaling policies, triggering the addition or removal of service instances to meet demand.

Errors track the rate of requests that fail, either explicitly (HTTP 5xx, gRPC internal errors) or implicitly (HTTP 200 OK with wrong or degraded content).

• Explicit vs. Implicit: Monitor both hard failures (e.g., 500 errors, timeouts) and soft failures (e.g., successful responses with invalid data, high latency that triggers client-side timeouts).
• Error budget consumption: The error rate directly consumes your service's error budget (1 - SLO). A sustained high error rate signals an imminent breach of reliability commitments.
• Golden signal for canaries: A rising error rate in a canary deployment compared to the baseline is often the fastest indicator of a problematic release.

Saturation measures how "full" your service is, indicating the utilization of its most constrained resource (the bottleneck).

• Resource-focused: This could be CPU utilization, memory consumption, disk I/O queue length, or network bandwidth. The key is identifying the limiting resource.
• Proactive signal: Saturation often increases before latency degrades or errors spike. A service at 100% saturation has no capacity to handle traffic spikes, leading to cascading failures.
• Example Metrics: CPU usage >80%, memory swap rate, disk queue length, or network interface congestion. The saturation threshold is service-dependent.

In Production Canary Analysis, the four golden signals are compared between the baseline (stable) deployment and the canary (new) deployment.

• Automated Comparison: Tools like Kayenta or Flagger statistically analyze differences in latency distributions, error rates, and traffic patterns between control and canary groups.
• Deployment Verdict: A significant degradation in any golden signal (e.g., higher p99 latency, increased error percentage, or abnormal resource saturation) triggers an automated rollback.
• Holistic Health View: Together, they provide a complete picture of whether the new model or service version performs as well as or better than the current one under real load.

For AI/ML services, the golden signals framework expands to include model-specific quality metrics.

• Latency: Model inference time, token generation speed.
• Traffic: Predictions per second, token throughput.
• Errors: Inference failures, hardware (GPU) errors.
• Saturation: GPU memory utilization, accelerator compute load.
• Augmented Signals: Must also monitor model performance drift (e.g., prediction score distribution shifts), hallucination rates for LLMs, and business metric impact (e.g., conversion rate in a recommendation canary).

A target level of reliability or performance for a service, expressed as a measurable goal over a rolling time window. SLOs are the quantitative benchmarks against which golden signals are judged during a canary analysis.

• Example: "99.9% of requests shall have a latency under 200ms over a 30-day window."
• Relationship to Golden Signals: Defines the acceptable thresholds for latency (performance) and error rate (reliability).
• Error Budget: The allowable amount of unreliability (1 - SLO); a canary failure consumes this budget.

The controlled routing of a defined percentage of live user requests to different versions of a service. This is the mechanism that enables a canary deployment by directing a small slice of traffic (one of the golden signals) to the new model.

• Implementation: Often managed by a service mesh (e.g., Istio VirtualService) or a deployment controller.
• Progressive Rollout: Traffic percentage is gradually increased from 1% to 100% based on successful canary analysis.
• Key Consideration: Must ensure traffic splitting is statistically representative to avoid skewed metric comparisons.

The final decision to promote a new candidate to full production or rollback to the previous stable version. This verdict is the primary output of an Automated Canary Analysis (ACA) process that evaluates the golden signals.

• Automation: The ideal state is a fully automated verdict based on breached SLOs.
• Criteria: Based on statistical comparisons of canary vs. control metrics for errors, latency, and sometimes business KPIs.
• Rollback Trigger: A significant degradation in any golden signal typically triggers an automated rollback to limit blast radius.

The scope and potential impact of a failure introduced by a new deployment. A core goal of canary analysis is to minimize the blast radius by initially exposing the change to a very small subset of users or infrastructure.

• Containment Strategy: Limited initially by low traffic percentage in a canary.
• Golden Signals as Early Warning: A spike in errors or latency in the canary group signals a problem while the impact is contained.
• Escalation: If the canary is healthy, the blast radius is intentionally increased through a progressive rollout.

The specific set of quantitative measurements collected and analyzed during a canary deployment. While the four golden signals provide a universal health check, canary metrics are often extended to include:

• Business KPIs: Conversion rates, transaction values, or user engagement scores.
• Domain-Specific Signals: For AI models, this includes prediction drift, hallucination rates, or confidence score distributions.
• Infrastructure Saturation: Memory, CPU, and GPU utilization beyond generic service saturation.

These metrics are visualized in a canary analysis dashboard for real-time decision-making.