Kayenta

The system evaluates deployments across a broad spectrum of metrics—not just technical system health but also business KPIs. It aggregates data from multiple sources to provide a holistic view of the canary's impact.

• Metric Types: Analyzes infrastructure metrics (CPU, memory), application metrics (error rates, latency percentiles), and business metrics (conversion rates, revenue per session).
• Multi-Dimensional Analysis: Can segment analysis by dimensions like geographic region, user cohort, or device type to detect issues that only affect specific subsets of traffic.

Kayenta is designed as a pluggable service that fetches time-series data from industry-standard monitoring backends. It does not replace your observability tools but acts as an analysis layer on top of them.

• Supported Providers: Native integrations include Atlas, Datadog, Graphite, New Relic, Prometheus, and Stackdriver.
• Unified Analysis: This allows teams to use a single canary analysis service regardless of their underlying monitoring choices, standardizing the release process across an organization.

Success criteria are defined declaratively through configuration files. Users specify which metrics to analyze, their relative importance, and the pass/fail thresholds for each.

• Weighted Scoring: Each metric is assigned a weight. The system calculates a composite score, and the canary must meet a minimum overall threshold to pass.
• Flexible Policies: Configurations can define different marginal and critical failure thresholds, allowing for nuanced policies where some metric regressions are tolerated more than others.

By codifying release criteria into configuration, Kayenta ensures that every deployment is evaluated against the same objective standards. This eliminates variance between teams or engineers and builds institutional knowledge around what constitutes a 'safe' release.

• Audit Trail: Provides a clear record of which metrics were analyzed, their results, and the final verdict, creating an auditable trail for compliance and post-mortem analysis.
• Continuous Improvement: Teams can iteratively refine their metric thresholds and weights based on historical analysis data, continuously improving the precision and safety of their deployment process.

The release strategy that ACA evaluates. A canary deployment is a technique where a new version of an application or model is initially deployed to a small, controlled subset of live production traffic. This strategy intentionally limits the blast radius of a potential failure. The canary's performance is compared to the baseline (control) version. If metrics are acceptable, traffic is gradually shifted; if not, the deployment is rolled back.

The foundational metrics for analysis. Kayenta's statistical tests are performed on Service Level Indicators (SLIs), which are quantitative measures of service performance (e.g., request latency, error rate, throughput). These are evaluated against Service Level Objectives (SLOs), which are target values or thresholds for those SLIs. The error budget (1 - SLO) defines the acceptable amount of unreliability, which a canary deployment must not consume excessively.

The mechanism that enables the canary pattern. Traffic splitting is the controlled routing of a percentage of user requests to different versions of a service. In a Kayenta-managed deployment, an infrastructure component (like a service mesh's Istio VirtualService or a load balancer) splits traffic between the control and canary pods. Kayenta analyzes the metrics from both pools to inform whether to adjust the split (e.g., increase canary traffic) or initiate a rollback.