Canary Metrics

These metrics measure the fundamental operational stability of the new version, ensuring it does not introduce regressions in core service functionality.

• Error Rate: The percentage of requests resulting in a 5xx server error or a critical application-level failure.
• Request Success Rate: The inverse of error rate, often expressed as a percentage of successful (e.g., 2xx) HTTP responses.
• Service Availability: Uptime percentage, measured by the success of internal health checks or heartbeat probes.
• Crash Rate: For long-running processes or applications, the frequency of unexpected process terminations.

These are the primary signals for an automated rollback, as a significant degradation indicates the new version is fundamentally broken.

This category tracks the responsiveness and efficiency of the new version, as increased latency can degrade user experience and increase infrastructure costs.

• Latency Percentiles (p50, p95, p99): The request duration at the 50th, 95th, and 99th percentiles. The p95 and p99 are critical for understanding tail latency that affects the worst-case user experience.
• Throughput: The number of successful requests processed per second (RPS/QPS).
• Resource Utilization: Changes in CPU, memory, or GPU usage per request, indicating potential inefficiencies.
• Time to First Byte (TTFB): For request-response services, the time until the first byte of the response is sent.

Performance regressions, especially in tail latency (p99), can be a leading indicator of underlying architectural problems.

These metrics evaluate the impact on user-facing outcomes and the quality of the service's core function, which is especially critical for AI/ML models.

• Model Quality Metrics: For AI canaries, this includes precision, recall, F1-score, or a custom business loss function comparing predictions between versions.
• User Engagement Signals: Click-through rate (CTR), conversion rate, session duration, or feature adoption rate for the canary cohort.
• Revenue Impact: Average order value (AOV) or revenue per user for the exposed traffic segment.
• Hallucination Rate: For generative AI models, the frequency of factually incorrect or unsupported outputs.
• Instruction Following Accuracy: For agentic systems, the rate at which the model correctly adheres to complex task constraints.

These are the key success indicators that determine if a new model version provides tangible business value.

This category monitors the infrastructure footprint and operational cost of the new version, ensuring scalability and financial efficiency.

• Cost Per Request: The compute cost (e.g., in dollars) for each inference or transaction, factoring in instance type and runtime.
• Memory/GPU Memory Leaks: Steady increases in memory allocation over time, indicating resource management issues.
• Network Egress: Changes in the volume of data transferred out of the service, which can impact cloud costs.
• Inference Efficiency: For AI models, metrics like tokens generated per second per dollar.

A new version that delivers identical quality at a significantly lower cost per request represents a major operational win.

Adapting the classic Four Golden Signals (latency, traffic, errors, saturation) specifically for AI-powered services and autonomous agents.

• Latency: End-to-end inference time, including retrieval (for RAG), tool execution (for agents), and generation time.
• Traffic: Request rate (RPS) for the AI endpoint, segmented by model version or prompt template.
• Errors: Model-serving errors (e.g., OOM), validation failures, and critical business logic errors in agentic workflows.
• Saturation: Utilization of constrained resources like GPU memory, context window capacity, or rate-limited external API quotas.

Monitoring these four signals provides a comprehensive, high-level view of any AI service's operational health during a canary.

These are not raw measurements but calculated values used for robust statistical comparison between the control (baseline) and canary (new version) groups.

• Delta/Relative Difference: The percentage or absolute change in a metric (e.g., (canary_p99 - baseline_p99) / baseline_p99).
• Confidence Intervals: A statistical range (e.g., 95% CI) calculated for key metric deltas to determine if an observed change is significant or likely due to random noise.
• Mann-Whitney U Test / t-test: Non-parametric and parametric statistical tests used by Automated Canary Analysis (ACA) tools like Kayenta to formally compare metric distributions between groups.
• Trend Analysis: Observing the direction and stability of a metric over the canary period, not just a point-in-time comparison.

These derived metrics are the foundation of automated deployment verdicts, moving beyond simple threshold checks to data-driven statistical decision-making.

A Service Level Indicator (SLI) is a quantitative measure of a specific aspect of a service's performance. In canary analysis, SLIs are the raw metrics collected and compared between versions.

• Examples for AI/ML: Model inference latency (p95, p99), prediction error rate, token throughput, business KPIs like conversion rate.
• Purpose: Provides the foundational data used to calculate compliance with a Service Level Objective (SLO).
• Golden Signals: Key SLIs often include latency, traffic, errors, and saturation.

A deployment verdict is the final automated or manual decision resulting from canary analysis. It determines whether to promote the new version to full production or initiate a rollback.

• Automated Criteria: Based on breaches of predefined metric thresholds (e.g., error rate > 0.1% for 5 minutes).
• Promote: The canary version is deemed healthy and becomes the new baseline.
• Rollback: The canary version is deemed unhealthy, and traffic is routed back to the stable version, often via automated rollback.

The champion-challenger model is a deployment pattern where the stable production model (the champion) is compared against one or more candidate models (the challengers) using live traffic.

• Canary as Challenger: A canary deployment is a practical implementation of this pattern.
• Objective: To statistically determine if a challenger model outperforms the champion on key business and performance metrics.
• Use Case: Common in financial services and recommendation systems for rigorous model validation.

Blast radius refers to the scope and potential impact of a failure during a deployment. Canary deployments are explicitly designed to minimize blast radius.

• Containment: By initially exposing the new version to only a small subset of users or infrastructure, the negative impact of a defective release is contained.
• Risk Management: A core principle of progressive delivery strategies.
• Expansion: The blast radius is intentionally expanded only as confidence in the new version grows through successful metric analysis.