Real User Monitoring (RUM)

The time elapsed from a user's request to the delivery of the model's final output, measured as percentiles. This is the primary user-perceived performance metric for interactive AI features.

• P50 (Median): Represents the typical user experience.
• P95/P99 (Tail Latency): Critical for understanding worst-case scenarios, which often correlate with user abandonment. A spike in P99 latency during a canary is a strong rollback signal.
• Example: A chatbot's response time or an image generation model's time-to-first-token.

The percentage of user requests where the model fails to produce a valid, usable response, triggering either an error or a fallback to a default/heuristic system.

• HTTP 5xx Errors: Indicate infrastructure failures (e.g., GPU OOM, container crashes).
• Application Errors: Include malformed outputs, serialization failures, or context window overflows.
• Fallback Rate: Tracks how often a safety net or less-capable model is invoked. A rising fallback rate in a canary suggests the new model is less reliable than the champion.

Domain-specific success metrics tied directly to user satisfaction and business outcomes. These are the ultimate determinants of a model's value.

• For a RAG System: Click-through rate on cited sources, session length.
• For a Chatbot: Conversation completion rate, user satisfaction score (post-interaction survey).
• For a Recommendation Model: Conversion rate, add-to-cart rate.
• For Code Generation: Acceptance rate of suggested code, developer edit distance.
• A successful canary must show non-inferiority or improvement in these KPIs.

Measures of computational resource consumption and system capacity derived from real user traffic patterns.

• Tokens per Request: Directly correlates with cost for LLM APIs (e.g., OpenAI, Anthropic). A canary model generating longer outputs can significantly increase operational expenses.
• Requests per Second (RPS): Indicates the load pattern and helps validate autoscaling configurations.
• Concurrent User Sessions: Gauges the system's ability to handle stateful, multi-turn interactions under load.

Metrics capturing failures or degradations in the user's browser or mobile application when interacting with the AI service.

• JavaScript Error Rate: Errors from the frontend SDK or widget integrating the model.
• Web Vitals for AI Features: Largest Contentful Paint (LCP) for AI-generated content, Interaction to Next Paint (INP) for chat interfaces.
• Mobile App Crashes: Crashes attributed to the native SDK handling model responses.
• These metrics are essential for full-stack canary analysis, as a model change can inadvertently break client-side integrations.

The segmentation of core RUM metrics by user location, device type, or other relevant attributes to ensure equitable performance.

• Regional Latency: Model inference latency for users in Europe vs. Asia-Pacific, which may be routed to different data centers.
• Device Performance: Latency and error rate on low-end mobile devices versus desktop computers.
• Key Use: Detecting performance regression bias where a new model performs well for one user cohort but poorly for another, which would be masked by global averages.

A software release strategy where a new version is deployed to a small, controlled subset of live production traffic. Its performance is evaluated against a stable baseline before a full rollout.

• Core Mechanism: Uses traffic splitting to route a percentage of real user requests to the new version.
• Primary Goal: To limit the blast radius of potential failures by exposing only a fraction of users.
• Evaluation: Relies on canary metrics (e.g., error rates, latency) from RUM and system telemetry to make a deployment verdict.

The process of using statistical analysis on predefined metrics to automatically evaluate the health of a canary deployment and decide whether to promote or roll back.

• Automation: Tools like Kayenta, Flagger, or Argo Rollouts execute ACA by comparing metrics between control (old) and canary (new) groups.
• Inputs: Consumes golden signals (latency, errors, traffic, saturation) and business KPIs derived from RUM and infrastructure monitoring.
• Output: Produces a pass/fail deployment verdict without manual intervention, enabling safe, continuous deployment.

A proactive monitoring technique that uses scripted, simulated transactions from external locations to test application performance and availability.

• Contrast with RUM: While RUM measures actual user experience, synthetic monitoring tests expected performance from predefined paths and geographies.
• Use Case: Ideal for establishing a performance baseline, testing critical user journeys before launch, and monitoring uptime where real user traffic is sparse.
• Combined Strategy: Used alongside RUM to get a complete picture: synthetic for consistency and alerts, RUM for real-world variability.

The controlled routing of a percentage of user requests to different versions of a service, enabling canary deployments and A/B/n testing.

• Enabling Technology: Implemented via service mesh resources (e.g., Istio VirtualService), API gateways, or dedicated rollout controllers.
• Precision: Allows for fine-grained control, such as routing 5% of traffic to a new model variant based on user attributes or request headers.
• Foundation: The core mechanism that makes progressive rollouts and champion-challenger model comparisons possible using live traffic.

Quantitative measures and targets for service reliability and performance. SLIs are the measured metrics (e.g., latency p99, error rate), while SLOs are the target values for those metrics.

• Role in Canaries: SLOs define the success criteria for a canary release. An ACA tool checks if the canary's SLIs violate the SLO, consuming the error budget.
• RUM Data: User-centric SLIs like page load time or transaction success rate are directly sourced from RUM data.
• Governance: Provides an objective, business-aligned framework for automated deployment decisions.

A release strategy where all incoming production traffic is duplicated and sent to a new version running in parallel, without affecting the live user response.

• Core Mechanism: Uses traffic mirroring to send identical requests to the shadow instance.
• Primary Use: To validate a new version's functional correctness, performance under real load, and output fidelity (e.g., for a new ML model) with zero user impact.
• Data Source: The shadow system's logs and performance metrics, which can be compared to the primary system, provide a rich dataset for evaluation without risk.