Canary Deployment

A deployment strategy where two identical production environments—blue (current) and green (new)—run in parallel. Traffic is switched instantaneously from blue to green after validation.

• Atomic cutover: 100% of traffic shifts at once, unlike canary's gradual ramp
• Instant rollback: Revert by pointing the load balancer back to blue
• Double resource cost: Both environments run simultaneously during the switch
• Best for: Stateless model serving where cold start is not a concern

Example: A recommendation model v2 is deployed to the green cluster, validated with synthetic traffic, then promoted via a single DNS update.

A software pattern that decouples deployment from release by wrapping new model logic in conditional toggles evaluated at runtime.

• Per-request control: Route individual inference requests to model variants based on user ID, geography, or randomized cohorts
• Kill switch: Disable a problematic model instantly without redeployment
• A/B test integration: Use the same flagging system to measure model performance
• Technical debt risk: Stale flags must be cleaned up after full rollout

Example: A canary deployment uses a feature flag to route 5% of premium-tier users to the new pricing model while monitoring P99 latency.

A risk-free validation technique where a new model version receives a mirrored copy of live production traffic but its predictions are discarded—only logged for analysis.

• Zero user impact: The shadow model's outputs never reach end users
• Performance comparison: Compare latency distributions and prediction distributions side-by-side against the production model
• Resource overhead: Requires duplicate inference compute without serving real requests
• Pre-canary step: Often used before a canary deployment to validate model behavior under real load

Example: A fraud detection model v3 shadows 100% of transactions for one week; its anomaly scores are compared to v2 before any canary traffic is routed.

A Kubernetes-native deployment strategy that incrementally replaces old Pod instances with new ones, one at a time, without creating a separate environment.

• maxSurge and maxUnavailable: Parameters controlling how many pods are added/removed during the rollout
• No traffic splitting: The Service load balancer distributes requests across both old and new pods during the transition
• Simpler than canary: No need for separate traffic management infrastructure like Istio
• Risk: A bad version affects a fraction of users immediately with no granular control

Example: A model serving Deployment with 10 replicas and maxUnavailable: 2 replaces two pods at a time until all run the new image.

An infrastructure-level capability—often implemented via a service mesh like Istio—that duplicates live requests to an alternate destination for testing.

• Envoy filter: Configured at the proxy layer to fork TCP streams
• Fire-and-forget: Responses from the mirrored destination are silently dropped
• Latency isolation: Mirroring adds no latency to the primary request path
• Use case: Validate a new model's inference latency under genuine production traffic patterns before canary routing

Example: An Istio VirtualService mirrors 100% of traffic from the recommender-v1 service to recommender-v2 while only v1 responses reach the client.

An umbrella term encompassing canary deployments, blue-green, and feature flagging—any strategy that exposes new software to users in a controlled, incremental manner.

• Observability-driven: Progression gates depend on metrics like error rate, P99 latency, and business KPIs
• Automated rollback: Tools like Argo Rollouts or Flagger automatically revert if metrics breach SLOs
• Analysis templates: Define Prometheus queries that gate promotion to the next canary step
• Goal: Minimize the blast radius of a bad deployment

Example: Flagger runs a canary analysis where traffic to the new model increases from 5% to 50% over 10 minutes, with automated rollback if HTTP 5xx errors exceed 1%.