Progressive Delivery

A feature flag is a configuration mechanism that allows teams to dynamically enable or disable specific functionality for different user segments without deploying new code. This decouples deployment from release, enabling:

• Safe rollouts: Enable a new model for internal users first.
• Instant kill switches: Disable a problematic feature without a rollback.
• Targeted experimentation: Expose features to specific user cohorts (e.g., beta testers). Flags are managed via a central service and evaluated at runtime, making them foundational for controlled exposure.

A canary release is a deployment strategy where a new version is initially rolled out to a small, specific subset of users or infrastructure. Performance and stability are monitored against key metrics (latency, error rate, business KPIs) before a full rollout. This technique:

• Limits blast radius by exposing only a small percentage of traffic (e.g., 5%).
• Uses real-user validation under actual production load.
• Enables automated rollback if metrics breach predefined Service Level Objectives (SLOs). The name derives from 'canary in a coal mine,' serving as an early warning system.

A/B testing is a controlled experiment methodology that compares two or more model variants by randomly splitting user traffic to measure which performs better against a predefined objective (e.g., click-through rate). Multi-armed bandit algorithms extend this by dynamically reallocating traffic to favor the best-performing variant, optimizing for reward while exploring alternatives. This technique provides:

• Statistical rigor for causal inference.
• Business metric validation beyond technical performance.
• Dynamic optimization that reduces opportunity cost during experiments.

In shadow mode, a new model processes live production traffic in parallel with the current champion model, but its predictions are logged and not used to affect user-facing decisions. Also known as a dark launch, this technique allows for:

• Zero-risk performance comparison on identical, real-time data.
• Infrastructure validation under true load before cutting over.
• Detection of prediction outliers or edge cases. It requires dual execution, increasing compute cost but providing the safest form of validation.

Blue-green deployment maintains two identical production environments (blue and green). The stable version runs on one environment (e.g., blue) while the new version is deployed to the other (green). A router or load balancer performs an instantaneous traffic switch, enabling:

• Zero-downtime updates and instant rollbacks by switching back.
• Elimination of version mismatch during deployment.
• Simplified disaster recovery with a known-good environment always on standby. This pattern is often combined with canary releases for phased traffic shifting within the green environment.

Automated rollbacks are triggered by continuous monitoring of health checks and SLOs (e.g., error rate > 1%, p95 latency > 200ms). A circuit breaker is a resilience pattern that temporarily stops sending requests to a failing model endpoint to prevent cascading failures. Together, they form a safety net:

• Rollback strategies revert to a last-known-good version using feature flags or traffic switching.
• Circuit breakers protect downstream services from unstable models.
• Kill switches provide manual overrides for emergency shutdowns. This automation is critical for meeting reliability targets without constant human intervention.

A deployment technique where a new model processes live production traffic in parallel with the current (champion) model, but its predictions are logged and not used to affect user-facing decisions.

• Key Mechanism: Implements traffic mirroring to send duplicate requests to the shadow model.
• Progressive Delivery Role: Allows for safe, zero-risk performance comparison and validation using real-world data before any user is exposed to the new model's output.

The practice of routing a controlled percentage of incoming inference requests to different model versions or endpoints. This is the fundamental routing mechanism that enables canary releases and A/B tests.

• Key Mechanism: Typically controlled by a load balancer, service mesh (e.g., Istio, Linkerd), or an API gateway.
• Progressive Delivery Role: The operational backbone for gradually shifting user exposure from 1% to 100% during a rollout, based on success criteria.

A predefined plan and set of procedures for reverting a system, such as a model deployment, to a previous known-good state in response to detected failures or performance regressions.

• Key Mechanism: Often automated via CI/CD pipelines or triggered manually; relies on immutable infrastructure and precise model versioning.
• Progressive Delivery Role: The essential safety net. A progressive delivery system is incomplete without a fast, reliable rollback capability, often executed by flipping a feature flag or adjusting traffic split to 0%.