Health Check

A health check's primary function is to proactively verify that a service instance is running and responsive. It is a simple request, often to a dedicated /health endpoint, that expects a successful HTTP status code (e.g., 200) within a strict timeout period.

• Mechanism: Typically a lightweight HTTP GET or TCP ping.
• Purpose: To inform load balancers and orchestration systems (like Kubernetes) if an instance should receive traffic or be restarted.
• Key Distinction: Differs from readiness probes, which check if a service is fully initialized and ready for work (e.g., model loaded, database connected).

Health checks are the control signal for infrastructure automation. They are consumed by systems that manage traffic routing and instance lifecycle.

• Load Balancers: Use health checks to populate their pool of healthy backends. An unhealthy instance is removed from rotation.
• Orchestrators (e.g., Kubernetes): Use liveness probes to decide when to restart a container and readiness probes to manage when to add a pod to a service's endpoints.
• Service Meshes (e.g., Istio): Utilize health checks to manage traffic within the mesh, enabling fine-grained control for canary deployments.

Beyond a simple binary pass/fail, health checks in advanced canary systems collect granular metrics that feed into Automated Canary Analysis (ACA).

• Latency: Response time percentiles (p50, p95, p99) are compared between the baseline (control) and canary deployments.
• Error Rates: The percentage of failed health checks or application errors.
• Throughput: The rate of successful requests handled.
• Golden Signals: Health checks instrument the core golden signals—latency, traffic, errors, and saturation—providing the data for statistical comparison.

The pass/fail state of a health check is governed by predefined success criteria aligned with Service Level Objectives (SLOs).

• Threshold-Based: A check fails if latency exceeds 500ms or the error rate surpasses 0.1%.
• SLO Alignment: Criteria are derived from the service's error budget. A canary failing its health checks consumes this budget, triggering a rollback.
• Multi-Dimensional: Success is rarely a single metric. A comprehensive health check evaluates a basket of indicators, including business KPIs for AI models (e.g., prediction quality scores).

Production health checks are often layered, moving from shallow to deep validation, and are aware of service dependencies.

• Shallow Check: Verifies the web server process is running.
• Deep Check: Validates critical downstream dependencies (e.g., vector database connectivity, model inference endpoint latency, external API availability).
• Dependency State: A service may be marked unhealthy if a critical downstream service (like a payment processor or ML feature store) is unavailable, preventing cascading failures.

Health checks are the primary trigger for automated remediation actions in modern deployment pipelines.

• Automated Rollback: In a canary deployment, a sustained failure of health checks on the new canary instances triggers an automated rollback to the stable version.
• Traffic Shifting: Tools like Flagger or Argo Rollouts watch health check metrics and automatically halt a progressive rollout if thresholds are breached.
• Alerting: Health check failures generate alerts for site reliability engineers (SREs), but the goal is automated response to minimize Mean Time to Recovery (MTTR).

A software release strategy where a new version is deployed to a small, controlled subset of live production traffic. This allows for real-world evaluation of its performance, stability, and correctness against the baseline version before a full rollout. It is the primary context in which automated health checks are analyzed.

• Key Mechanism: Uses traffic splitting to route a percentage of requests to the new version.
• Primary Goal: To detect regressions with minimal user impact, limiting the blast radius of a potential failure.

The process of using predefined canary metrics and statistical analysis to automatically evaluate the health of a canary deployment. ACA systems compare metrics from the canary (new version) and baseline (old version) to generate a deployment verdict (promote or rollback).

• Core Function: Replaces manual dashboard monitoring with algorithmic decision-making.
• Common Tools: Kayenta (Netflix), Flagger (Kubernetes operator), and Argo Rollouts.
• Inputs: Metrics like error rates, latency percentiles (p50, p99), and business KPIs.

The controlled routing of a percentage of user requests to different versions of a service. This is the enabling mechanism for canary deployments and A/B/n testing.

• Implementation: Often managed by a service mesh (e.g., Istio VirtualService) or an API gateway.
• Patterns: Can be simple (5% to new, 95% to old) or complex, based on user attributes or geography.
• Purpose: To isolate the new version's performance for direct comparison against the baseline.

Quantitative measures and targets that define a service's reliability. Health checks are a low-level indicator, while SLIs/SLOs provide the business context for ACA.

• Service Level Indicator (SLI): A direct measurement of service performance (e.g., request success rate, latency). Health check success/failure rate is a fundamental SLI.
• Service Level Objective (SLO): A target value or range for an SLI (e.g., "99.9% of health checks must pass").
• Error Budget: The allowable amount of unreliability (1 - SLO), which dictates when a failing canary must be rolled back.

A deployment safety mechanism that automatically reverts a software release to a previous stable version when predefined failure conditions are breached. It is the fail-safe action triggered by a failed health check or ACA verdict.

• Trigger Conditions: Based on golden signals (latency, errors, traffic, saturation) exceeding thresholds defined in the SLO/error budget.
• Requirement: Tight integration with the deployment orchestration system (e.g., Kubernetes, Spinnaker).
• Benefit: Enforces a "fast failure" principle, minimizing user impact from a bad release.

A release strategy where all incoming production traffic is duplicated and sent to a new version running in parallel. The new version processes the traffic but its responses are discarded; it does not affect users.

• Contrast with Canary: Zero user impact, but also no direct feedback loop on user experience. Used for performance testing and correctness validation under real load.
• Health Check Role: The shadow instance still runs health checks to ensure it is operational and can process the mirrored traffic without crashing.