Health Check

Health checks are executed via different probe mechanisms, each serving a distinct operational purpose.

• HTTP/HTTPS GET Probe: The most common type. The orchestrator sends an HTTP request to a specified endpoint (e.g., /health). A response with a 2xx or 3xx status code indicates health; 4xx/5xx codes indicate failure.
• TCP Socket Probe: The orchestrator attempts to open a TCP connection to a specified port. Success is based on establishing a connection, regardless of application-layer logic.
• Command Execution Probe: The orchestrator executes a command inside the container (e.g., a script). A zero exit code indicates success. This is highly flexible but adds complexity.
• gRPC Health Checking Protocol: A standard for gRPC services where the server implements a defined Health service, allowing for precise, service-specific health reporting.

In platforms like Kubernetes, health checks are categorized into two fundamental types that control different lifecycle events.

• Liveness Probe: Answers "Is the process alive?" If this probe fails, the orchestrator assumes the application is in a broken state (e.g., deadlocked) and restarts the container to try to recover it.
• Readiness Probe: Answers "Is the application ready to serve traffic?" If this probe fails, the orchestrator removes the pod's IP address from service endpoints, stopping new traffic from being sent to it. This is used during startup (e.g., waiting for a database connection) or during temporary overload.
• Startup Probe: A third type used for legacy applications with slow startup times. It disables liveness and readiness checks until it succeeds, preventing premature restarts.

The behavior of a health check is finely tuned through a set of timing parameters that balance responsiveness with stability.

• Initial Delay Seconds: The wait time after the container starts before probes are initiated. Crucial for allowing application bootstrap.
• Period Seconds: How often (e.g., every 10 seconds) the probe is executed.
• Timeout Seconds: The time after which a probe is considered failed if no response is received. Must be less than the period.
• Success/Failure Threshold: The number of consecutive successes or failures required for the probe to flip its verdict. A failure threshold greater than 1 prevents transient network blips from causing unnecessary restarts or traffic removal.

Beyond container orchestrators, health checks are a foundational feature of load balancers (Application, Network, Gateway) and service meshes.

• Purpose: To dynamically update the pool of healthy backend targets. An instance failing its health check is automatically drained of new connections and removed from the rotation.
• Advanced Checks: Cloud load balancers often support more sophisticated checks, including validating response body content, checking TLS certificates, or monitoring for specific HTTP headers.
• Draining/Connection Draining: When an instance is marked unhealthy, existing connections may be allowed to complete during a grace period before termination, enabling graceful shutdowns during deployments or scaling-in events.

The endpoint that services the health check request (e.g., /health or /ready) must be carefully engineered.

• Depth of Check: Ranges from shallow (process is up) to deep (all critical downstream dependencies like databases, caches, and internal services are responsive). Deep checks must be fast and have their own timeouts to avoid cascading failures.
• Security: The endpoint should typically be exposed on an internal port or network, not to the public internet, to prevent denial-of-service or information disclosure.
• Statelessness: The health check should not modify application state or trigger significant side effects.
• Performance Impact: Frequent, deep health checks can add load. Caching results internally for a short period (e.g., 1 second) is a common optimization.

Misconfigured health checks can themselves become a source of system instability.

• Noisy Neighbor: A deep health check that calls a overloaded downstream service can exacerbate its load, causing a cascading failure.
• Flapping: An instance repeatedly passing and failing health checks due to aggressive timeouts or threshold settings, causing constant traffic churn and restarts.
• Slow Startup Catastrophe: If the initial delay is too short, a slow-starting application will be killed by its liveness probe before it can become ready, entering a crash loop.
• Zombie Instances: An instance that passes a shallow health check but is functionally broken (e.g., its logic is corrupted) will continue to receive traffic and serve errors. This highlights the need for appropriate check depth and complementary external synthetic monitoring.

The most common implementation, where a load balancer or orchestrator makes a periodic HTTP GET request to a designated endpoint (e.g., /health or /status) on the application. A successful response (typically HTTP 200 OK) indicates the instance is healthy.

• Key Components: Lightweight endpoint, fast response time (< 100ms), minimal dependencies.
• Best Practice: The endpoint should perform a shallow check of critical internal dependencies (e.g., database connection pool, in-memory cache) but avoid deep, expensive queries.
• Example: A web service's /health endpoint checks if its connection to a primary database is alive and returns {"status": "UP"}.

A lower-level check where the orchestrator attempts to open a TCP connection to the application's designated port (e.g., 8080). Success is determined by the ability to establish a connection, not by application logic.

• Use Case: Ideal for non-HTTP services (e.g., databases, gRPC servers, custom TCP protocols) or for verifying the network stack and process are listening before application initialization is complete.
• Limitation: Only confirms the process is listening on a port, not that the application logic is functioning correctly. Often used alongside a readiness probe.

The orchestrator executes a specified command inside the application container or host. A zero exit code indicates success. This offers maximum flexibility for custom health logic.

• Kubernetes Example: A livenessProbe using exec could run a script that validates internal file locks or process states.
• Consideration: The command must be lightweight; a hanging or resource-intensive command can cause false failure signals. It is commonly used for legacy applications without built-in HTTP endpoints.

A critical distinction in platforms like Kubernetes that defines an application's lifecycle stage.

• Liveness Probe: Answers "Is the application running?" A failed liveness probe causes the container runtime to restart the pod. It diagnoses a deadlocked or otherwise broken application.
• Readiness Probe: Answers "Is the application ready to serve traffic?" A failed readiness probe removes the pod from service load balancers. It handles startup delays, temporary dependency unavailability, or maintenance modes.

Best Practice: Use both. A service might be live (process running) but not ready (waiting for cache warm-up).

An advanced pattern where a service's health endpoint aggregates the status of its downstream dependencies (databases, APIs, caches). This provides a holistic view of the service's operational capacity.

• Implementation: The /health endpoint performs concurrent, time-bound checks to each critical dependency, categorizing results (e.g., UP, DOWN, DEGRADED).
• Output: A composite status, often following a fail-fast principle: if a primary database is down, overall status is DOWN. For secondary caches, status might be DEGRADED.
• Tooling: Frameworks like Spring Boot Actuator, Micrometer, or custom libraries standardize this aggregation.

A proactive check that executes a simplified but representative user transaction to validate full business logic pathways. This goes beyond connectivity to test functional correctness.

• Example: A health check for a payment service might create a test cart, call a payment API with a test credit card, and validate a mock successful response, then clean up all test data.
• Complexity: These checks are more resource-intensive and slower, so they are run less frequently (e.g., every 30 seconds vs. every 5 seconds for an endpoint probe).
• Benefit: Catches logical failures, such as misconfigured API keys, broken code deploys, or corrupted data schemas, that shallow checks miss.

A cloud capability that automatically adjusts the number of compute resources (e.g., virtual machines, containers) based on observed metrics like CPU utilization or request queue length. Health checks are critical for scaling decisions.

• Scale-In: Unhealthy instances are terminated first during a scale-down event.
• Integration: Cloud providers' auto-scaling groups (AWS) or Kubernetes' Horizontal Pod Autoscaler (HPA) rely on health status to ensure only healthy pods are counted toward the desired replica count.