Health Probe

Health probes are categorized by their operational purpose. A liveness probe determines if a container or process is running. A failure typically triggers a restart. A readiness probe determines if a container is ready to accept traffic (e.g., dependencies initialized, warm caches loaded). A failure prevents traffic from being sent to the pod. A third type, the startup probe, is used for legacy applications with long initialization times, disabling liveness/readiness checks until it succeeds.

Probes are executed by the orchestrator's kubelet against a container according to a defined schedule. The three primary mechanisms are:

• HTTP GET: The most common. The kubelet sends an HTTP request to a specified path and port. A success code (200-399) passes the probe.
• TCP Socket: The kubelet attempts to open a TCP connection to a specified port. Success is established if a connection is made.
• Exec Command: The kubelet executes a specified command inside the container. A zero exit code indicates success.

Probe behavior is finely tuned via parameters to balance responsiveness with stability, preventing flapping (rapid, cyclical failures). Key parameters include:

• initialDelaySeconds: Wait time after container start before initiating probes.
• periodSeconds: How often to perform the probe.
• timeoutSeconds: Number of seconds after which the probe times out.
• successThreshold: Minimum consecutive successes for the probe to be considered successful after a failure.
• failureThreshold: Number of consecutive failures required for the probe to be considered failed.

Probes are integral to the container orchestrator's control loops. In Kubernetes, probe results directly inform the decisions of core controllers:

• The kubelet uses liveness probes to decide when to restart a container.
• The kubelet uses readiness probes to add or remove a pod's IP from the endpoints list of a matching Service.
• The Deployment controller considers pod readiness during rolling updates, ensuring new pods are ready before scaling down old ones. This creates a deterministic, self-healing feedback loop.

A well-designed probe endpoint is lightweight, stateless, and checks critical internal dependencies. Best practices include:

• Checking internal in-memory state or a local cache.
• Performing a shallow check on a crucial downstream dependency (e.g., database connection pool).
• Avoiding deep dependency checks that cascade failures or heavy computational logic that consumes significant resources. The endpoint should return quickly to avoid blocking the orchestrator's control loop.

Health probes operate at the infrastructure layer, while patterns like the Circuit Breaker operate at the application layer. A circuit breaker trips based on business logic failure rates, while a readiness probe fails on a technical health check. Together, they provide layered fault tolerance. Probe metrics (success/failure counts, latency) are critical observability signals, feeding into dashboards and alerts to provide a real-time view of system resilience and the effectiveness of self-healing mechanisms.

A software design pattern that prevents an application from repeatedly attempting an operation likely to fail. It acts as a proxy for operations, monitoring for failures. When failures exceed a threshold, the circuit opens, failing fast and preventing cascading failures and resource exhaustion. After a timeout, it enters a half-open state to test if the underlying problem has resolved before closing again. This pattern is critical for building fault-tolerant microservices and is a complementary control mechanism to health probes.

A periodic, lightweight message sent from a subordinate component (like a service instance) to a monitoring system or orchestrator to indicate liveness. Unlike a health probe, which is an active check initiated by the orchestrator, a heartbeat is a passive, push-based signal. If heartbeats stop arriving, the monitoring system infers the component has failed. Heartbeats are often used in conjunction with probes for comprehensive liveness detection in distributed systems like Kubernetes (kubelet node status) and consensus algorithms like Raft.

A fault isolation design inspired by ship compartments. It partitions system resources (e.g., thread pools, connections, memory) into isolated groups for different consumers or operations. A failure in one bulkhead (e.g., a downstream service timeout exhausting its dedicated connection pool) does not cascade and drain resources from unrelated parts of the system. This pattern ensures graceful degradation. Health probes often operate within a specific bulkhead, and their failure should only affect the associated partitioned resources.

A distributed coordination process where nodes in a cluster autonomously agree on a single node to act as the leader or coordinator. The leader typically manages critical tasks like assigning work or maintaining consensus. Health probes (or heartbeats) are fundamental to this process: the failure of a leader's health check triggers a new election. Algorithms like Raft and Paxos implement robust election protocols. This ensures continuous operation and is a core pattern for high-availability systems like databases (etcd, Consul) and orchestrators.

A fundamental control loop in declarative systems like Kubernetes. It continuously observes the actual state of the world (e.g., pod statuses from health probes) and compares it to the declared desired state (e.g., a deployment manifest). It then computes and executes the necessary actions (kill, create, restart) to converge the actual state with the desired state. Health probes provide the critical observability signal that drives this loop. This pattern is central to GitOps and self-healing infrastructure, enabling autonomous recovery from drift and failure.

A fault-tolerance philosophy central to the Erlang/OTP and Actor models. Instead of writing complex defensive code to handle every possible internal error, processes are designed to fail fast upon encountering an unexpected condition. A supervising process, equipped with a restart strategy (e.g., one-for-one, rest-for-one), detects the crash (via a monitoring mechanism analogous to a health probe) and restarts the failed process from a clean state. This creates resilient systems where failure is isolated and recovery is automated, aligning with the goals of health probes in container orchestrators.