Liveness Probe

The primary function of a liveness probe is to identify deadlocks—states where the agent process exists but cannot make progress. This typically manifests as an infinite loop, resource contention, or a thread pool exhaustion. The probe does not check if the agent is correct, only if it is alive. A common implementation is a /healthz HTTP endpoint that returns a 200 status code only if the main event loop is still cycling. If the probe fails, the orchestrator (e.g., Kubernetes kubelet) sends a SIGTERM signal and restarts the container.

Liveness probes must test the internal state of the agent, not just the process table. A naive check that only verifies the PID exists will miss a deadlocked process.

• Internal Check: The agent exposes an endpoint that validates its own event loop, memory health, and sub-component connectivity.
• External Check: The orchestrator monitors resource metrics like CPU usage flatlining at 100% or memory growth without garbage collection. A robust probe combines both: a lightweight internal /live endpoint backed by external metric thresholds.

A liveness probe must be computationally trivial. It should execute in milliseconds and avoid any dependency on external services. If the probe itself queries a database or calls a remote API, a network partition could cause a false-positive failure and trigger an unnecessary restart cascade. Best practices dictate that the probe handler runs on a dedicated, high-priority thread and returns a simple boolean or HTTP status code. Avoid logging, complex serialization, or lock acquisition within the probe path.

Misconfigured probe thresholds are a leading cause of flapping—where an agent is repeatedly killed and restarted. Key parameters include:

• initialDelaySeconds: Time to wait before the first probe, allowing the agent to initialize.
• periodSeconds: Frequency of the check (e.g., every 10 seconds).
• failureThreshold: Number of consecutive failures before a restart is triggered (e.g., 3).
• timeoutSeconds: Maximum time to wait for a probe response. Setting the timeoutSeconds too low on a garbage-collection-heavy agent can cause false restarts.

A critical architectural distinction exists between liveness and readiness probes. A liveness probe failure results in a hard restart—the container is killed and recreated. A readiness probe failure simply removes the pod from the service load balancer, allowing it to recover without termination. Confusing the two is dangerous: if a temporary database connection blip triggers a liveness probe failure, the entire agent restarts, causing cascading state loss. Liveness probes should only fail on irrecoverable internal states.

In sidecar architectures, a dedicated watchdog process can act as the liveness probe target. The main agent process writes a heartbeat to a shared file or Unix socket, and the sidecar exposes the /live endpoint. If the heartbeat stops updating, the sidecar returns a 503, and the orchestrator restarts the entire pod. This pattern decouples the probe logic from the agent code and prevents a deadlocked agent from blocking its own health reporting. Tools like supervisord or custom Go routines implement this pattern.