Liveness Probe

A liveness probe's primary function is to answer a simple binary question: Is the target process alive? It does this by executing a predefined check, such as:

• Sending an HTTP request to a specific endpoint.
• Opening a TCP socket connection.
• Running a command inside the agent's container or process. A successful response indicates the process is running; a failure or timeout triggers a failure state. This is distinct from a readiness probe, which checks if the agent is ready to perform work.

Liveness probes are defined by several key parameters that control their behavior and sensitivity:

• Initial Delay: Time to wait after the agent starts before initiating probes.
• Period: How often (e.g., every 10 seconds) to execute the probe.
• Timeout: Maximum time allowed for the probe to complete before it's considered a failure.
• Success/Failure Thresholds: The number of consecutive successes or failures required to change the agent's state.
• Probe Type: HTTP GET, TCP Socket, or Exec command. Proper tuning of these parameters is essential to avoid false positives from slow-starting processes or transient network issues.

When a liveness probe fails repeatedly, the orchestration system initiates a recovery action. The most common action is to restart the container or process. In a robotics context, this might involve:

• Issuing a software reboot command to the agent.
• Failing over tasks to a healthy peer agent.
• Placing the agent in a quarantine state for manual inspection.
• Triggering a watchdog timer reset if hardware is involved. This automated response is fundamental to achieving high system availability without constant human intervention.

The output of liveness probes feeds directly into the system's fleet-wide view. A failed probe changes the agent's health status, which is aggregated into dashboards and alerts. This status is a key input for:

• Dynamic task allocation systems, which avoid assigning work to unhealthy agents.
• Load balancing algorithms that redirect traffic.
• Health score calculations that summarize overall fleet vitality. This integration ensures the orchestration layer has a real-time, accurate picture of operational capacity.

A well-designed liveness probe must be lightweight and have minimal performance impact on the agent it is monitoring. Best practices include:

• Using a dedicated, low-cost endpoint (e.g., /health/live).
• Avoiding complex logic or dependencies on external services within the check.
• Ensuring the check tests core process functionality without consuming significant CPU, memory, or I/O. A heavy probe can itself cause resource contention and performance degradation, defeating its purpose.

Liveness probes are one part of a holistic health-check strategy and must be distinguished from related mechanisms:

• Readiness Probe: Determines if an agent is ready to accept work (e.g., dependencies loaded, cache warm). An agent can be live but not ready.
• Startup Probe: Used for legacy applications with long initialization times; disables liveness/readiness checks until the app has started. In practice, all three probe types are often used together to manage an agent's lifecycle precisely, from startup through readiness to continuous liveness verification.

Within a service mesh like Istio or Linkerd, liveness probes are used by the data plane proxies (sidecars) to determine the health of the attached application container. This ensures traffic is only routed to live instances. The mesh control plane may also use aggregated liveness data for cluster-wide health views and to inform load-balancing decisions, preventing requests from being sent to unresponsive endpoints.

In heterogeneous fleet orchestration, a liveness probe monitors the core software processes on each robot. It checks if the agent's main control loop, perception stack, and communication daemon are running. A failure triggers an immediate graceful stop command to prevent unsafe motion, logs the incident for root cause analysis, and may initiate a failover to a backup controller or schedule the robot for manual inspection, ensuring safety in dynamic physical environments.

For edge-deployed models and IoT devices, liveness probes verify the operational state of the inference service or device management agent. This is critical for remote diagnostics and maintaining a fleet-wide view. A persistent liveness failure can trigger over-the-air (OTA) update rollbacks, alert human operators, or mark the device for predictive maintenance scheduling, preventing silent failures in distributed networks.

Liveness probes for stateful services like databases (PostgreSQL, Redis) or message brokers (Kafka) must be carefully designed. A simple TCP check confirms the process is listening, but a poorly configured deep check (e.g., a complex query) can cause cascading failures under load. Best practice is a shallow check paired with a separate readiness probe. Failure typically triggers a restart, but for stateful systems, this is coordinated with replication and failover mechanisms to avoid data loss.

For batch jobs, scientific simulations, or training pipelines that run for hours/days, a liveness probe ensures the process hasn't hung. Instead of restarting on failure, the outcome is often alerting. The probe might check for progress updates in a log file, heartbeat files on disk, or incremental updates to a metrics endpoint. This distinguishes between a slow, healthy process and a dead one, enabling intervention without losing expensive intermediate computation.

A diagnostic check to determine if an agent or service is fully initialized and ready to accept work or traffic, beyond merely being alive. While a liveness probe answers "Is it running?", a readiness probe answers "Is it ready to serve?"

• Purpose: Prevents the orchestrator from sending traffic to an agent that is booting, loading large datasets, or establishing external connections.
• Example: A mobile robot's readiness probe might check that its localization system has acquired a fix, its safety scanners are active, and its task queue API is listening.
• Key Difference: An agent failing a liveness probe is typically restarted. An agent failing a readiness probe is simply removed from the available pool until it passes.

A periodic, push-based status message sent by an agent to a central monitor to indicate it is alive and functioning. The absence of expected heartbeats triggers downtime detection.

• Mechanism: The agent proactively emits a signal (e.g., a UDP packet, a message on a pub/sub channel) at a fixed interval (e.g., every 5 seconds).
• Contrast with Liveness Probe: A heartbeat is push (agent → monitor), while a liveness probe is pull (monitor → agent). Heartbeats are simpler but can fail if the agent's network outbound path is broken.
• Use Case: Ideal for lightweight, constant affirmation of aliveness in large fleets where the overhead of frequent pull requests from a central system is undesirable.

A hardware or software timer that resets a system or triggers a failover if not periodically refreshed by a healthy agent. It is a low-level mechanism to detect and recover from process hangs or infinite loops.

• Operation: The agent's software must regularly "kick" or "pet" the watchdog. If the kicking stops (due to a software freeze), the timer expires and triggers a hardware reset or alerts a supervisor process.
• Level of Defense: Acts as a last-resort safety net when higher-level liveness probes (which run in user space) may also be frozen.
• Robotics Application: Critical for autonomous mobile robots (AMRs) where a software lock-up could lead to a physical safety hazard; the watchdog forces a reboot.

A programmatic interface (typically HTTP/gRPC) that allows an orchestration system to query the operational status of an agent. A liveness probe is one specific type of request made to this API.

• Endpoint: Often exposed as a path like /healthz or /status.
• Comprehensive Response: May return a structured JSON payload containing multiple sub-statuses: liveness, readiness, battery level, component diagnostics, and version information.
• Standardization: Enables a heterogeneous fleet (different robot brands, manual forklifts with IoT kits) to present a unified health interface to the central orchestrator, abstracting their internal complexity.

A stability design pattern that prevents a system from repeatedly trying to execute an operation that's likely to fail (like calling an unhealthy agent). It allows the system to fail fast and recover gracefully.

• Analogy: Like an electrical circuit breaker. After a threshold of failures, the "circuit" opens and further requests are immediately rejected for a period, allowing the downstream service (the agent) time to recover.
• Relation to Probes: The circuit breaker's state (open/closed/half-open) can be informed by liveness and readiness probe results. It protects the orchestrator from wasting resources on known-unhealthy agents.
• Benefit: Prevents cascading failures and resource exhaustion in the orchestrator when multiple agents are experiencing issues.

A system design principle where an agent or service maintains partial, reduced functionality in the face of partial failures, rather than failing completely. Health probes inform the level of degradation.

• Example: An AMR's liveness probe is successful, but its readiness probe fails because its primary camera is offline. The orchestrator could still assign it tasks that do not require visual perception (e.g., simple transit).
• Fleet Impact: Enables the overall fleet to continue operating at a reduced capacity instead of a total halt. The orchestrator's task allocator must understand agent capabilities derived from health checks.
• Contrast with Failover: Degradation keeps the impaired unit in service with limited function, while failover switches to a redundant unit.