Liveliness Probe

A liveliness probe operates proactively by periodically checking the agent's health, rather than waiting for a user request to fail. This allows the orchestration system (e.g., Kubernetes) to detect and remediate issues like deadlocks, infinite loops, or memory leaks before they impact end-users. The probe's frequency is configurable, balancing detection speed against system load.

• Example: A Kubernetes pod running an LLM agent might have an HTTP livenessProbe hitting a /health endpoint every 10 seconds.

A probe's behavior is defined by its action type and failure thresholds. Common actions include:

• HTTP GET: Checks for a 2xx or 3xx response from a specified endpoint.
• TCP Socket: Attempts to open a TCP connection to a specified port.
• Command Execution: Runs a shell command inside the agent's container; a zero exit code indicates success.

The failureThreshold determines how many consecutive probe failures must occur before the system declares the agent unhealthy and triggers a restart.

The primary remedial action for a failed liveliness probe is a container restart. The orchestrator terminates the unresponsive agent process and instantiates a new one from the same image. This is based on the assumption that a fresh start will clear any transient state causing the hang. This mechanism is distinct from a readiness probe, which controls traffic routing but does not trigger restarts.

An effective liveliness check should test the agent's core process with minimal dependencies. It should not rely on the availability of downstream databases, external APIs, or network filesystems. A probe that fails due to a downstream outage could cause unnecessary restarts of a healthy agent. The check should be a lightweight, internal verification of process aliveness and basic functionality.

It is critical to distinguish a liveliness probe from a readiness probe.

• Liveliness: "Is the process alive and functional?" Failure → Restart.
• Readiness: "Is the process ready to accept traffic?" Failure → Remove from load balancer. An agent may be "live" but not "ready" during initial startup while it loads large models or connects to essential services. Using both probes together ensures traffic is only sent to fully initialized agents.

A liveliness probe often works in tandem with an internal agent heartbeat signal. While the probe is an external check performed by the orchestrator, a heartbeat is an internal, periodic signal emitted by the agent itself to a monitoring system. A missing heartbeat can be a leading indicator for a probe failure. Together, they provide a dual-layer approach to failure detection from both outside and inside the agent's runtime environment.

A readiness probe is a health check that determines if an agent has completed its initialization and is prepared to accept work. Unlike a liveliness probe, which asks "Is the process alive?", a readiness probe asks "Is the agent ready?"

• Key Difference: A failed readiness probe typically prevents new traffic from being routed to the agent but does not trigger a restart.
• Common Checks: Verifies connections to databases, vector stores, external APIs, and that internal caches (like a KV Cache) are populated.
• Use Case: In Kubernetes, a pod passes its readiness probe before being added to a Service's load-balancing pool.

An agent heartbeat is a periodic, proactive signal emitted by an agent to a monitoring system to affirm it is operational. It is a push-based mechanism, whereas a liveliness probe is typically a pull-based check.

• Implementation: Often a simple message published to a message queue or a timestamp written to a shared state persistence layer.
• Failure Detection: The absence of expected heartbeats within a time window indicates an unresponsive or crashed agent.
• Telemetry Integration: Heartbeat intervals and metadata (e.g., session state ID) are core agent telemetry pipeline signals.

State checkpointing is the periodic or conditional saving of an agent's complete operational state to durable storage. This creates recovery points that enable state rollback or state rehydration after a failure detected by a liveliness probe.

• Mechanism: Involves serializing in-memory state (context, intermediate reasoning) and persistent state to a snapshot.
• Efficiency: Advanced systems use state deltas (incremental changes) instead of full snapshots to reduce overhead.
• Recovery Link: A restart triggered by a failed liveliness probe will often load the most recent checkpoint to resume work.

Degraded mode is an operational state where an agent continues to function with reduced capability after a partial dependency failure. A sophisticated health check system might report a "degraded" status instead of failing a liveliness probe outright.

• Trigger: Loss of a non-critical external service (e.g., a secondary LLM provider, enhanced logging).
• Behavior: The agent may disable specific tool calling capabilities or fall back to simpler algorithms while core functions remain alive.
• Observability: Entering degraded mode is a key event for agent behavior auditing and agentic anomaly detection systems.

Deadlock detection is the monitoring process that identifies when an agent is permanently blocked, waiting for a condition that will never be satisfied. A simple liveliness probe (e.g., an HTTP endpoint) may still respond, but the agent is functionally stuck.

• Beyond Liveliness: Requires deeper agent reasoning traceability or execution trace analysis to detect loops or unresolvable waits.
• Common Causes: Circular dependencies in multi-agent system orchestration, or an agent waiting for its own output.
• Resolution: Often requires intervention, state rollback, or injecting a resolution command, as a restart alone may not solve the underlying logical issue.

Failover state is the pre-configured data and context maintained on a standby replica so it can rapidly assume the workload of a primary agent that fails its liveliness probes.

• Synchronization: Involves continuous or periodic replication of the primary agent's session state and conversation context to the standby.
• Hot vs. Warm Standby: A hot standby has nearly identical in-memory state loaded; a warm standby requires state rehydration from a recent checkpoint.
• Orchestration: Managed by platforms like Kubernetes (with pod anti-affinity) or custom multi-agent observability controllers.