Agent Health Check

Health checks are categorized by their purpose. A liveness probe determines if an agent is running. A failed liveness probe typically triggers a restart. A readiness probe determines if an agent is ready to accept work (e.g., has loaded its model, connected to dependencies). A failed readiness probe removes the agent from the service pool but does not restart it. This distinction prevents routing traffic to agents that are alive but not yet operational.

Health checks are executed by the orchestrator (e.g., Kubernetes kubelet) using one of three primary mechanisms:

• HTTP GET: The orchestrator sends an HTTP request to a specified endpoint on the agent; a success code (2xx-3xx) passes the check.
• TCP Socket: The orchestrator attempts to open a TCP connection to a specified port on the agent; success is establishing the connection.
• Exec Command: The orchestrator executes a command inside the agent's container; a zero exit code indicates success. The probe's periodSeconds, timeoutSeconds, and failureThreshold parameters define its timing and sensitivity.

Health checks are the primary trigger for agent self-healing. When a liveness probe fails consecutively, the orchestration system's control loop initiates a corrective action. This is a core tenet of declarative configuration: the system continuously reconciles the actual state (failed agent) with the desired state (healthy agent). Corrective actions include restarting the agent container, rescheduling the pod to a new node, or, in stateful systems, triggering a failover to a replica.

Health check design differs for stateful and stateless agents. For stateless agents, a simple endpoint check is often sufficient. For stateful agents, the probe must be aware of the agent's internal state. A poorly designed check for a stateful agent (e.g., one performing a long database transaction) could cause unnecessary restarts and data corruption. Probes for stateful agents should verify the integrity of the agent's core state management loop without being overly intrusive or blocking critical operations.

An effective health check evaluates not just the agent process, but its critical dependencies. A readiness probe should fail if the agent cannot connect to its vector database, LLM API, or message broker. However, the check must be scoped carefully. It should not fail for transient issues with non-critical, external services the agent can temporarily operate without. This design ensures the orchestrator only marks an agent as 'not ready' when it truly cannot perform its core function.

Health checks introduce overhead. Frequent, complex probes consume CPU cycles and network bandwidth. A poorly configured probe (e.g., a 1-second HTTP check on a computationally intensive endpoint) can degrade agent performance. The initialDelaySeconds parameter is critical to prevent checks from running before the agent has finished initializing. The goal is to find a balance between detection speed and system load, often starting with conservative intervals (e.g., 10-30 seconds) and adjusting based on observed latency.

An orchestration capability where the system automatically detects agent failures (via health checks) and takes corrective action. This is the primary purpose of a health check system.

• Corrective Actions: Can include restarting the agent, rescheduling it to a healthy node, or triggering a failover to a standby instance.
• Automated Remediation: Transforms a monitoring signal (failed health check) into an automated operational response, reducing manual intervention.
• Example: A Kubernetes pod with a failed liveness probe is automatically terminated and a new one is created by the ReplicaSet controller.

The automated collection and transmission of operational data from an agent for observability. While a health check provides a binary status (healthy/unhealthy), telemetry offers granular, continuous metrics.

• Key Data: Includes CPU/memory usage, request latency, error rates, and custom business metrics.
• Proactive vs. Reactive: Telemetry enables proactive anomaly detection (e.g., memory leak trend), whereas health checks are often reactive (process is dead).
• Integration: Health check endpoints often expose a subset of telemetry data (e.g., /health returning app-specific metrics).

A control loop that continuously observes the actual state of agent resources and acts to align them with the declared desired state. Health checks are a critical input to this loop.

• Controller Pattern: The core mechanism behind operators and orchestration platforms like Kubernetes.
• Input Signal: A failed health check changes the "actual state," triggering the reconciliation logic.
• Action: The loop executes the logic defined in the desired state, such as "ensure 3 replicas are healthy," leading to a pod restart or rescheduling.

A Kubernetes policy that limits voluntary disruptions to agent pods during maintenance. It works in concert with health checks to ensure high availability.

• Voluntary Disruptions: Actions like node drains, cluster upgrades, or manual pod deletions.
• Function: Guarantees a minimum number or percentage of pods from a deployment/statefulset remain available.
• Interaction with Health: The orchestrator respects the PDB only for pods that pass their readiness checks. Pods failing health checks may be evicted outside the PDB's protection.

The controlled shutdown process for an agent, allowing it to complete tasks and persist state. A proper health check system must distinguish between a graceful termination and a failure.

• PreStop Hook: Often used to run a cleanup script before the container runtime sends the termination signal.
• Readiness Probe: Should start failing as soon as a termination signal is received, so the agent is removed from service load balancers.
• Liveness Probe: Must remain passing during the graceful shutdown period to prevent the orchestrator from forcing an immediate kill.

A dedicated infrastructure layer for managing communication between agents. It often implements and enhances health checking capabilities beyond the basic orchestration layer.

• Advanced Health Checks: Can perform protocol-specific checks (e.g., gRPC health checks) and synthetic transactions.
• Traffic Management: Automatically routes traffic away from instances failing health checks (circuit breaking).
• Unified Observability: Provides a centralized view of health status across all services in the mesh, distinct from pod-level liveness.