Health Check API

These are the two fundamental types of checks performed by a Health Check API. A Liveness Probe determines if an agent's process is running (e.g., can the operating system schedule it?). A Readiness Probe determines if the agent is fully initialized and ready to accept work (e.g., are dependencies loaded, is the network stack ready?). Distinguishing between these states prevents sending traffic to agents that are alive but not yet ready.

A robust Health Check API returns a consistent, machine-readable data structure. This typically includes:

• Status: A simple indicator like UP, DOWN, STARTING, or OUT_OF_SERVICE.
• Components: A nested breakdown of sub-system health (e.g., database connection, disk space, sensor status).
• Metrics: Key performance indicators like current CPU load, memory usage, or queue depth.
• Timestamp: The time of the check for synchronization and drift detection. This schema allows the orchestrator to make automated decisions without parsing unstructured text.

Health checks operate at multiple levels of granularity to provide a complete picture:

• System-Level: Basic OS vitals (CPU, memory, disk I/O).
• Process-Level: Application runtime status.
• Service-Level: Business logic and internal module health.
• Dependency-Level: Status of external connections (databases, message brokers, other microservices). The API aggregates these into a composite health status, enabling precise root cause isolation—distinguishing a local disk failure from a network partition.

The API endpoint must be designed for frequent, low-overhead polling. Key configuration parameters include:

• Check Interval: How often the orchestrator polls the endpoint (e.g., every 5 seconds).
• Timeout: The maximum wait time for a response before considering the check failed.
• Success/Failure Thresholds: The number of consecutive successful or failed checks required to change the agent's overall health state. This provides hysteresis, preventing flapping status due to transient network glitches.

Beyond a binary UP/DOWN, a sophisticated Health Check API can signal graceful degradation. For example, an agent might report a status of DEGRADED with details indicating:

• High latency on a non-critical sensor.
• Reduced battery capacity affecting maximum operational time.
• Operating in a fallback mode due to a secondary system failure. This allows the orchestrator to make intelligent routing decisions, such as assigning less critical tasks to the degraded agent while scheduling maintenance.

Health check results are a primary telemetry source. The API integrates with broader observability systems by:

• Emitting structured log events on state transitions (e.g., HEALTH_STATE_CHANGE).
• Publishing metrics (like health check duration) to time-series databases (e.g., Prometheus).
• Triggering alerts in monitoring systems (e.g., PagerDuty, OpsGenie) when a critical component fails. This creates a closed-loop system where health checks drive both automated orchestration and human operator awareness.

A periodic status message sent autonomously by an agent to a central monitor to affirm it is alive and responsive. The absence of expected heartbeats within a defined timeout window is the primary trigger for downtime detection. This is a push-based mechanism, contrasting with the pull-based query of a Health Check API.

• Key Mechanism: Unidirectional, time-based signaling.
• Primary Use: Liveness detection and network connectivity verification.
• Failure Mode: Missed heartbeats indicate a crash, freeze, or network partition.

An active diagnostic check performed by an orchestration system to determine if an agent's core process is running. It answers the question, "Is the container/process alive?" A liveness probe failure typically triggers a restart of the agent instance. This is a specific, often simpler, type of query that a Health Check API might expose.

• Typical Check: Can the agent accept a TCP connection or respond to an HTTP GET request?
• Orchestrator Action: Restart the unhealthy pod or container.
• Contrast with Readiness: Liveness checks for crashes; readiness checks for startup and dependencies.

A diagnostic check to determine if an agent is fully initialized and ready to accept work or network traffic. An agent may be "live" (process running) but not "ready" if it is still loading dependencies, connecting to a database, or warming up its internal state. A failed readiness probe typically removes the agent from a load balancer pool.

• Typical Check: Can the agent handle a specific API request that validates internal dependencies?
• Orchestrator Action: Stop sending traffic to the agent until it reports ready.
• Critical for: Preventing traffic from being sent to agents that cannot yet process it successfully.

A hardware or software timer that must be periodically reset ("petted") by a healthy agent. If the timer expires, it assumes the agent's software has hung and triggers a hardware reset or a failover to a backup system. This is a last-resort safety mechanism for detecting and recovering from catastrophic freezes that a Health Check API might not be able to report.

• Implementation Level: Often low-level, closer to the hardware or OS kernel.
• Recovery Action: Forced reboot or system reset.
• Use Case: Critical for embedded systems and robotics where a software hang could cause physical damage or unsafe conditions.

A continuous, push-based flow of operational data (metrics, logs, events) from agents to a central collection system. While a Health Check API provides a snapshot on demand, a telemetry stream offers a real-time, historical view of system behavior for monitoring and analysis. Health check results themselves are often emitted as part of this stream.

• Data Types: CPU usage, memory, custom business metrics, error rates, state changes.
• Protocols: Often uses MQTT, gRPC streams, or custom UDP/TCP protocols.
• Purpose: Enables real-time dashboards, historical trend analysis, and feeding data into anomaly detection systems.

A composite, weighted numerical value (e.g., 0-100) that summarizes the overall operational status of an agent or system. It is derived by aggregating and scoring the results of multiple underlying checks from a Health Check API, telemetry metrics, and probe results. This abstraction simplifies operational dashboards and enables automated alerting based on threshold breaches.

• Calculation: Often uses a formula incorporating criticality weights for different subsystems (e.g., navigation, comms, battery).
• Visualization: Displayed as a color-coded gauge or score in fleet management UIs.
• Automation: Triggers maintenance tickets or alerts when the score falls below a defined SLO threshold.