Readiness Probe

A readiness probe specifically checks if an agent is ready to perform its primary function, while a liveness probe only confirms the process is running. This distinction is critical for heterogeneous fleets where agents may be 'alive' but not yet operational due to:

• Initialization sequences (e.g., loading maps, calibrating sensors, connecting to databases).
• Dependency availability (e.g., waiting for a central planning service or a required API).
• Warm-up periods (e.g., a machine learning model loading into memory). A successful readiness check signals the orchestration platform that the agent can be added to the available pool for task assignment.

Readiness probes are implemented through programmatic checks that test specific functional capabilities. Common types include:

• HTTP GET Probe: The orchestrator sends an HTTP request to a designated endpoint (e.g., /health/ready) on the agent. A 2xx or 3xx response indicates readiness.
• TCP Socket Probe: The orchestrator attempts to open a TCP connection to a specified port on the agent. Success indicates the required service is listening.
• Command Execution Probe: The orchestrator executes a command inside the agent's container or process (e.g., a script that validates internal state). A zero exit code indicates success.
• Custom gRPC Probe: For agents using gRPC, a check is made against a defined service method to verify readiness. The probe must be lightweight, fast, and have minimal external dependencies to avoid creating a false failure cascade.

To handle varying startup times and transient issues, readiness probes are configured with tunable parameters that define their behavior:

• Initial Delay Seconds: The time to wait after the agent starts before initiating the first probe. This accommodates baseline boot time.
• Period Seconds: How often (e.g., every 10 seconds) the probe is executed after the initial delay.
• Timeout Seconds: The maximum time allowed for the probe to complete before it is considered a failure.
• Success Threshold: The number of consecutive successful probes required to transition an agent from an unready to a ready state. Often set to 1.
• Failure Threshold: The number of consecutive probe failures required to transition an agent from ready to unready. A value greater than 1 (e.g., 3) prevents flapping due to transient network glitches.

The readiness probe's result directly controls traffic routing and task allocation in the orchestration layer.

• Service Mesh & Ingress Controllers: A failing readiness probe causes the agent's endpoint to be removed from the load balancer pool, preventing new work or network traffic from being routed to it.
• Task Schedulers: In platforms like Kubernetes or robotic fleet managers, a pod or agent in an 'Unready' state will not receive scheduled tasks, even if its liveness probe passes.
• Graceful Shutdown Support: During termination, the orchestrator marks the agent as 'Not Ready' (often via a separate endpoint or signal) to stop new task assignment, while allowing in-progress tasks to complete before shutting down the process.

An effective readiness probe validates that all critical internal and external dependencies required for work are available. This often involves composite checks that go beyond a simple port check. Examples include:

• Verifying connectivity to a required message broker (e.g., RabbitMQ, ROS 2 middleware).
• Confirming a valid lease on a required resource or zone within a warehouse.
• Checking that a local cache of operational data (e.g., a digital twin map) is loaded and current.
• Validating that essential hardware (e.g., LiDAR, gripper) responds to a diagnostic command. The probe should fail if any of these core dependencies are unavailable, ensuring the agent does not accept work it cannot complete.

A readiness probe failure triggers defined remediation workflows to maintain overall fleet health. Common Responses:

1. Mark Unready: The agent is taken out of the available pool. No new tasks are assigned.
2. Logging & Alerting: The failure generates a diagnostic event, often tagged with the probe type and error, for the telemetry stream.
3. Restart Policies: If coupled with a failing liveness probe, the orchestration platform may restart the agent's container or process based on a defined policy.
4. Escalation: Persistent failures may trigger alerts for remote diagnostics or predictive maintenance systems, indicating a deeper hardware or software issue. This mechanism is foundational for achieving graceful degradation at the fleet level, as unhealthy agents are automatically isolated.

The primary use case is verifying that all internal dependencies are active before an agent accepts tasks. For an Autonomous Mobile Robot (AMR), this means checking that its motor controllers, localization system (e.g., SLAM), and safety laser scanners have completed their boot sequence and calibration. A probe failing here prevents the robot from being scheduled, avoiding scenarios where it receives a navigation command before it knows its own position.

For software agents managing logistics, a readiness probe ensures critical data structures are loaded. This includes:

• Verifying connections to a Redis cache or PostgreSQL database.
• Confirming that in-memory route graphs or inventory maps are fully populated.
• Checking that gRPC or WebSocket connections to the central orchestrator are established. A service is only marked 'ready' after these data dependencies are satisfied, preventing queries to empty caches.

During a fleet-wide Over-the-Air (OTA) update, new agent software versions are deployed incrementally. The orchestrator uses the readiness probe to manage this rollout:

• It starts new agent instances but does not route traffic to them until their probe succeeds.
• Only then does it drain traffic from old instances and terminate them. This ensures zero-downtime deployments and prevents new, unready instances from causing cascading failures if they receive work prematurely.

A readiness probe can check the health of external, non-critical dependencies that are required for full functionality. For example, a telemetry agent might:

• Verify it can publish metrics to a metrics pipeline like Prometheus.
• Confirm it can reach a secondary logging service. If these external services are temporarily unavailable, the probe can be configured to fail, signaling the agent should not yet be considered operational, thus maintaining data integrity.

A liveness probe answers "Is the process running?" (e.g., can it respond to a ping). A readiness probe answers "Is it fully operational?" In practice:

• A failed liveness probe causes the orchestrator to restart the container or agent process.
• A failed readiness probe causes the orchestrator to temporarily remove the agent from the load balancer or task scheduler. This distinction is critical for handling slow-starting services without unnecessary restarts.

Probes are typically implemented as a lightweight HTTP endpoint, a TCP socket check, or a command execution. Key configuration parameters include:

• Initial Delay: Time to wait after startup before starting probes.
• Period: How often to execute the probe (e.g., every 10 seconds).
• Timeout: How long to wait for a response.
• Success/Failure Threshold: Number of consecutive passes/failures required to change status. Proper tuning prevents false positives during normal resource contention.

A diagnostic mechanism used to determine if an agent or service process is running and alive, typically by checking if it can respond to a simple request (e.g., a ping or TCP connection). It is a more basic check than a readiness probe, answering "Is it up?" rather than "Is it ready?"

• Key Difference: A liveness probe failure typically triggers a process restart, while a readiness probe failure triggers traffic isolation.
• Use Case: Detecting a process that is running but stuck in an infinite loop, which would fail a liveness check.

A periodic status message sent by an agent to a central monitor to indicate it is alive and functioning. The absence of expected heartbeats triggers downtime detection and alerts.

• Mechanism: Usually a lightweight, time-stamped message sent over a network at regular intervals (e.g., every 5 seconds).
• Purpose: Provides continuous proof of life and basic network connectivity, forming the foundation for fleet-wide view dashboards.

A programmatic interface (typically a REST endpoint) that allows an orchestration system to query the operational status of an agent. It often aggregates results from multiple internal checks, including liveness, readiness, and custom diagnostics.

• Response: Returns a structured payload (e.g., JSON) with an overall status code (HTTP 200/503) and detailed component health.
• Integration: The primary endpoint polled by external orchestration middleware to make routing and failover decisions.

A composite numerical value (e.g., 0-100) that summarizes the overall operational status of an agent or system. It is derived from a weighted aggregation of multiple underlying metrics and probe results.

• Components: May incorporate readiness probe results, battery State of Charge (SoC), error rates, and custom business logic.
• Utility: Enables priority-based routing and load balancing by providing a single, comparable metric for fleet-wide agent selection.

A system design principle where an agent or service maintains partial, reduced functionality in the face of partial failures, rather than failing completely. Readiness probes are a key enabler.

• Implementation: A readiness probe may fail if a non-critical subsystem (e.g., a high-resolution camera) is offline, while core mobility functions remain operational.
• Benefit: Allows the fleet to continue operating at a reduced capacity while exception handling frameworks schedule repairs.

The capability of an agent to automatically test and validate its own internal components and software to detect faults. A readiness probe is often the external manifestation of internal self-diagnostic routines.

• Scope: Can include checks for sensor calibration, motor current draw, memory leaks, and software dependency availability.
• Output: Results feed into the health check API and contribute to the agent's health score and telemetry stream.