Readiness Probe

Readiness probes differ fundamentally from liveness probes in their lifecycle scope. A readiness probe is only relevant during initialization and degradation events—it does not run continuously to check if the process is alive. Once an agent passes its readiness check, the probe may be polled periodically to detect regression into a non-ready state, but its primary function is traffic gating, not crash detection.

• Startup phase: Agent loads models, establishes database connections, warms caches
• Steady-state phase: Probe confirms continued ability to serve requests
• Degradation phase: Probe fails, orchestrator stops routing traffic

A readiness probe endpoint must be idempotent and side-effect-free. Each probe invocation should return the same result given the same internal state, without modifying that state. A poorly designed probe that increments counters, writes to databases, or consumes queue messages introduces probe-induced load that can cascade into system degradation.

• Use GET requests exclusively—never POST or DELETE
• Avoid logging each probe invocation at INFO level
• Cache probe results for short intervals (100-500ms) to handle burst polling

A robust readiness probe validates transitive dependencies, not just the agent process itself. An agent whose HTTP server is listening but whose vector database connection pool is exhausted is not truly ready. The probe should perform lightweight, non-blocking checks against critical downstream services.

• Database: Execute SELECT 1 or ping connection pool
• Message broker: Verify topic/queue existence and publish permissions
• Model server: Confirm model is loaded and inference endpoint responds
• File system: Check read/write access to required mount points

Caution: Avoid deep dependency chains that make the probe slow or brittle. Each dependency check should have a sub-100ms timeout.

Orchestrators like Kubernetes use configurable failure thresholds to prevent flapping. A single probe failure should not immediately remove an agent from the service pool. Instead, configure failureThreshold (e.g., 3 consecutive failures) and periodSeconds to create a debounce window that tolerates transient blips.

• initialDelaySeconds: Time to wait before first probe (accommodates slow startups)
• periodSeconds: Frequency of probe execution (typically 5-10 seconds)
• failureThreshold: Consecutive failures before marking NotReady (typically 3)
• successThreshold: Consecutive successes before marking Ready after failure (typically 1)

Misconfiguration here causes premature pod termination or zombie endpoints receiving traffic they cannot serve.

When an agent receives a SIGTERM signal, it must immediately fail its readiness probe to initiate traffic draining before process termination. The orchestrator removes the endpoint from the service pool upon probe failure, allowing in-flight requests to complete during the terminationGracePeriodSeconds window.

• Step 1: Agent catches SIGTERM
• Step 2: Readiness endpoint returns 503 Service Unavailable
• Step 3: Load balancer stops routing new connections
• Step 4: Agent completes in-flight work
• Step 5: Process exits cleanly

Without this coordination, terminated agents drop active requests, causing user-facing errors.

Generic "200 OK" responses are insufficient for agents with complex initialization sequences. A readiness probe should expose granular status that reflects domain-specific readiness criteria. An agent may be "alive" but not "ready" if its tool authorization tokens have not yet been refreshed or its policy engine is still compiling rules.

• 200: Agent is fully ready to accept work
• 503: Agent is initializing or degraded—do not route traffic
• Response body: Optional JSON with {"status": "not_ready", "reason": "model_loading", "progress_pct": 72}

This structured response enables operator dashboards and automated alerting on prolonged non-ready states.

A diagnostic check that determines if an agent's core process is running but unresponsive—a deadlock state where the process exists but cannot make progress. While a readiness probe checks initialization, a liveness probe detects runtime hangs. If a liveness check fails, the orchestrator typically restarts the agent rather than just withholding traffic.

• Checks for deadlocked event loops or blocked threads
• Failure triggers a SIGTERM and process restart
• Often implemented as a simple /healthz HTTP endpoint

A specialized diagnostic for agents with slow initialization sequences, such as loading large models or establishing database connection pools. Startup probes run before readiness probes and prevent the orchestrator from killing an agent that is legitimately still booting.

• Configured with a longer failureThreshold than readiness probes
• Protects agents loading multi-gigabyte model weights
• Once the startup probe succeeds, readiness probes take over

A resilience pattern that prevents an agent from repeatedly calling a failing downstream dependency. When failure thresholds are exceeded, the circuit breaker trips and immediately fails all calls for a cooldown period. This prevents cascading failures and gives the downstream service time to recover.

• States: Closed (normal), Open (failing fast), Half-Open (testing recovery)
• Prevents resource exhaustion from retry storms
• Complements readiness probes by protecting external calls

A design strategy where an agent maintains limited but safe functionality when a component fails, rather than crashing entirely. A readiness probe can be configured to report degraded but available status, allowing the agent to serve a reduced feature set while non-critical dependencies are restored.

• Example: An agent serves cached results when its vector database is unreachable
• Requires explicit fallback paths in agent logic
• Contrasts with fail-closed configurations that block all access

A hardware or software timer that monitors an agent's operation and triggers a corrective action—typically a system reset—if the agent fails to periodically pet the dog by signaling it is functioning correctly. Unlike a liveness probe which is polled externally, a watchdog is an internal failsafe.

• Common in embedded and safety-critical systems
• Requires the agent to actively write to a kick register
• Failure triggers a hardware-level reset, not just a process restart

A pre-defined, secure condition that an autonomous system automatically enters upon detecting a critical malfunction. When a readiness probe fails, the system should transition to a fail-safe state that minimizes potential damage—for example, a robotic arm moving to a neutral position or an API gateway returning cached responses.

• Designed during threat modeling sessions
• Must be deterministic and independently verifiable
• Contrasts with fail-secure states that prioritize data protection