Circuit Breaker

A circuit breaker operates as a deterministic finite state machine with three distinct states:

• Closed: Requests flow normally to the downstream dependency. A failure counter tracks consecutive errors.
• Open: After a configurable failure threshold is breached, the breaker trips. All subsequent requests are immediately rejected without attempting the downstream call, conserving resources.
• Half-Open: After a cooldown period expires, a limited number of probe requests are permitted. If they succeed, the breaker resets to Closed; if they fail, it reverts to Open. This prevents the orchestrator from wasting cycles on a known-faulty agent driver or protocol adapter.

The breaker does not trip on a single transient error. It monitors for sustained failure patterns:

• Consecutive Failure Count: A simple counter that increments on each failure and resets on a single success. A threshold of 5 consecutive failures is a common starting point.
• Failure Rate in Sliding Window: A more sophisticated approach that tracks the ratio of failures to total requests over a rolling time window (e.g., the last 60 seconds). The breaker opens if the error rate exceeds 50%.
• Slow Call Detection: Treats requests exceeding a defined latency percentile (e.g., 99th percentile > 2 seconds) as failures, protecting against degraded but not fully dead dependencies.

When a circuit is Open, the orchestrator must not simply fail. A robust implementation provides a fallback strategy:

• Cached Response: Return the last known valid response from a local cache, suitable for read operations against a state synchronization endpoint.
• Default Value: Return a safe, pre-configured default that allows the workflow to degrade gracefully.
• Alternative Service: Redirect the request to a redundant instance or a different agent driver providing equivalent capability.
• Queue for Retry: Persist the command to a command queue for deferred execution when the circuit closes. Without a fallback, the circuit breaker merely converts a timeout failure into an immediate failure.

Circuit breakers must be applied with granular scope to prevent a single faulty dependency from blocking unrelated operations:

• Per-Dependency Breakers: Each downstream service, agent driver, or external API gets its own independent circuit breaker. A failure in the VDA 5050 adapter for a specific AGV fleet does not trip the breaker for a ROS 2-based AMR fleet.
• Thread Pool Isolation: Assign separate thread pools to each downstream dependency. A slow, saturated pool for one service cannot exhaust threads needed by another, a pattern known as bulkheading.
• Resource Exhaustion Prevention: The immediate rejection of requests when a circuit is Open prevents thread pool saturation and cascading timeouts that could bring down the entire control plane.

A circuit breaker must be an observable component within the agentic observability and telemetry framework:

• State Change Events: Every transition between Closed, Open, and Half-Open must emit a log event and a metric increment, enabling real-time alerting.
• Failure Reason Tracking: Log the specific exception type (e.g., timeout, connection refused, HTTP 503) that caused the breaker to trip, aiding root cause analysis.
• Metrics Export: Expose counters for successful calls, failed calls, rejected calls, and the current state as Prometheus metrics for dashboard visualization.
• Manual Override: Provide an administrative API to manually trip or reset a breaker for maintenance windows or emergency interventions.

A circuit breaker alone cannot guarantee safe retries. It must be paired with idempotency keys to prevent duplicate command execution:

• When a request times out and the breaker is Half-Open, the orchestrator cannot know if the downstream service processed the command before failing.
• By attaching a unique idempotency key to every command, the retried request is safely ignored by the receiver if already executed.
• This combination ensures exactly-once semantics even across circuit breaker state transitions, which is critical for commands like 'pick pallet' or 'dock to charger' that must not be duplicated.

The circuit breaker acts as a stateful proxy between the orchestrator and a downstream service (e.g., a robot's onboard controller or a zone management API). It monitors for failures and transitions through three distinct states:

• Closed: Requests flow normally. A failure counter tracks consecutive errors.
• Open: When the failure threshold is breached, the breaker trips and immediately fails all new requests without attempting the call, preserving resources.
• Half-Open: After a configurable timeout, a limited number of probe requests are allowed. If they succeed, the breaker resets to Closed; if they fail, it returns to Open.

In fleet orchestration, a circuit breaker must distinguish between transient and systemic failures across diverse agent types:

• Transient failures: A single AMR's Wi-Fi blip or a momentary VDA 5050 message timeout. The breaker should tolerate a configurable number of these before tripping.
• Systemic failures: A protocol adapter crash or a message bus partition. The breaker should trip immediately on specific exception types like ConnectionRefused or ServiceUnavailable.
• Slow calls: A robot's onboard service that responds with high latency can be treated as a failure via a slow-call threshold, preventing thread pool exhaustion in the orchestrator.

When a circuit breaker is Open, the orchestrator must not simply fail the entire workflow. Instead, it executes a predefined fallback to maintain operational safety:

• Stale Cache: Return the last known good state of an agent from the Agent Registry cache, marked with a staleness flag.
• Degraded Mode: Re-route a task to a less optimal but available agent type, accepting reduced throughput.
• Safe Stop: For a tripped breaker on a collision avoidance service, command all affected agents to execute an immediate controlled stop.
• Queued Retry: Persist the failed command to a dead-letter queue for asynchronous replay when the dependency recovers.

Modern fleet orchestration platforms built on the JVM often use Resilience4j, a lightweight fault-tolerance library. A circuit breaker for a robot's command endpoint is configured declaratively:

• failureRateThreshold: 50% — trip if half of calls in a rolling window fail.
• slidingWindowSize: 100 — evaluate the last 100 calls.
• waitDurationInOpenState: 30s — wait 30 seconds before transitioning to Half-Open.
• permittedNumberOfCallsInHalfOpenState: 3 — allow 3 probe requests.
• recordExceptions: Include java.net.ConnectException and org.springframework.web.client.HttpServerErrorException.

A circuit breaker is often paired with the Bulkhead pattern to prevent a single failing dependency from consuming all system resources. In fleet orchestration:

• Thread Pool Isolation: Assign a dedicated, bounded thread pool for communication with each agent type (e.g., one pool for all MiR robots, another for OTTO motors). If the MiR adapter hangs, its thread pool saturates and the circuit breaker trips, but the OTTO pool remains unaffected.
• Semaphore Isolation: Limit the number of concurrent calls to a shared resource like a map server, preventing a surge in path-planning requests from starving other services.

The state of every circuit breaker must be exposed to the fleet's observability stack for proactive operations:

• Metrics: Export circuitbreaker_state (0=Closed, 1=Half-Open, 2=Open), circuitbreaker_failure_rate, and circuitbreaker_slow_call_rate to Prometheus.
• Alerts: Trigger a PagerDuty alert if any breaker remains Open for more than 2 minutes, indicating a dependency that is not self-recovering.
• Dashboards: Visualize breaker state transitions in Grafana alongside agent health metrics to correlate fleet degradation with specific service failures.
• Distributed Tracing: Propagate the breaker decision into trace spans (e.g., via OpenTelemetry) so an Open circuit is visible in the end-to-end request flow.

A transient failure handling strategy that works in tandem with circuit breakers. When a call fails, the client waits before retrying, with the delay increasing exponentially (e.g., 1s, 2s, 4s, 8s) up to a maximum cap. This prevents a thundering herd problem where many clients simultaneously hammer a recovering service. Combined with jitter (randomized delay), it spreads out retry attempts to avoid synchronized retry spikes across the fleet.

An isolation strategy that partitions resources into pools to prevent a failure in one component from consuming all resources and cascading to the entire system. In fleet orchestration, this means dedicating separate thread pools or connection limits for different agent types or critical functions. If the AGV control loop exhausts its connections, the manual vehicle tracking system remains unaffected, preserving partial fleet visibility.

A dedicated API endpoint (e.g., /health or /ready) that the circuit breaker polls to determine if a downstream service is healthy. A liveness probe confirms the service is running; a readiness probe confirms it can accept traffic. The orchestrator uses these signals to automatically transition the circuit breaker from OPEN (failing) to HALF-OPEN (testing) state, allowing a limited number of trial requests to verify recovery before fully closing the circuit.

A critical parameter that defines the maximum duration a caller will wait for a response before aborting the request. In fleet middleware, timeouts must be tuned per agent type: a forklift executing a pallet pick may require a 30-second timeout, while a sensor data query should timeout in 500ms. Improperly configured timeouts can cause the circuit breaker to trip prematurely or fail to detect genuine hangs, making them a key tuning parameter.

The logic executed when a circuit breaker is OPEN and a request is rejected. Fallbacks provide graceful degradation rather than complete failure. Examples in fleet orchestration include:

• Returning a cached last-known position when the localization service is down
• Queuing a task for deferred execution when the task dispatcher is unavailable
• Assigning a task to a backup agent type when the primary agent class is unreachable

A complementary pattern that proactively prevents overload before the circuit breaker must react. A rate limiter caps the number of requests a client can send to a service within a time window using algorithms like token bucket or sliding window log. In a multi-node orchestrator, a distributed rate limiter backed by Redis or a similar shared store ensures that the aggregate request rate across all orchestrator instances respects the downstream service's capacity limits.