Circuit Breaker Pattern

The pattern's core logic is defined by a state machine with three states:

• CLOSED: Normal operation. Requests flow to the service. Failures increment a counter.
• OPEN: The circuit has 'tripped.' All requests fail immediately without calling the service, returning a pre-defined fallback or error.
• HALF-OPEN: After a timeout, a limited number of test requests are allowed. Success resets the circuit to CLOSED; failure returns it to OPEN.

The circuit monitors for failures to decide when to trip. Key configuration parameters include:

• Failure Threshold: The count or percentage of recent calls that must fail (e.g., 5 failures in 10 seconds).
• Failure Definition: What constitutes a failure—a timeout, a specific HTTP status code (e.g., 5xx), or a thrown exception.
• Sliding Window: Failures are typically tracked within a recent time window, ensuring the circuit responds to current conditions.

This state enables automatic system healing. After a configurable reset timeout in the OPEN state, the circuit transitions to HALF-OPEN. A single probe request or a small batch is sent. If successful, the circuit assumes the service is healthy and resets to CLOSED. If the probe fails, it immediately returns to OPEN for another reset period. This prevents overwhelming a recovering service.

When the circuit is OPEN or a call fails, the system does not simply crash. It executes a fallback strategy. This can include:

• Returning a cached, stale response.
• Providing a default or empty value.
• Queuing the request for later retry.
• Failing fast with a meaningful error to the caller. This allows the overall application to degrade functionality gracefully instead of failing completely.

Effective circuit breakers are deeply instrumented. They emit metrics and events for:

• State Changes (CLOSED → OPEN).
• Request counts, failure rates, and latency percentiles.
• Rejection counts when the circuit is OPEN. These signals are vital for dashboards and alerts, allowing SREs and developers to visualize system health and identify chronically failing dependencies.

While retry logic with exponential backoff is complementary, the Circuit Breaker Pattern serves a different, higher-order purpose:

• Retries operate on a per-request basis, hoping a transient fault clears.
• Circuit Breakers operate on a service dependency level. They stop all requests after detecting a pattern of failure, protecting the broader system. Using both together is a best practice: retry transient faults, but trip the circuit if retries consistently fail.

A fleet orchestration platform relies on external services like map providers, weather APIs, or warehouse management systems. A circuit breaker monitors the latency and error rate of calls to these services.

• Closed State: Normal operation; requests flow.
• Open State: After error threshold breaches, the circuit 'opens.' All requests fail fast without attempting the call, returning a cached response or default value.
• Half-Open State: After a timeout, a single test request is allowed. Success resets the circuit to 'Closed'; failure returns it to 'Open.'

This prevents the orchestration engine from being blocked by a slow or failing external dependency, maintaining core scheduling functions.

In a heterogeneous fleet, Autonomous Mobile Robots (AMRs) and manual forklifts communicate status via a central broker. If an agent's software crashes or network connectivity degrades, its message queue can back up.

A circuit breaker on the broker's connection to that specific agent can detect:

• Timeouts on heartbeat messages.
• Failed state update acknowledgments.

When triggered, the circuit breaker isolates the faulty agent. The orchestration system can then:

• Mark the agent's last known state as 'unreliable.'
• Re-assign its pending tasks to healthy agents.
• Initiate a diagnostic or recovery procedure. This prevents the system from wasting resources on an unresponsive agent.

Fleet agents stream telemetry (lidar, camera, battery, position). A downstream real-time analytics service processes this data for anomaly detection. If this service becomes overloaded or fails:

• A circuit breaker in the data ingestion layer trips.
• Telemetry is temporarily routed to a durable, high-throughput buffer (like Apache Kafka).
• The core control loop continues using the last known good state.

This prevents backpressure from a failing analytics module from choking the primary command-and-control data flow, ensuring the fleet remains operational even when secondary systems degrade.

A critical subsystem failure, like a zone controller managing a high-traffic intersection, could cause a ripple effect. Agents waiting for clearance from the failed controller would block paths for others.

A circuit breaker pattern applied at the system integration level:

• Monitors the health of key subsystems (path planners, traffic managers).
• Upon detecting failure, it triggers a graceful degradation protocol.
• Example: Falls back to a simpler, decentralized collision avoidance protocol while the central planner is offline.

This containment prevents a single point of failure from halting the entire fleet, enabling partial operability while the root cause is addressed.

The circuit breaker pattern works in concert with other resilience patterns:

• Health Checks: Provide the liveness/readiness signals that inform the circuit breaker's logic.
• Load Balancers: Use circuit breaker states to drain traffic from an 'Open' or 'Half-Open' instance.

In a microservices-based orchestration architecture, if the Task Allocation Service instance starts throwing errors, its circuit breaker opens. The service registry or API gateway, seeing this, stops routing new allocation requests to that instance. This allows the failing service to recover without being bombarded by retries from the entire fleet.

Effective circuit breakers provide rich telemetry crucial for AIOps and site reliability engineering.

• State Transition Logging: Every flip from Closed→Open→Half-Open is logged with timestamps and error context.
• Metrics Exposure: Counters for failed calls, request volumes, and circuit state duration are exposed (e.g., to Prometheus).
• Dashboard Integration: Real-time visualization of all circuit states across the system provides an at-a-glance fault topology.

This data feeds into higher-order AI systems for predictive maintenance, identifying services or agent types that are chronically failing and may require software updates or hardware replacement.

A complementary fault-handling strategy where a client automatically re-attempts a failed operation, progressively increasing the wait time between attempts. This reduces load on a struggling service and increases the chance of recovery.

• Core Mechanism: Wait time increases exponentially (e.g., 1s, 2s, 4s, 8s) or with jitter to prevent synchronized retry storms.
• Primary Use: Handles transient failures like network timeouts or temporary service unavailability.
• Key Difference from Circuit Breaker: Retry logic is client-side and persistent; a circuit breaker gives up after a threshold to protect the system. They are often used together: retry logic handles brief blips, while the circuit breaker trips after repeated failures.

A holding queue for messages that cannot be delivered or processed successfully after multiple retry attempts. It acts as a safety net for unprocessable events, preventing data loss and enabling forensic analysis.

• Function: Captures failed messages (e.g., due to malformed payloads, downstream outages, or business logic errors).
• Operational Role: Allows operators to inspect, repair, and optionally re-queue messages. Essential for debugging in event-driven systems.
• Relationship to Circuit Breaker: When a circuit breaker is open, messages/requests may be immediately failed and routed to a DLQ instead of being sent to the unresponsive service, ensuring they are not simply discarded.

A periodic probe or test sent to a service instance to determine its operational status and readiness to handle requests. Health checks are a primary signal for load balancers and orchestration systems.

• Types: Liveness probes check if the process is running. Readiness probes check if the service can accept traffic (e.g., database connections are healthy).
• Implementation: Typically an HTTP endpoint (e.g., /health) or a lightweight script that returns a status code.
• Integration with Circuit Breaker: Health check failures can be a direct input to a circuit breaker's failure count. A pattern may monitor health check endpoints instead of, or in addition to, actual business request failures.

A resilience pattern that isolates elements of an application into pools (bulkheads) so that a failure in one pool does not drain resources and cause a cascading failure across the entire system.

• Analogy: Like the watertight compartments on a ship.
• Implementation: Uses separate thread pools, connection pools, or even processes for different service calls or customer segments.
• Synergy with Circuit Breaker: While a circuit breaker stops calls to a failing service, a bulkhead prevents that failure from consuming all threads/connections and starving other healthy parts of the application. They are foundational patterns for fault isolation.

A predefined alternative response or action taken when a primary operation fails or a circuit breaker is open. Its purpose is to provide a degraded but acceptable user experience during partial outages.

• Examples: Returning cached data, static default values, a simplified calculation, or a friendly error message.
• Design Consideration: The fallback logic must not itself call another failing or external service, or it risks compounding the failure.
• Circuit Breaker Integration: A robust circuit breaker implementation invokes a fallback routine immediately when in the OPEN state, providing a fast failover path without waiting for timeouts.

A pattern for controlling the rate of requests sent or received by a network interface controller, application, or user. It protects services from being overwhelmed by too many requests in a given time window.

• Purpose: Prevents resource exhaustion, mitigates denial-of-service attacks, and enforces usage quotas.
• Common Algorithms: Token bucket, leaky bucket, and fixed window counters.
• Contrast with Circuit Breaker: Rate limiting is proactive and focuses on preventing overload. A circuit breaker is reactive and focuses on stopping calls to a failing service. A service might rate limit clients, and a client might use a circuit breaker when it detects failures from that service.