Circuit Breaker Pattern

The core of the pattern is a state machine with three distinct states that govern the flow of requests to a protected service:

• CLOSED: The normal operating state. Requests flow freely to the service. Failures are counted. If failures exceed a configured threshold, the breaker trips to OPEN.
• OPEN: The failure state. All requests to the service fail immediately without attempting the operation. A timer is set. After this timeout, the breaker moves to HALF-OPEN.
• HALF-OPEN: The trial state. A limited number of test requests are allowed to pass. Success resets the breaker to CLOSED. Failure returns it to OPEN.

The breaker must detect failures to trip effectively. This is governed by configurable thresholds and time windows:

• Failure Count/Threshold: The number of failures (e.g., timeouts, 5xx errors) required to trip the breaker (e.g., 5 failures).
• Sliding Time Window: Failures are counted within a specific duration (e.g., the last 60 seconds). This prevents a single historical spike from permanently tripping the breaker.
• Failure Criteria: Defines what constitutes a failure (e.g., HTTP 500, timeout > 2s, specific exception types).

When the breaker is OPEN, calls must fail fast. The pattern enables graceful degradation through defined fallback strategies:

• Default Response: Return a cached, stale, or sensible default value.
• Alternative Service: Route the request to a secondary, possibly less capable, service.
• Informative Error: Return a clear, user-friendly error message (e.g., "Service temporarily unavailable"). This prevents the calling system from hanging on timeouts and allows the user experience to degrade gracefully rather than fail completely.

The HALF-OPEN state is the pattern's self-healing mechanism. After a configured reset timeout in the OPEN state, the breaker allows a probe request through.

• Probe Request: A single request or a small batch is permitted to test if the underlying service has recovered.
• Success Criteria: If the probe succeeds, the breaker assumes health is restored and transitions to CLOSED, resuming normal operations.
• Failure Response: If the probe fails, the breaker immediately returns to the OPEN state and the reset timer restarts, preventing a flood of requests to a still-unhealthy service.

Effective circuit breakers are highly observable. They emit metrics and events critical for system health monitoring:

• State Transition Events: Log when the breaker trips from CLOSED to OPEN, or resets.
• Request Metrics: Track counts of successful, failed, short-circuited (rejected while OPEN), and timeout requests.
• Latency Histograms: Monitor the latency of calls when the breaker is CLOSED. These metrics are essential for Site Reliability Engineering (SRE), enabling alerting on abnormal trip rates and providing data for capacity planning and debugging.

In fleet orchestration, the Circuit Breaker Pattern works in tandem with other resilience patterns:

• Load Balancer Health Checks: A tripped circuit breaker can trigger a load balancer to mark an instance as unhealthy, stopping all traffic routing.
• Retry Patterns: Circuit breakers should be placed outside of retry logic. Retrying against an OPEN breaker is pointless. Use an exponential backoff retry behind a closed breaker.
• Orchestrator Actions: In platforms like Kubernetes, a pod with a consistently tripped breaker may signal the need for a pod restart or rescheduling, integrating with liveness probes.

In a microservices architecture, a circuit breaker is placed in front of calls to external dependencies (e.g., a payment service, inventory API). If the service begins to time out or return errors, the breaker trips after a failure threshold. Subsequent calls immediately fail fast, preventing thread pool exhaustion in the calling service. This allows the failing service time to recover while the caller can provide a graceful fallback (e.g., cached data, default response).

• Key States: Closed (normal operation), Open (fast-fail), Half-Open (probing for recovery).
• Example: An e-commerce checkout service uses a circuit breaker on its payment gateway call. During a gateway outage, the breaker opens, and the UI displays a "Try again later" message instead of hanging indefinitely.

In heterogeneous fleet orchestration, a circuit breaker can manage interactions with individual Autonomous Mobile Robots (AMRs) or their subsystems. For instance, if a robot's navigation stack repeatedly fails to provide localization data, a circuit breaker on the state estimation channel can trip. This triggers the orchestration middleware to reassign the robot's tasks, mark it as unhealthy, and route it to a maintenance zone.

• Prevents Cascade: Stops a single faulty agent from blocking a multi-agent path planning queue.
• Enables Graceful Degradation: The fleet continues operating with reduced capacity while the issue is diagnosed.

Circuit breakers guard against failures when integrating with third-party APIs that have rate limits or unreliable performance. Instead of repeatedly hitting a rate-limited API and receiving 429 errors, a circuit breaker can trip based on these specific failure modes. It protects the client from being blacklisted and allows for implementing a fallback strategy, such as using a different provider or a cached result.

• Combined with Retries: Often used with a retry policy with exponential backoff. The circuit breaker trips if retries consistently fail.
• Example: A weather service integration uses a circuit breaker that opens after five consecutive 429 (Too Many Requests) responses, switching to a less accurate but free internal model for 5 minutes.

A primary use case is preventing application failure due to database unavailability. If a database cluster becomes slow or unresponsive, connections from the pool may timeout and pile up, exhausting the pool. A circuit breaker on the database access layer trips after a configurable number of connection timeouts. This causes non-critical application features that require the database to fail immediately, preserving connection pool resources for critical functions or for when the database recovers.

• Preserves Resources: Prevents thread/connection pool exhaustion, a common cause of cascading failure.
• Half-Open State: After a timeout, a single test query is allowed to check if the database is responsive before closing the circuit.

In embodied intelligence systems and IoT, circuit breakers can isolate faulty hardware components. For example, if a lidar sensor on an AMR starts returning corrupted or out-of-range data, a software circuit breaker on the sensor driver can trip. This signals the collision avoidance system to switch to a secondary sensor (e.g., cameras) or a degraded safety mode, preventing erroneous navigation commands.

• Physical Analogy: Directly mimics an electrical circuit breaker preventing damage from a short circuit.
• Key for Resilience: Essential for building robust edge AI architectures that must operate despite hardware degradation.

Circuit breakers are a core concept behind health checks in load balancers. If a backend server (e.g., in a Kubernetes pod) fails its health checks repeatedly, the load balancer's internal "circuit" to that instance opens, draining connections and removing it from the healthy pool. This prevents user traffic from being sent to a failing instance. The instance is periodically probed (half-open state) and returned to the pool only after passing consecutive health checks.

• Foundation for Reliability: Enables patterns like blue-green deployments and auto-scaling by providing a clear signal of instance health.
• Example: An Application Load Balancer (ALB) uses HTTP health checks. After 3 failures, it stops routing new requests to the target, implementing the Open state.

The Fallback Pattern provides an alternative execution path or default response when a primary operation fails. It is the primary mechanism for providing graceful degradation when a circuit breaker is open.

• Key Mechanism: When a call fails or a circuit is open, the system executes a predefined fallback routine instead of throwing an error to the user.
• Common Implementations:
  • Returning cached or stale data.
  • Calling a secondary, less-capable service.
  • Providing a default or placeholder response.
  • Queueing the request for later processing.
• Example: An e-commerce product page might display a cached price and "Add to Cart" button (queuing the action) if the real-time inventory service is unavailable, rather than showing an error page.

A Health Check is a periodic probe sent to a service instance to verify its operational status and readiness to receive traffic. It is a critical input for load balancers and can inform circuit breaker state transitions.

• Key Mechanism: An endpoint (e.g., /health) returns a simple status code (HTTP 200) if the service is healthy. Checks can be liveness (is the process running?) or readiness (can it handle requests?).
• Integration with Load Balancing: Load balancers use health checks to automatically drain unhealthy instances from the traffic pool.
• Relation to Circuit Breaker: While a circuit breaker uses actual request failures as its primary signal, a health check can provide a proactive, synthetic test. Some circuit breaker implementations can use health check results to transition from Half-Open to Closed state.