Circuit Breaker Pattern

The core of the pattern is a state machine with three distinct states that govern request flow:

• CLOSED: The normal state. Requests flow through to the operation. Failures increment a counter.
• OPEN: The circuit is tripped. Requests fail immediately without attempting the operation. A timer is set.
• HALF-OPEN: After the timer expires, a limited number of test requests are allowed. Success resets the circuit to CLOSED; failure returns it to OPEN. This stateful logic is what differentiates it from a simple retry mechanism.

The circuit breaker monitors for failures to decide when to trip. Key configurable thresholds include:

• Failure Count/Threshold: The number of consecutive failures (e.g., 5) or the failure rate (e.g., 50% over 100 requests) needed to trip the circuit.
• Timeout Duration: The maximum time to wait for a response before considering it a failure.
• Trip Duration: The length of time the circuit stays OPEN before transitioning to HALF-OPEN. These parameters allow fine-tuning for specific service latency and failure profiles.

When the circuit is OPEN, the pattern enables fail-fast behavior. Instead of waiting for a timeout or consuming threads on a doomed request, the caller receives an immediate error. This allows the system to:

• Conserve critical resources (threads, memory, connections).
• Reduce latency for users by avoiding long waits.
• Execute a predefined fallback strategy, such as returning cached data, a default value, or a user-friendly error message, enabling graceful degradation of service.

The pattern provides a structured path for the failing system to recover autonomously. The HALF-OPEN state acts as a probationary period, allowing the system to test if the underlying problem has been resolved without being overwhelmed by a flood of requests. This is a form of automatic backoff. Successful test requests prove stability and close the circuit, resuming normal operations. This prevents a recovering service from being immediately knocked offline again by a sudden surge of traffic.

The Circuit Breaker is most effective when combined with other resilience patterns:

• Retry Policies: Used inside a CLOSED circuit for transient faults (e.g., network blips). The circuit breaker trips if retries consistently fail.
• Exponential Backoff: Often used within the retry logic to space out retry attempts.
• Fallback Strategy: The immediate action taken when the circuit is OPEN or a call fails. This could be a static response, a call to a secondary service, or a cached result.
• Bulkhead Pattern: Isolates the circuit breaker for one service from others, preventing a single point of failure from consuming all application resources.

Effective circuit breakers are highly observable. They should expose metrics and events for operations teams to monitor system health:

• State Transitions: Logs or events for CLOSED → OPEN, OPEN → HALF-OPEN, etc.
• Request Counts: Success, failure, short-circuited (rejected while OPEN), and timeout counts.
• Latency Percentiles: Performance of calls through the circuit. This telemetry is crucial for tuning thresholds, diagnosing systemic issues, and understanding the impact of downstream service outages. It turns the pattern from a blind safety switch into a key component of system observability.

The primary use case is to shield a client application from repeatedly calling a failing or unresponsive downstream service (e.g., a database, payment API, or inventory service).

• Mechanism: After consecutive failures exceed a threshold, the circuit trips to OPEN state.
• Effect: Subsequent calls fail immediately without attempting the network request, conserving client resources (threads, memory) and preventing request pile-up.
• Example: An e-commerce checkout service calling a failing payment gateway. After 5 timeouts, the circuit opens, and the UI immediately shows a "Payment system temporarily unavailable" message instead of hanging for 30 seconds.

In microservices architectures, a single service failure can propagate upstream, overwhelming the entire system. The circuit breaker acts as a bulkhead to contain the blast radius.

• Scenario: Service A depends on Service B. If B fails and A continues to retry, A's resources (thread pools, connections) become exhausted. This can cause A to fail, which then impacts its own callers, creating a cascade.
• Containment: By opening the circuit, Service A stops calling B, preserving its own resources and stability. This allows the overall system to gracefully degrade (e.g., showing cached data) rather than collapsing entirely.

Third-party APIs (SaaS providers, geocoding services, weather data) are outside your control and can experience outages or rate limits. A circuit breaker is essential for managing these external dependencies.

• Handles Rate Limiting: If an external API returns 429 Too Many Requests, the circuit can interpret this as a failure and open, preventing further calls that would be rejected.
• Cost Control: For paid APIs, failing fast prevents wasted spend on calls that are certain to fail.
• Fallback Strategy: When the circuit is open, the system can use a fallback like a default value, stale cached data, or an alternative, less-critical service.

The pattern enables applications to provide a better, more responsive user experience during partial outages by failing fast and predictably.

• Immediate Feedback: Instead of a spinning loader that eventually times out, users get an instant "Feature unavailable" message.
• Prioritization: Non-critical features with broken dependencies can be disabled (circuit open) while core functionality remains operational.
• Half-Open State: The circuit's HALF-OPEN state allows for periodic, single-threaded probes to check if the service has recovered. This enables automatic restoration of functionality without manual intervention when the dependency is healthy again.

By blocking doomed requests, the circuit breaker conserves critical system resources that would otherwise be wasted.

• Thread Pool Protection: Prevents all application threads from being blocked waiting for a timeout from a single slow service, which would starve other healthy parts of the application.
• Connection Pool Management: Avoids exhausting database or HTTP client connection pools with hanging connections.
• Memory & CPU: Reduces overhead from serializing/deserializing requests and managing retry logic for operations that will not succeed. This stability is crucial for maintaining Service Level Objectives (SLOs) for healthy parts of the system.

In robotics and logistics, the circuit breaker pattern manages communication with individual Autonomous Mobile Robots (AMRs) or external subsystems (e.g., a vision system, elevator controller, warehouse management system).

• Agent Health Monitoring: If an AMR stops responding to status polls or task assignments, its circuit opens. The orchestrator stops sending it commands and may mark it as offline for maintenance.
• External System Integration: A circuit can guard calls to a legacy Warehouse Management System (WMS) API. If the WMS times out, the orchestrator can switch to a degraded mode using local task queues until connectivity is restored.
• Preventing Command Flooding: Stops the system from repeatedly sending navigation goals to a robot that is stuck or has a sensor fault, allowing a human-in-the-loop operator to be notified and intervene.

A defined strategy that specifies the conditions and mechanisms under which a failed operation should be automatically reattempted. It is a prerequisite for the Circuit Breaker Pattern, which monitors these retries to detect systemic failure.

• Key parameters include maximum retry count, delay between attempts, and which types of exceptions are retriable.
• A circuit breaker often wraps a retry policy, opening when retries consistently fail, thus preventing a wasteful retry storm on an unresponsive dependency.

A specific retry algorithm that progressively increases the waiting time between retry attempts, often using an exponential function (e.g., 1s, 2s, 4s, 8s). This is used in conjunction with a retry policy to reduce load on a failing system and increase the chance of recovery.

• It helps prevent overwhelming a struggling service.
• The circuit breaker's half-open state often uses a similar backoff strategy before attempting to probe the failed service again.

A resilience pattern that isolates elements of an application into independent pools (bulkheads) of resources (threads, connections, memory). If one component fails, its bulkhead contains the failure, preventing it from draining resources and causing a cascading failure in other parts of the system.

• Analogy: Like watertight compartments on a ship.
• Complements the circuit breaker by providing failure isolation; a circuit breaker on a dependency protects the caller, while a bulkhead protects the system's overall resource pool.

A predefined alternative course of action or default response that a system executes when a primary operation fails or a service becomes unavailable. It is the action taken when a circuit breaker is open.

• Examples: Returning cached data, using a default value, or calling a secondary, less-capable service.
• This enables graceful degradation, allowing the system to maintain limited functionality instead of complete failure.

A persistent queue used to store messages, events, or tasks that cannot be processed successfully after multiple retry attempts. It acts as a failure quarantine zone.

• Used for later analysis, manual intervention, or reprocessing after the root cause is fixed.
• In a workflow protected by a circuit breaker, failed requests that cannot be handled by a fallback may be placed in a DLQ for offline handling, ensuring no work is silently lost.

A dedicated API endpoint (e.g., /health or /ready) that exposes the operational status of a service. It allows external systems, like load balancers or orchestration platforms, to determine its readiness and liveness.

• A circuit breaker's monitoring logic can be informed by health check results.
• In a half-open state, the circuit breaker may call a lightweight health endpoint before allowing full traffic to resume, acting as a canary request.