Circuit Breaker Pattern

The pattern operates through a finite state machine with three primary states that govern its behavior.

• CLOSED: The normal, operational state. Requests flow freely to the downstream service. Failures are counted, and if they exceed a configured threshold within a time window, the breaker trips to the OPEN state.
• OPEN: The protective state. All requests to the failing service fail immediately without attempting the operation. A timer is set; after this reset timeout elapses, the breaker transitions to the HALF-OPEN state.
• HALF-OPEN: The probing state. A limited number of test requests are allowed to pass through. Their success or failure determines the next state: success moves the breaker back to CLOSED (service is healthy), while failure returns it to OPEN (service is still unhealthy).

The breaker must be configured with precise rules to determine when a service is deemed unhealthy. This is not a simple on/off switch but a statistical guard.

• Failure Count/Percentage: The breaker tracks failures (e.g., timeouts, HTTP 5xx errors). It trips when a defined count (e.g., 5 failures) or a percentage (e.g., 50% of the last 100 calls) is exceeded within a sliding time window.
• Timeout Handling: Slow responses are often treated as failures. A call exceeding a defined slow call duration can be counted toward the failure threshold, protecting against degraded performance, not just total outages.
• Ignored Exceptions: Business logic exceptions (e.g., "item not found") can be configured to not count as failures, ensuring the breaker only reacts to systemic faults.

When the circuit is OPEN, the calling application must have a defined strategy to handle the failure gracefully, maintaining user experience and system stability.

• Default Response: Return a cached value, a static default, or a user-friendly error message.
• Degraded Functionality: Disable non-essential features that depend on the failing service.
• Alternative Service Call: Route the request to a backup or secondary service instance, if available.
• Fast Failure: The core benefit—failing immediately in the OPEN state is preferable to making the user wait for a timeout, freeing up threads and resources for other operations.

The Circuit Breaker Pattern is often used in conjunction with, but is distinct from, retry mechanisms. They serve complementary roles in a resilience strategy.

• Retry with Backoff: Handles transient faults (e.g., network blips) by re-attempting a failed operation with increasing delays. This occurs inside the CLOSED state of the circuit breaker.
• Circuit Breaker Role: Protects against persistent faults. When retries consistently fail and the threshold is breached, the breaker opens to stop the retry storm, allowing the downstream service time to recover. Retry logic should be disabled when the circuit is OPEN.

The state changes of a circuit breaker are critical operational events that must be exposed for system health monitoring and debugging.

• State Transition Metrics: Emit events or increment counters for each state change (CLOSED → OPEN, OPEN → HALF-OPEN, etc.).
• Failure Rate Reporting: Continuously report the current failure rate or count to a metrics dashboard.
• Logging: Log state transitions with contextual information (service name, threshold breached). This is essential for post-mortem analysis of incidents to understand why a breaker tripped.

The Circuit Breaker is one pillar of a comprehensive resilience architecture, often deployed alongside other patterns.

• Bulkhead Pattern: Isolates resources (thread pools, connections) for different services. If one service fails and consumes all threads, the bulkhead prevents it from starving other services. Used in conjunction with circuit breakers.
• Timeout: A prerequisite. Every remote call must have a timeout; without it, the circuit breaker cannot detect a failure.
• Fallback: As described, the strategy invoked when the circuit is open.
• Retry with Exponential Backoff: The mechanism for handling transient errors before the circuit breaker trips for persistent ones.

A resilience design pattern that isolates elements of an application into distinct, independent resource pools (bulkheads). If one component fails or is overwhelmed, its failure is contained within its own bulkhead, preventing it from consuming all system resources (like threads or connections) and causing a cascading failure across the entire application. This is analogous to the watertight compartments in a ship's hull.

• Key Mechanism: Resource isolation (e.g., separate thread pools, connection pools, or even processes for different downstream services or user groups).
• Primary Benefit: Fault containment and improved system stability under partial failure.
• Common Use: Often used alongside the Circuit Breaker; while the Circuit Breaker stops calling a failing service, the Bulkhead ensures the failure of service A doesn't starve service B of resources.

An algorithm used to space out repeated retry attempts to a failing operation by progressively increasing the wait time between retries. The delay typically follows an exponential sequence (e.g., 1s, 2s, 4s, 8s...) and is often combined with jitter (random variation) to prevent synchronized retry storms from multiple clients.

• Purpose: To reduce the load on a recovering service, giving it time to stabilize, and to avoid exacerbating a failure scenario.
• Relationship to Circuit Breaker: Exponential backoff is a retry strategy, while a Circuit Breaker is a failure detection and proxy layer. A robust system often employs both: the client uses backoff for retries, and the Circuit Breaker opens if failures persist beyond a threshold, temporarily halting all calls.

A fundamental resilience pattern where an application transparently retries a failed operation in the expectation that the failure is transient (e.g., network timeout, temporary service unavailability). Effective implementation requires:

• Idempotency: Ensuring the operation can be safely retried without adverse side effects.
• Termination Logic: Defining a maximum number of retry attempts or a fallback action.
• Backoff Strategy: Using techniques like exponential backoff to space retries.

Critical Distinction: The Retry Pattern is proactive (it tries to succeed), while the Circuit Breaker is reactive and protective (it stops trying to prevent harm). They are complementary: retries handle transient faults, and the circuit breaker activates when faults indicate a deeper, sustained problem.

A design pattern that provides an alternative execution path or default response when a primary operation fails. The goal is to maintain some level of functionality or provide a graceful, user-friendly degradation of service instead of a complete failure.

• Examples: Returning cached data, switching to a secondary/read-only data source, providing a default or stale value, or displaying a user-friendly message.
• Integration with Circuit Breaker: A Circuit Breaker's open state is the ideal trigger for a fallback. When the circuit is open, instead of throwing an exception immediately, the system can invoke a predefined fallback mechanism. This turns a hard failure into a controlled, degraded experience.

A persistent queue or storage mechanism used in message-based and event-driven systems to hold messages that cannot be processed successfully after multiple retry attempts. It serves as an audit trail for failures and allows for manual or automated forensic analysis and reprocessing.

• Resilience Role: Acts as a final safety net, preventing poison-pill messages from blocking entire processing pipelines.
• Connection to Resilience Patterns: While not a direct alternative, a DLQ operationalizes the failure management implied by patterns like Retry and Circuit Breaker. If a service protected by a circuit breaker is consistently failing, related messages might be diverted to a DLQ after retries are exhausted, ensuring system flow isn't halted.

A dedicated, lightweight endpoint (e.g., /health or /ready) exposed by a service to programmatically report its operational status. A robust health check typically verifies internal dependencies (like database connectivity, disk space, or downstream service status).

• Mechanism for Probes: Used by load balancers, orchestrators (Kubernetes), and circuit breaker libraries to determine service availability.
• Circuit Breaker Integration: A circuit breaker implementation can be configured to call a service's health check endpoint as a probe during its half-open state. If the health check succeeds, the circuit breaker can safely close, resuming normal traffic. This provides a more deterministic recovery signal than simply waiting for a timeout period to expire.