Circuit Breaker Pattern

The pattern's core logic is defined by a state machine with three primary states:

• CLOSED: The normal operational state. Requests flow through to the protected service. Failures are counted.
• OPEN: The circuit is tripped. All requests to the service fail immediately without attempting the call, returning a pre-defined error or fallback. A timeout is set before moving to the HALF-OPEN state.
• HALF-OPEN: After the timeout, a limited number of trial requests are allowed. Their success or failure determines the next state—closing the circuit on success or reopening it on failure.

The transition from CLOSED to OPEN is governed by configurable thresholds that detect systemic failure, not transient blips. Common implementations track:

• A sliding window of recent calls (e.g., last 100 requests).
• A failure ratio (e.g., 50% failures within the window).
• A count-based threshold (e.g., 5 consecutive failures). Once the threshold is breached, the circuit trips open. This prevents the caller from waiting on timeouts for every request, freeing resources immediately.

When the circuit is OPEN, it is not permanent. A reset timeout (e.g., 30 seconds) is configured. After this period elapses, the circuit transitions to HALF-OPEN, permitting a probe request. This allows for automatic recovery without manual intervention if the underlying service has healed. If the probe succeeds, the circuit resets to CLOSED; if it fails, it returns to OPEN for another timeout period.

When the circuit is OPEN or a call fails, the pattern does not just throw an error. It should trigger a fallback strategy to maintain partial functionality. This is key to graceful degradation. Examples include:

• Returning cached stale data.
• Providing a default or empty response.
• Queuing the request for asynchronous retry later.
• Delegating to a secondary, less-capable service. This ensures the user experience degrades usefully instead of breaking completely.

Effective circuit breakers expose detailed metrics and events for operational observability. Essential data points include:

• Current state (CLOSED, OPEN, HALF-OPEN).
• Failure counts and ratios.
• Request volume through the circuit.
• State transition timestamps. This telemetry is vital for Service Level Objective (SLO) tracking, understanding system health, and debugging. It answers whether the breaker is protecting the system or itself causing issues.

The Circuit Breaker Pattern is complementary to, but distinct from, retry logic. They are often used in tandem:

• Retry Logic handles transient errors (e.g., network timeouts) by immediately re-attempting the same operation.
• Circuit Breaker handles persistent failures by stopping all attempts for a period. A best-practice architecture applies fast retries with exponential backoff and jitter at the call site, protected by a circuit breaker at the service boundary. This prevents retry storms from overwhelming a sick dependency.

A retry algorithm where the delay between consecutive retry attempts increases exponentially (e.g., 1s, 2s, 4s, 8s). It is a complementary strategy to the circuit breaker, used before the breaker trips to handle transient faults by reducing retry pressure on a struggling service.

• Purpose: To prevent retry storms and increase the likelihood of a recovering service handling a request.
• Implementation: Often combined with jitter (randomized delays) to prevent client synchronization.

A resilience design pattern that isolates resources (like thread pools, connections, or memory) for different service calls or consumer groups. Its goal is to prevent a failure in one part of the system from cascading and exhausting all resources.

• Analogy: Like watertight compartments on a ship.
• Relationship to Circuit Breaker: While a circuit breaker stops calls to a failing dependency, a bulkhead isolates failures within the caller to protect overall system stability. They are often used together.

A predefined alternative action taken when a primary operation fails. When a circuit breaker is open, a robust system executes a fallback instead of simply failing fast.

• Examples: Returning cached or stale data, providing a default value, switching to a degraded but functional code path, or queuing the request for later processing.
• Purpose: Enables graceful degradation, maintaining a useful—if limited—user experience during partial outages.

Control mechanisms that restrict the request rate a client or service can send or receive. While a circuit breaker reacts to failures, rate limiting proactively prevents overload.

• Rate Limiting: Enforces a strict cap on requests per time window (e.g., 1000 requests/hour). Often uses algorithms like the Token Bucket or Leaky Bucket.
• Throttling: Dynamically slows down request processing under high load. A service may throttle clients before it becomes unhealthy and triggers their circuit breakers.

A periodic diagnostic probe (e.g., an HTTP GET to a /health endpoint) used to assess a service's operational status. Health checks are a primary signal for circuit breaker state transitions.

• Liveness Probe: Determines if the service is running.
• Readiness Probe: Determines if the service is ready to accept traffic (e.g., dependencies connected).
• Use Case: A circuit breaker in a half-open state may use a health check as a trial request to determine if it should close.

A durable storage queue for messages or requests that have repeatedly failed all processing attempts, including retries. It acts as a final fault isolator.

• Relationship to Circuit Breaker: After a circuit breaker opens and retries are exhausted, a system may place the failed request into a DLQ for offline analysis and manual or automated reprocessing.
• Purpose: Prevents blocking of main workflows, ensures no data is silently lost, and provides an audit trail for persistent failures.