Circuit Breaker Pattern

The core logic of a circuit breaker is governed by a finite state machine with three distinct states:

• Closed: Requests flow normally to the downstream service. Failures are counted.
• Open: The circuit is 'tripped.' All requests fail immediately without attempting the call, returning a pre-defined fallback or error.
• Half-Open: After a timeout, a limited number of test requests are allowed. Success moves the circuit back to Closed; failure returns it to Open.

The breaker monitors calls for specific failure conditions to decide when to trip. Key configurable thresholds include:

• Failure Count/Percentage: The number or rate of failures (e.g., timeouts, 5xx errors) within a sliding time window.
• Timeout Duration: The maximum time to wait for a call before considering it a failure.
• Slow Call Threshold: A call exceeding a defined duration can be counted as a failure, even if it eventually succeeds, protecting against degraded performance.

When the circuit is Open or a call fails, the system must provide a graceful response instead of propagating the error. Common fallback strategies are:

• Static Default: Return a cached value or a safe, default response.
• Stubbed Response: Provide a minimal, non-operational response that allows other system parts to proceed.
• Alternative Service Call: Route the request to a backup or degraded-functionality service.
• Exception Propagation: Throw a specific, well-known exception (e.g., CircuitBreakerOpenException) for upstream logic to handle.

This state enables the system to probe for recovery without being overwhelmed. Critical behaviors include:

• Reset Timeout: The duration the circuit stays Open before transitioning to Half-Open.
• Probe Limits: In Half-Open, only a small, finite number of test requests are permitted.
• Success Threshold: The number of consecutive successful probe requests required to close the circuit. A single failure during probing typically re-opens it immediately.

Circuit breakers and retries are complementary but distinct patterns. They must be coordinated to avoid conflict:

• Circuit Breaker First: The breaker should wrap the external call. If the circuit is Open, no retry is attempted.
• Retry Inside Breaker: For a Closed circuit, a limited, fast-failing retry (e.g., with exponential backoff) can be attempted within the protected call. A series of rapid retries should quickly trip the breaker.
• Avoid Retry Storms: Never place an unbounded retry loop outside a circuit breaker, as this defeats its purpose.

Effective circuit breakers are highly observable, providing essential telemetry for system health:

• State Transitions: Log events (INFO/ERROR) for every state change (Closed → Open, Open → Half-Open, etc.).
• Metrics: Expose counters for call attempts, successes, failures, timeouts, and the current circuit state (e.g., via gauges).
• Dependency Tracking: In distributed traces, annotate spans to indicate when a call was short-circuited.

The canonical implementation uses a finite state machine with three distinct states:

• CLOSED: Requests flow normally. Failures increment a counter.
• OPEN: The circuit trips after a failure threshold is met. All requests fail fast with an exception, bypassing the call.
• HALF-OPEN: After a timeout, a single test request is allowed. Success resets the circuit to CLOSED; failure returns it to OPEN. This stateful design is the core of libraries like Netflix Hystrix and Resilience4j.

In a microservices architecture, circuit breakers are deployed at the API gateway or service mesh layer (e.g., Istio, Linkerd). They protect the entire call graph. For example, if a payment service times out, the gateway can:

• Immediately return a 503 "Service Unavailable" to the client.
• Route subsequent requests to a fallback service or cached response.
• Prevent thread pool exhaustion in the calling service by failing fast.

Applied to database drivers and connection pools to handle backend degradation. If a database cluster becomes unresponsive, the circuit breaker on the application server will:

• Trip after a configured number of connection timeouts.
• Throw an immediate exception to the application, which can use cached data or a read-only replica.
• Periodically attempt a health check query in the HALF-OPEN state to see if the primary database has recovered.

Used when calling third-party APIs (e.g., payment gateways, geocoding services, SMS providers). Configuration is critical:

• Timeout: Set aggressively (e.g., 2 seconds) to prevent blocking.
• Failure Threshold: Low (e.g., 5 failures) to trip quickly.
• Fallback: Return a default value, use a secondary provider, or queue the request for later retry. This prevents a slow external provider from making your application unusable.

Circuit breakers are most powerful as part of a resilience policy chain. A common pattern is Retry → Circuit Breaker → Fallback:

1. Retry: Attempt the call with exponential backoff (e.g., 100ms, 200ms, 400ms).
2. Circuit Breaker: If all retries fail, the circuit trips to OPEN.
3. Fallback: While the circuit is OPEN or after a final failure, execute a predefined fallback method. This is implemented in libraries like Polly for .NET and go-resilience for Go.

While a circuit breaker protects against remote service failure, a Bulkhead pattern protects against resource exhaustion. Key differences:

• Circuit Breaker: Operates on a logical operation (calls to Service X).
• Bulkhead: Isolates physical resources (thread pools, connections, memory) per service/consumer. Used together, they provide comprehensive fault tolerance: bulkheads prevent one failed service from consuming all threads, and the circuit breaker stops calls to that service entirely.

A fault-tolerant strategy where a system provides a predefined alternative response or workflow when a primary operation fails. This allows the system to maintain partial or degraded functionality.

• Examples: Returning cached data, using a default value, switching to a less-capable but more reliable algorithm, or displaying a user-friendly message.
• Relationship to Circuit Breaker: The circuit breaker's half-open state is a form of fallback for testing recovery. A common pattern is: when the circuit is open, all requests immediately fail fast and execute a fallback (e.g., 'Service X is temporarily unavailable'), avoiding the latency of a doomed call.

A system design principle where functionality is progressively and intentionally reduced in a controlled manner under failure, high load, or partial outage conditions. The goal is to maintain availability of core services.

• Contrast with Fallback: Graceful degradation is a system-wide strategy, often involving load shedding, disabling non-essential features, or simplifying workflows. A fallback is a component-level alternative.
• Circuit Breaker's Role: Circuit breakers are a key enabler of graceful degradation. By failing fast on non-critical or failing dependencies, the system conserves resources (like threads and connections) to keep core user journeys operational.