Circuit Breaker

A circuit breaker operates as a deterministic finite state machine with three core states:

• CLOSED: Normal operation. Requests pass through to the downstream service. A failure counter tracks consecutive or time-windowed errors.
• OPEN: The breaker trips when the failure threshold is exceeded. All subsequent requests are immediately rejected without attempting the downstream call, preserving resources.
• HALF-OPEN: After a configurable reset timeout, a limited number of probe requests are allowed through. If they succeed, the breaker resets to CLOSED; if they fail, it returns to OPEN.

This state machine logic prevents thundering herd retries from overwhelming a recovering service.

The precision of a circuit breaker depends on correctly tuning its trip thresholds. Key parameters include:

• Error Rate Threshold: The percentage of failed requests (e.g., >50%) over a sliding window that triggers the OPEN state.
• Sliding Window Type: Time-based windows (e.g., last 30 seconds) or count-based windows (e.g., last 100 requests) determine the recency of the failure signal.
• Minimum Request Volume: A floor value that prevents the breaker from tripping on statistically insignificant sample sizes during low-traffic periods.
• Consecutive Failure Count: A simpler alternative to rate-based thresholds, useful for services where any failure is a strong signal.

Misconfiguration leads to false positives (unnecessary tripping) or false negatives (failing to protect the system).

When a circuit breaker is OPEN, the calling service must implement a fallback strategy to maintain partial functionality:

• Cached Responses: Return stale but acceptable data from a local cache or edge CDN.
• Default Values: Provide a safe, static response that allows the user experience to continue without the failed dependency.
• Alternative Service: Route requests to a redundant endpoint or a degraded read-replica.
• Queued Retry: Persist the failed request to a dead letter queue for asynchronous replay once the downstream service recovers.

Without a fallback, the circuit breaker merely replaces a timeout error with an immediate exception, providing no user-facing benefit.

Circuit breakers are most effective when combined with the bulkhead pattern to limit resource contention. Key integration points:

• Thread Pool Isolation: Assign a dedicated thread pool to each downstream dependency. When a circuit breaker trips, only that pool's threads are released, preventing a slow service from exhausting the entire application server's worker threads.
• Connection Pool Limits: Restrict the maximum number of concurrent connections to a downstream service. A tripped breaker immediately releases these connections back to the pool.
• Timeout Coupling: The circuit breaker's failure detection must integrate with request timeouts. A request that hangs for 30 seconds before timing out is a failure that should count toward the threshold.

This combination ensures that a single misbehaving microservice cannot trigger a cascading failure across the entire distributed system.

A circuit breaker must expose granular telemetry to enable operators to diagnose system health:

• State Change Events: Every transition between CLOSED, OPEN, and HALF-OPEN must be logged and emitted as a metric for alerting.
• Failure Counters: Track the number of requests blocked while OPEN versus the number of successful requests in CLOSED state.
• Trip Cause Attribution: Log the specific exception type or status code that caused the threshold to be exceeded (e.g., HTTP 503, ConnectionRefused, SocketTimeout).
• Reset Timer Visibility: Expose the remaining time until the breaker transitions to HALF-OPEN, allowing operations teams to estimate recovery windows.

Integrating these metrics into dashboards and alerting systems like Prometheus and Grafana is essential for production debugging.

Several mature libraries provide production-grade circuit breaker implementations:

• Resilience4j: A lightweight, functional-programming library for Java designed for Java 8 and functional interfaces. It provides a decorator-based API to wrap synchronous, asynchronous, and reactive calls.
• Polly: A comprehensive resilience and transient-fault-handling library for .NET. It allows combining circuit breaker policies with retry, timeout, and bulkhead policies into a resilience pipeline.
• Hystrix (Maintenance Mode): The original Netflix library that popularized the pattern. While now in maintenance, its architectural concepts of command wrapping and thread pool isolation remain influential.
• Istio/Envoy: Service mesh implementations that apply circuit breaking at the sidecar proxy level, requiring no application code changes and enforcing policy uniformly across polyglot environments.

A resilience pattern that partitions system resources into isolated pools, preventing a failure in one component from cascading and consuming all available resources. In practice, this means assigning dedicated thread pools or connection limits to different services.

• Thread pool isolation: Assigning separate thread pools per downstream dependency
• Connection pool partitioning: Limiting connections per service to prevent resource exhaustion
• Ship analogy: Like watertight compartments in a vessel, a breach in one bulkhead doesn't sink the entire ship

A complementary pattern to circuit breaking that governs how and when to retry failed operations. Rather than immediately retrying, the system waits progressively longer between attempts, often adding jitter to avoid thundering herd problems.

• Base delay: Starting wait time (e.g., 100ms)
• Multiplier: Each subsequent wait is multiplied (e.g., 2x)
• Max retries: Hard limit before surfacing the failure
• Jitter: Random variation to desynchronize retry storms

A queue that stores messages that cannot be processed successfully after all retries and circuit breaker attempts are exhausted. DLQs enable asynchronous failure recovery by preserving failed events for later inspection, replay, or manual intervention.

• Poison message handling: Automatically routing malformed messages away from the main processing pipeline
• Replay capability: Operators can reprocess messages after fixing the root cause
• Observability: DLQ depth serves as a critical health metric for system operators

Circuit breakers rely on properly configured timeouts to detect failures. Without timeouts, a slow downstream service can hold connections indefinitely. Deadline propagation carries a request's total allowed time budget across service boundaries.

• Connection timeout: Maximum time to establish a TCP connection
• Request timeout: Maximum time to wait for a complete response
• Context deadline: gRPC and similar frameworks propagate deadlines through the call chain, ensuring no service wastes resources on an already-expired request

Proactive mechanisms that detect unhealthy instances before they cause failures. Health checks continuously probe service endpoints, while load shedding deliberately rejects excess requests when a service approaches overload, preserving capacity for critical operations.

• Liveness probes: Is the service running?
• Readiness probes: Is the service able to accept traffic?
• Adaptive load shedding: Rejecting low-priority requests based on queue depth or latency thresholds, preventing overload from triggering circuit breakers unnecessarily

The discipline of experimenting on distributed systems to build confidence in their resilience. Practitioners deliberately inject failures—including triggering circuit breakers—to verify that the system degrades gracefully rather than failing catastrophically.

• Game days: Simulated production incidents to test operator response
• Fault injection: Deliberately introducing latency, errors, or resource exhaustion
• Steady-state hypothesis: Defining normal behavior and verifying it holds during chaos experiments
• Netflix Chaos Monkey: Pioneering tool that randomly terminates production instances