Circuit Breaker Pattern

A circuit breaker operates through a finite state machine with three primary states:

• Closed: The normal operational state. Requests pass through to the dependent service. Failures are counted, and if they exceed a defined threshold within a time window, the breaker trips to the Open state.
• Open: The fail-fast state. All requests to the dependent service immediately fail without attempting the operation. A timer is set for a retry timeout period.
• Half-Open: A probationary state entered after the retry timeout expires. A limited number of test requests are allowed to pass. Their success or failure determines the next state: success resets the breaker to Closed; failure returns it to Open.

The core logic for tripping the breaker is based on configurable thresholds that detect abnormal failure rates.

• Failure Count/Threshold: The number of failures (e.g., timeouts, 5xx HTTP errors) required to trip the breaker (e.g., 5 failures).
• Sliding Time Window: Failures are counted within a recent time window (e.g., the last 60 seconds), ensuring the breaker responds to current conditions, not historical ones.
• Failure Ratio: An alternative to a simple count, this trips the breaker when a percentage of recent calls fail (e.g., 50% failure rate over the last 100 requests).

This mechanism distinguishes transient network blips from a genuine service outage.

When the circuit is Open, calls do not reach the failing service. Instead, the pattern mandates a fallback strategy to maintain partial functionality.

• Default/Cached Response: Return a static default value or a stale, cached version of the data.
• Alternative Service: Route the request to a secondary, possibly less capable, service.
• Informative Error: Return a user-friendly message indicating a temporary degradation (e.g., "Recommendations temporarily unavailable").

This prevents user-facing timeouts and allows the overall system to remain responsive, even if some features are reduced.

The Half-Open state enables automatic, cautious recovery without manual intervention.

• After the configured reset timeout in the Open state expires, the breaker moves to Half-Open.
• A single request or a small batch of requests is allowed to pass as a probe.
• Success Criteria: If the probe request(s) succeed, the breaker assumes the underlying service has recovered and transitions back to Closed, resetting its failure count.
• Failure Criteria: If the probe fails, the breaker immediately returns to the Open state, and the reset timer starts again. This prevents a recovering but still unstable service from being flooded.

Circuit breakers are a critical source of system health telemetry. Their state changes should be treated as prominent operational events.

• State Transition Logging: Log entries or emit events for every state change (Closed → Open, Open → Half-Open, Half-Open → Closed/Open).
• Metrics Export: Expose metrics like request counts, failure rates, and the current state for dashboards and alerts.
• Integration with Distributed Tracing: Annotate traces to show when a call was short-circuited, providing crucial context for debugging latency or error issues.

This observability allows SRE and platform teams to correlate breaker activity with downstream service outages.

Effective implementation requires careful tuning of several parameters:

• Timeout Duration: The call timeout for the wrapped operation, distinct from the breaker's reset timeout.
• Reset Timeout: How long the breaker stays Open before allowing a probe (Half-Open).
• Half-Open Call Limit: The maximum number of concurrent probe calls allowed in the Half-Open state.
• Ignored Exceptions: A list of exception types (e.g., business logic validation errors) that should not count as failures for tripping the breaker.
• Implementation Libraries: Widely used in libraries like Resilience4j (Java), Polly (.NET), and Hystrix (legacy, Java). In service meshes like Istio or Linkerd, circuit breaking is configured at the network proxy layer.

A strategy for handling transient failures by transparently retrying a failed operation. Exponential backoff progressively increases the wait time between retries (e.g., 1s, 2s, 4s, 8s), often combined with random jitter. This reduces load on a recovering system and prevents retry storms.

• Use Case: Ideal for network timeouts, temporary service unavailability, or throttling responses.
• Circuit Breaker Synergy: A circuit breaker often sits above a retry logic layer. Retries handle transient faults; the circuit breaker trips after persistent failures to allow for recovery.

A persistent queue that acts as a holding area for messages or requests that cannot be delivered or processed successfully after multiple attempts. It is a critical companion to circuit breakers and retry logic.

• Error Analysis: DLQs enable offline inspection of failed messages to diagnose root causes (e.g., malformed data, buggy handlers).
• Manual Intervention: Engineers can reprocess, modify, or discard messages after fixing the underlying issue.
• Prevents Data Loss: Ensures failed work items are not silently dropped, maintaining data integrity in asynchronous workflows.

A system design principle where functionality is reduced in a controlled, deliberate manner when a component fails or resources are constrained. A fallback strategy is the specific alternative action taken.

• Circuit Breaker Role: The circuit breaker's open state triggers graceful degradation by failing fast and invoking a fallback.
• Fallback Examples: Returning cached data, default values, a simplified feature, or a user-friendly error message.
• Goal: Maintains core system availability and a usable, if reduced, experience instead of a complete outage.

A dedicated API endpoint (e.g., /health or /ready) that returns the operational status of a service. It is a fundamental mechanism for automated failure detection.

• Liveness Probe: Indicates the service process is running.
• Readiness Probe: Indicates the service is ready to accept traffic (e.g., database connections are healthy).
• Circuit Breaker Integration: Orchestrators (Kubernetes) and load balancers use these endpoints to route traffic away from unhealthy instances. A circuit breaker may use internal health checks to inform its trip/close logic.

Rate limiting controls the request rate a client or service can make, protecting against overuse. Load shedding is the proactive dropping of non-critical requests when a system is under extreme load.

• Failure Prevention: Both techniques prevent resource exhaustion that could lead to cascading failures.
• Circuit Breaker Context: A circuit breaker might trip if a downstream service returns rate-limiting (HTTP 429) errors persistently. Load shedding is a broader system-wide policy to maintain stability, while a circuit breaker protects a caller from a specific failing dependency.