Circuit Breaker

A Circuit Breaker operates as a state machine with three distinct states:

• CLOSED: Normal operation. Requests flow through, and failures are counted.
• OPEN: The circuit is tripped. Requests fail immediately without attempting the operation. A timeout is set.
• HALF-OPEN: After the timeout, a limited number of trial requests are allowed. Success resets the circuit to CLOSED; failure returns it to OPEN. This stateful logic is the core mechanism that differentiates it from simple retries.

The breaker monitors for consecutive failures or a failure rate percentage within a sliding time window. Common configurable thresholds include:

• Failure Count: Trip after N consecutive failures (e.g., 5).
• Failure Rate: Trip if X% of the last Y requests fail (e.g., 50% of last 100 calls).
• Slow Call Rate: Trip if calls exceed a duration threshold, treating slowness as a type of failure. These thresholds allow tuning based on the criticality and expected failure modes of the dependent service.

When in the OPEN state, the circuit breaker implements fail-fast semantics. Instead of letting calls timeout or block, it immediately throws an exception or returns a predefined fallback. This provides several benefits:

• Reduces Latency: Clients experience immediate failure feedback.
• Conserves Resources: Prevents thread pools from being exhausted by waiting on unresponsive services.
• Enables Graceful Degradation: Applications can provide fallback logic (e.g., cached data, default values, simplified functionality).

The HALF-OPEN state enables automatic, probationary recovery. After a configured reset timeout, the circuit allows a single request or a small batch of requests to pass through.

• Success Criteria: If these trial requests succeed, the breaker assumes the underlying fault is resolved and transitions back to CLOSED.
• Failure Criteria: If a trial request fails, the breaker immediately transitions back to OPEN, restarting the reset timeout. This mechanism allows systems to self-heal without manual intervention when intermittent issues are resolved.

Effective circuit breakers are deeply instrumented for observability, emitting metrics, logs, and events for each state transition. Key telemetry includes:

• State Changes: Logs for CLOSED → OPEN, OPEN → HALF-OPEN, etc.
• Request Metrics: Counts of successful, failed, slow, and short-circuited (rejected) calls.
• Latency Histograms: Performance data for calls through the breaker. This data is critical for SLO validation, debugging cascading failures, and tuning breaker thresholds (e.g., Error Budget consumption).

A system design principle where functionality is reduced in a controlled, prioritized manner when a failure or overload condition is detected. The system maintains its core, essential operations while temporarily disabling non-critical features.

• Purpose: To ensure overall system availability and a basic user experience even when components fail.
• Contrast with Circuit Breaker: While a Circuit Breaker fails fast on a specific failing dependency, Graceful Degradation defines a fallback hierarchy for the entire application's feature set.
• Example: A video streaming service might reduce video resolution during peak load but continue playback, rather than failing completely.

An architectural pattern that isolates elements of an application into pools, so that if one fails, the others continue to function. It prevents a single point of failure from consuming all of a system's resources (like threads or connections).

• Mechanism: Resources are partitioned, similar to watertight compartments on a ship. A failure in one 'bulkhead' is contained.
• Synergy with Circuit Breaker: Often used together. A Circuit Breaker can be placed on calls to a service within a specific bulkhead. The failure is isolated to that bulkhead's resource pool.
• Example: A web server might use separate connection pools for its 'user auth service' and its 'product catalog service'. A failure in the auth service won't exhaust connections needed for the catalog.

A pattern where an application transparently retries a failed operation in the expectation that the failure is transient. It is typically implemented with an exponential backoff strategy to avoid overwhelming the recovering service.

• Key Logic: Defines when to retry (e.g., on timeout or 5xx error) and how many times.
• Critical Combo with Circuit Breaker: A naive Retry pattern can worsen outages. A Circuit Breaker should wrap the Retry logic. Once the circuit is OPEN, retries are immediately halted, enforcing the fail-fast behavior.
• Example: A service call fails with a 503 error. The client waits 1 second, retries, waits 2 seconds, retries, then gives up and triggers the circuit breaker.

A safety mechanism that requires a periodic signal or 'heartbeat' to confirm a system or process is operational. If the expected signal is not received within a timeout period, a corrective action (like a failover or shutdown) is automatically triggered.

• Proactive vs. Reactive: While a Circuit Breaker reacts to downstream failures, a Dead Man's Switch proactively monitors the liveness of a component itself.
• Use Case: Ensuring an agent or daemon process is still running and hasn't hung. If the heartbeat stops, a watchdog can restart it.
• Implementation: Often involves a separate monitoring thread or cron job that checks for a recently updated timestamp file.

A predefined alternative course of action an application takes when a primary operation fails. It provides a degraded but acceptable result instead of a complete failure.

• Execution Context: The fallback is invoked when the Circuit Breaker is OPEN or when a call fails and retries are exhausted.
• Types of Fallbacks:
  • Static Default: Return cached data or a default value.
  • Degraded Functionality: Switch to a less accurate or slower algorithm.
  • Alternative Service: Route the request to a secondary, backup service.
• Example: An e-commerce product page, when the recommendation service is down, displays a static list of 'popular items' instead of personalized recommendations.

A fundamental resilience mechanism that sets a maximum duration to wait for a response from a service or operation. If the timeout is exceeded, the call is considered failed, freeing up the calling thread or connection.

• First Line of Defense: Timeouts prevent threads from hanging indefinitely, which is a prerequisite for effective Circuit Breaker operation.
• Relationship: A series of rapid timeout failures is a primary signal for a Circuit Breaker to trip into the OPEN state.
• Configuration: Must be set at multiple layers (e.g., database driver, HTTP client, RPC framework) and should be shorter than the caller's own timeout to allow for fallback logic.