Circuit Breaker

A circuit breaker operates as a finite state machine with three distinct states:

• Closed: Requests flow normally to the service. Failures are counted.
• Open: All requests are immediately failed without attempting the call. A timer is set.
• Half-Open: After the timer expires, a limited number of trial requests are allowed. Success resets the breaker to Closed; failure returns it to Open. This stateful logic is the core mechanism that differentiates it from simple retry logic.

The transition from Closed to Open is triggered by configurable thresholds that detect a failing service. Common metrics include:

• Failure Count: A consecutive or rolling count of failed calls (e.g., 5 failures).
• Failure Ratio: A percentage of failed calls within a time window (e.g., 50% failure rate over 60 seconds).
• Timeout Duration: Treating slow calls as failures. This is critical for preventing thread pool exhaustion in synchronous systems.

When the circuit is Open, calls must be handled gracefully. A robust implementation provides a fallback mechanism, such as:

• Returning a cached or default response.
• Queuing the request for later processing.
• Failing fast with a meaningful error (e.g., "Service temporarily unavailable"). This ensures the user experience degrades gracefully rather than failing catastrophically or hanging indefinitely.

The Half-Open state enables automatic recovery. After a configured reset timeout, the breaker allows one or a few trial requests through:

• Success: If these succeed, the breaker assumes the service is healthy and resets to Closed.
• Failure: If they fail, the breaker immediately trips back to Open, restarting the timeout. This probe mechanism prevents a recovered service from being immediately overwhelmed by a flood of pent-up requests.

Effective circuit breakers are deeply instrumented. Key telemetry includes:

• State transitions (Closed → Open, etc.) logged as high-severity events.
• Request counts, success/failure rates, and latency percentiles for each state.
• Current state exposed as a health check or metrics endpoint (e.g., /health or Prometheus gauge). This data is essential for SREs to diagnose systemic issues and tune threshold parameters.

A circuit breaker is complementary to, but distinct from, retry logic with exponential backoff.

• Retry Policy: Handles transient, momentary failures (e.g., network blip). It operates at the call level.
• Circuit Breaker: Handles persistent, systemic failures (e.g., downstream service crash). It operates at the service level. Best practice is to combine them: use retries for momentary issues within a Closed circuit, but the breaker will trip if retries consistently fail, moving the system into a protective state.

A retry policy is a set of rules governing the automatic re-attempt of a failed API call. It is a complementary pattern to a circuit breaker, designed to handle transient faults. Effective policies incorporate:

• Exponential Backoff: Progressively increasing wait times between retries (e.g., 1s, 2s, 4s, 8s).
• Jitter: Adding randomness to backoff intervals to prevent retry storms from synchronized clients.
• Maximum Attempts: A cap on retries before definitive failure is declared. A circuit breaker often sits above the retry logic; if the breaker is open, no retries are attempted, preventing wasted resources on a known-bad endpoint.

Fallback strategies are predefined contingency plans executed when a primary operation fails. In the context of a circuit breaker, a fallback provides a graceful degradation of service. Common strategies include:

• Default Response: Returning a cached or static default value.
• Alternative Service: Routing the request to a backup or degraded-functionality service.
• Queue for Later: Placing the request in a dead-letter queue for asynchronous retry.
• Informative Error: Providing a user-friendly message while logging the failure for ops. A robust system combines circuit breakers with intelligent fallbacks to maintain user experience during partial outages.

The bulkhead pattern isolates elements of an application into pools so that if one fails, the others continue to function. It's a sister pattern to the circuit breaker for containing failure.

• Resource Isolation: Dedicating separate thread pools, connections, or memory segments to different service calls.
• Failure Containment: A failure in one bulkhead (e.g., payment service) does not exhaust resources needed by another (e.g., product catalog).
• Combined Use: Used alongside circuit breakers; a breaker can be placed on each isolated bulkhead. This prevents a single point of failure from cascading across the entire system, a concept critical for multi-agent AI systems where different tools have independent failure modes.

A deadline (or timeout) is a maximum duration allowed for an operation to complete. It is a fundamental guardrail used in conjunction with circuit breakers.

• Prevents Hanging Calls: Ensures a client isn't blocked indefinitely waiting for a unresponsive service.
• Triggers Failure Count: A timed-out request is typically counted as a failure by the circuit breaker's sliding window.
• Aggressive vs. Conservative Settings: Short timeouts improve responsiveness but may increase false failure rates. Long timeouts risk resource exhaustion. The circuit breaker's role is to stop making calls altogether after too many timeouts occur, protecting the system.

Health checks are active probes sent to a service to determine its operational status. They inform circuit breaker state transitions, especially the transition from Open to Half-Open.

• Liveness Probe: Answers "Is the service running?"
• Readiness Probe: Answers "Is the service ready to accept traffic?"
• Synthetic Transactions: Simulating a real user request to validate full functionality. In advanced implementations, a circuit breaker in the Half-Open state may use a health check or a single trial request to test the service before closing the circuit and resuming normal traffic.

Rate limiting controls the volume of requests a client can send to a service in a given timeframe. While a circuit breaker reacts to failures, rate limiting reacts to volume to prevent overload.

• Client-Side Throttling: A client self-regulates its request rate based on server feedback (e.g., 429 Too Many Requests responses).
• Server-Side Protection: The service rejects excess requests to protect its own resources.
• Interaction with Circuit Breaker: A high rate of 429 or 503 (Service Unavailable) responses from a service can be interpreted as failures by a client-side circuit breaker, triggering it to open. This protects the client from repeatedly hitting an overloaded endpoint.