Circuit Breaker

A circuit breaker operates through a finite state machine with three primary states:

• CLOSED: The normal operating state. Requests flow through to the downstream service. Failures are counted.
• OPEN: The circuit trips after failures exceed a threshold. All requests immediately fail fast without attempting the operation, allowing the failing service time to recover.
• HALF-OPEN: After a configured timeout, the circuit allows a single test request. Its success resets the circuit to CLOSED; its failure returns it to OPEN.

The circuit monitors for failures to decide when to trip. Key configurable parameters include:

• Failure Threshold: The count (e.g., 5 consecutive failures) or percentage of failures within a sliding time window that triggers the circuit to OPEN.
• Timeout Duration: The period the circuit remains OPEN before transitioning to HALF-OPEN.
• Supported Failure Types: Typically includes network timeouts, connection refused errors, and HTTP 5xx server errors. Application-level business logic errors are usually excluded.

When the circuit is OPEN, the primary benefit is fail-fast behavior. Instead of waiting for a timeout, the caller receives an immediate error (e.g., CircuitBreakerOpenException). This is often combined with a fallback strategy to maintain graceful degradation, such as:

• Returning a cached default value.
• Using a stale but available data source.
• Providing a user-friendly message indicating temporary unavailability. This prevents thread pool exhaustion and resource contention in the calling service.

Circuit breakers and retries are complementary but distinct patterns that must be composed carefully to avoid antagonism.

• Retry Logic: Attempts the same operation multiple times in case of transient failures.
• Risk: Aggressive retries on a failing service can worsen overload and delay recovery.
• Best Practice: Place the circuit breaker outside the retry logic. The retry handles transient faults for a single request, while the circuit breaker protects the system after multiple requests indicate a sustained failure.

Effective circuit breakers are heavily instrumented to provide operational visibility. Critical metrics include:

• Circuit State Changes (CLOSED → OPEN → HALF-OPEN): The primary health signal.
• Request Counts: Total, successful, and failed requests.
• Failure Rate: The percentage of failed requests.
• Latency: Percentile latencies for successful calls. These metrics are essential for tuning thresholds (e.g., failureThreshold, slowCallDuration) and understanding systemic dependencies.

A retry algorithm that progressively increases the waiting time between consecutive attempts to call a failing service. This prevents overwhelming a recovering system and is often used in conjunction with a circuit breaker.

• Key Mechanism: The delay between retries increases exponentially (e.g., 1s, 2s, 4s, 8s).
• Purpose: Reduces load on a struggling backend, giving it time to recover.
• Edge AI Context: Used when an edge device's inference request to a remote service or another device fails, ensuring the local system doesn't waste power and bandwidth on futile, rapid retries.

A resilience pattern that isolates elements of an application into pools, so if one fails, the others continue to function. It prevents a single point of failure from cascading and exhausting all system resources.

• Key Mechanism: Resources (threads, connections, memory) are partitioned into isolated groups for different services or operations.
• Purpose: Limits the "blast radius" of a failure, ensuring partial system availability.
• Edge AI Context: On a device running multiple models (e.g., object detection and audio classification), a failure in one model's execution pool won't starve resources from other, still-functioning models.

A pattern that enables an application to handle transient failures by transparently retrying a failed operation. It is a precursor to the circuit breaker; if retries consistently fail, the circuit opens.

• Key Mechanism: Automatically re-attempts an operation that failed due to a transient fault (e.g., network timeout, temporary unavailability).
• Purpose: Improves application stability by masking short-lived failures from the end user.
• Edge AI Context: Used when an edge device attempts to send telemetry data to a cloud endpoint or fetch a model update, handling brief network interruptions common in field deployments.

A predefined alternative course of action when a primary operation fails. It provides a graceful degradation of service rather than a complete failure.

• Key Mechanism: When a call fails (and a circuit is open), the system executes a fallback method, such as returning cached data, a default value, or a simplified local computation.
• Purpose: Maintains basic functionality and user experience during partial outages.
• Edge AI Context: Critical for edge AI; if a cloud-connected model fails, the device can fall back to a smaller, less accurate on-device model or use last-known-good inference results to maintain autonomous operation.

A pattern that ensures a process or request does not wait indefinitely for a response. It sets a maximum time limit for an operation to complete, after which it is considered a failure.

• Key Mechanism: Every call to an external service is bound by a configurable timeout duration.
• Purpose: Prevents resource exhaustion (blocked threads, memory) and provides predictable latency bounds.
• Edge AI Context: Essential for real-time edge systems; an inference request to a local accelerator or a neighboring device must complete within a strict timeframe to maintain system control loops.

A periodic probe or request sent to a service component to determine its operational status and readiness to accept work. Circuit breakers often use health checks to test if a failed service has recovered.

• Key Mechanism: A lightweight endpoint (e.g., /health) returns the service's status (UP, DOWN). Orchestrators use this to manage traffic routing and lifecycle.
• Purpose: Enables automated detection of unhealthy instances for removal from a pool and verification of recovery.
• Edge AI Context: Edge orchestrators ping device twins or local inference servers to verify an edge node is alive and its model is loaded before routing inference jobs to it.