Half-Open State

The defining characteristic of the half-open state is the allowance of a limited, controlled number of test requests to pass through to the failing service. This is a stark departure from the Open State, where all traffic is blocked. The purpose is to validate recovery without risking a flood of traffic that could overwhelm a still-unstable service. Typically, this is configured as a single request or a small, fixed percentage of the normal traffic volume.

The half-open state sits between the Open and Closed states, with strict rules governing its transitions.

• Entering Half-Open: After a predefined timeout period in the Open state, the circuit breaker transitions to Half-Open.
• Exiting on Success: If the probe request(s) succeed, the circuit breaker assumes recovery and transitions back to the Closed State, allowing all traffic to flow normally.
• Exiting on Failure: If the probe request(s) fail, the circuit breaker immediately transitions back to the Open State, restarting the timeout clock. This fail-fast behavior prevents further load on the unhealthy dependency.

A primary design goal of the half-open state is to prevent the thundering herd problem. When a failed service recovers, a sudden surge of retried requests from all waiting clients can immediately overwhelm it again, causing a second failure. The half-open state acts as a traffic governor:

• It allows only a trickle of traffic initially.
• This gives the recovering service time to warm up caches, establish connections, and stabilize.
• Once stability is confirmed, the circuit closes, and traffic ramps up gradually as clients independently retry their operations.

The behavior of the half-open state is tuned through several key parameters:

• Permitted Number of Calls: How many test requests are allowed (often 1).
• Timeout Duration: The length of the Open state before transitioning to Half-Open.
• Success Threshold: Some implementations require multiple consecutive successful probes before closing the circuit.
• Failure Threshold: A single probe failure is often enough to re-open the circuit. Libraries like Resilience4j and Hystrix expose these as configurable properties, allowing adaptation to different service latency and reliability profiles.

Modern fault-tolerance libraries provide robust implementations of the half-open state logic.

• Resilience4j's CircuitBreaker: Uses a ring bit buffer to track the outcomes of the permitted calls in the half-open state. A configurable threshold of successful calls triggers a state transition to CLOSED.
• Hystrix: Allows a single test request in half-open mode. Its result dictates the next state.
• Envoy Proxy / Service Mesh: Uses outlier detection to eject unhealthy hosts, which is a form of circuit breaking. A host is tested periodically (a half-open probe) before being reintroduced to the load balancing pool.

The half-open state's probe mechanism is distinct from, but complementary to, active health checks.

• Half-Open Probes: Are real user traffic or synthetic requests that follow the actual application execution path. They test the full integration.
• Active Health Checks: Are out-of-band, periodic requests (e.g., to a /health endpoint) that check basic liveness. A service might pass a health check but still fail under real load. A robust system often uses both: health checks for initial liveness detection, and the circuit breaker's half-open state for validating functional readiness under operational conditions.

The original catalyst for popularizing the circuit breaker pattern in microservices. Its implementation is now in maintenance mode but defined key behaviors.

• Sleep Window: Hystrix's term for the time in the open state before transitioning to half-open.
• Single Test Request: In half-open state, it allowed one request through. If it failed, the circuit immediately re-opened.
• Metrics-Driven: Used a rolling statistical window to track success/error percentages, feeding the half-open decision logic.
• Architectural Influence: Directly inspired later libraries and is a foundational case study in chaos engineering and resilience.

Managed services and SDKs provide built-in circuit breaking for their client libraries, abstracting the implementation.

• AWS SDK Retry & Throttling: SDKs have default retry logic with exponential backoff and jitter. While not a classic three-state breaker, they stop retrying after max attempts, acting as a de facto open circuit.
• Azure SDK Resilience: The .NET Azure SDK uses Polly internally, providing built-in retry policies and circuit breaker patterns for service calls.
• Google Cloud Client Libraries: Libraries often include graceful degradation and automatic retry mechanisms. For fine-grained control, developers must implement patterns like Resilience4j or Polly around the SDK calls.
• Managed Service Endpoints: Cloud load balancers and API gateways often provide health check-based routing and outlier detection, performing circuit breaking at the network tier.

The foundational software design pattern that the Half-Open State implements. It detects failures and prevents an application from repeatedly attempting an operation that is likely to fail. Its three core states are:

• Closed: Requests flow normally to the dependency.
• Open: Requests fail immediately without calling the dependency.
• Half-Open: A limited number of test requests are allowed to probe for recovery. The pattern's primary goal is to stop cascading failures and provide time for a failing downstream service to recover.

A retry strategy often used in conjunction with a circuit breaker. When a request fails, the system waits for a progressively longer interval before retrying. For example, delays might follow a sequence like 1s, 2s, 4s, 8s. This strategy:

• Reduces load on a struggling dependency.
• Increases the probability of successful recovery by giving the service more time.
• Is commonly implemented in the client-side logic that triggers before a circuit breaker trips to handle transient faults.

A diagnostic probe used to determine a service's operational status. In the context of a Half-Open State, the limited test requests function as active health checks.

• Liveness Probe: Determines if the service is running.
• Readiness Probe: Determines if the service is ready to accept traffic. Automated, periodic health checks can inform when a circuit breaker should transition from Open to Half-Open, initiating the recovery verification process.

A complementary resilience pattern that isolates elements of an application into independent pools of resources (threads, connections, instances).

• Prevents a single point of failure from consuming all resources and cascading to other system parts.
• When used with circuit breakers, a failure in one bulkhead (e.g., payment service) can be contained, while other bulkheads (e.g., product catalog) remain operational. This isolation makes the Half-Open State's recovery testing more stable and contained.

A predefined alternative action a system executes when a primary operation fails. It is a key behavior during the Open and potentially Half-Open states of a circuit breaker.

• Static Response: Returning cached data or a default message.
• Degraded Functionality: Switching to a less optimal but available service.
• Graceful Degradation: Maintaining core operations while disabling non-essential features. A well-designed fallback allows the system to remain partially functional while the circuit breaker probes for recovery in the Half-Open State.