Circuit Breaker

A circuit breaker operates through a finite state machine with three primary states:

• Closed: The normal operational state. Requests flow through, and failures are counted.
• Open: The circuit has 'tripped.' Requests fail immediately without attempting the operation, returning a fallback or error.
• Half-Open: A trial state after a timeout period. A limited number of test requests are allowed. Success resets the circuit to Closed; failure returns it to Open.

The transition from Closed to Open is triggered by configurable thresholds that detect a failing dependency.

• Failure Count/Percentage: The circuit trips after a defined number of consecutive failures or a rolling percentage of failed calls (e.g., 50% failure rate over the last 100 requests).
• Timeout Detection: Individual calls that exceed a specified duration can be counted as failures, protecting against latency spikes.
• Exception Classification: The system distinguishes between expected business logic failures (which may not trip the circuit) and true system failures like network timeouts or 5xx HTTP errors.

The Half-Open state enables automatic, cautious recovery without manual intervention.

• Reset Timeout: After the circuit is Open for a configured duration, it transitions to Half-Open.
• Probe Requests: A limited number of trial requests (often just one) are permitted to test if the underlying service has recovered.
• State Resolution: If the probe request(s) succeed, the circuit resets to Closed. If they fail, it immediately returns to Open, restarting the reset timeout. This prevents flooding a recovering service.

When the circuit is Open or a call times out, the pattern mandates a defined fallback strategy to maintain partial functionality.

• Default Values: Return cached data, static content, or empty/default results.
• Alternative Services: Route the request to a backup or degraded service tier.
• Fast Failure: Immediately return a meaningful error (e.g., "Service temporarily unavailable") instead of letting the client application hang. This is crucial for defining clear Service Level Objectives (SLOs) for availability.

Effective circuit breakers are deeply instrumented to provide critical observability signals for Agent Deployment Observability.

• State Transition Metrics: Emit events and metrics for every state change (Closed → Open, Open → Half-Open, etc.).
• Request Telemetry: Track call counts, latencies, and failure rates segmented by circuit state.
• Integration with Distributed Traces: Correlate circuit breaker activity with end-to-end request traces to visualize exactly where and why a circuit tripped within a complex workflow.

A resilience strategy where a client automatically re-attempts a failed operation a predetermined number of times. It is often used before a circuit breaker trips, but must be implemented with exponential backoff and jitter to avoid overwhelming a failing service.

• Key Mechanism: Implements a delay between retries that grows exponentially (e.g., 1s, 2s, 4s, 8s).
• Jitter: Adds random variation to backoff delays to prevent synchronized retry storms from multiple clients.
• Use Case: Ideal for transient faults like network timeouts or temporary service unavailability.

An isolation pattern that partitions system resources (like thread pools, connections, or memory) into separate groups, similar to watertight compartments on a ship. This prevents a failure in one part of the system from consuming all resources and causing a total outage.

• Implementation: Uses dedicated connection pools or thread pools for different downstream services or operations.
• Benefit: A failure in Service A will exhaust only its allocated pool, leaving resources for Service B and C fully available.
• Analogy: In a microservices architecture, it's the equivalent of failure domain isolation.

A strategy that provides an alternative response or action when a primary operation fails. It is the action taken after a circuit breaker opens, allowing the system to continue operating with degraded functionality.

• Types of Fallbacks:
  • Static: Return a default cached value or a neutral response.
  • Degraded: Switch to a less-capable but available backup service.
  • Plausible: For non-critical features, return a placeholder or null.
• Critical Design: The fallback logic itself must be simple and reliable to avoid introducing new failure points.

A fundamental control mechanism that defines the maximum duration to wait for a response from a service or operation. It is a primary input signal for a circuit breaker; repeated timeouts will trigger the breaker to open.

• Purpose: Prevents a client from waiting indefinitely for a non-responsive service, freeing up resources.
• Aggressive vs. Conservative: Setting timeouts requires balancing user experience (aggressive) against allowing for legitimate processing time (conservative).
• Layer Dependency: Timeouts should be configured at every level: connection, request, and application logic.

A distributed systems practice where a request's initial timeout (deadline) is propagated through all downstream service calls. This ensures the entire call chain respects the original caller's latency budget and allows for coordinated cancellation.

• Mechanism: The initial service includes a deadline timestamp in the request context (e.g., via gRPC metadata or HTTP headers). Each subsequent service checks this deadline before starting work.
• Benefit: Prevents "useless work" in deep service chains when the caller has already given up, conserving system resources.
• Relation to Circuit Breaker: Complements circuit breakers by providing a proactive, request-scoped failure limit rather than a service-scoped one.

A periodic query to determine if a service instance is operational and ready to accept traffic. While a circuit breaker monitors for failure during execution, health checks assess readiness before execution.

• Liveness Probe: Answers "Is the process running?" Failure triggers a restart.
• Readiness Probe: Answers "Is the service ready for requests?" Failure triggers removal from load balancer pools.
• Integration: A sophisticated circuit breaker implementation might consult health check status when deciding to transition from Half-Open to Closed state.