Circuit Breaker Pattern

In the Closed state, the circuit breaker allows all requests to pass through to the protected service. A failure counter tracks consecutive or rolling-window failures. If the failure count exceeds a predefined threshold within a time window, the breaker transitions to the Open state. This state represents healthy operation where the downstream dependency is responsive and returning successful results.

When the circuit is Open, the breaker immediately rejects all incoming requests without attempting to call the downstream service. This fail-fast mechanism returns an error or fallback response instantly, preserving thread pool resources and preventing cascading timeouts. The system avoids the thundering herd problem where thousands of blocked threads wait for an unresponsive service. A cooldown timer begins upon entering this state.

After the cooldown period expires, the breaker transitions to Half-Open. A limited number of trial requests are permitted to probe the downstream service. If these requests succeed, the breaker resets to Closed. If any trial request fails, the breaker immediately returns to Open and restarts the cooldown timer. This prevents premature recovery attempts from overwhelming a partially degraded service.

Circuit breakers require careful tuning of two critical parameters:

• Failure Rate Threshold: The percentage of failed requests (e.g., 50%) that triggers the Open state
• Sliding Window Size: The time interval or request count used to calculate the failure rate
• Cooldown Duration: How long the breaker stays Open before attempting Half-Open
• Minimum Request Threshold: Prevents tripping on low traffic volumes where a single failure skews the rate Misconfigured thresholds cause either false positives (unnecessary tripping) or false negatives (failing to detect degradation).

When the circuit is Open, the system must provide a graceful degradation response. Common fallback patterns include:

• Cached Response: Return stale but acceptable data from a local cache
• Default Value: Provide a safe, pre-configured default response
• Empty Response: Return an empty list or null result
• Alternative Service: Route to a redundant backup service or different region
• Queued Retry: Persist the request for later processing when the circuit closes The fallback must never throw exceptions that propagate upstream.

Production circuit breakers must expose metrics for operational visibility:

• State Transitions: Count of Closed→Open, Open→Half-Open, Half-Open→Closed events
• Success/Failure/Rejection Counts: Tracked per state for throughput analysis
• Latency Percentiles: p50, p95, p99 response times for successful calls
• Bulkhead Saturation: Thread pool utilization when the breaker is Closed These metrics feed into alerting rules that notify operators of repeated circuit trips, indicating systemic downstream issues requiring intervention.

A resilience design pattern that isolates system resources into independent pools to prevent a failure in one component from cascading to the entire system.

• Partitions thread pools, connection pools, or CPU resources by service boundary
• A single exhausted pool cannot starve other critical operations
• Commonly implemented in payment authorization flows to isolate card network gateways from each other
• Named after ship compartmentalization: one breached hull section won't sink the vessel

A transient fault-handling strategy where failed requests are reattempted with progressively increasing delays between attempts to avoid overwhelming a recovering service.

• Delay formula: base_delay * 2^attempt with random jitter to prevent thundering herd
• Works in tandem with circuit breakers: retries handle blips, breakers handle sustained outages
• Critical for idempotency key generation to prevent duplicate transaction submissions
• Common configuration: 3 retries with 100ms, 200ms, 400ms delays

The practice of setting maximum wait durations for external service calls to prevent resource exhaustion from hung connections.

• Connection timeout: max time to establish a TCP connection
• Read timeout: max time to wait for a response after connection
• Must be tuned below the circuit breaker's trip threshold to avoid false positives
• For P99 latency targets in fraud scoring, timeouts are typically set at 200-500ms for critical path services

Alternative execution paths invoked when a circuit breaker opens, providing graceful degradation instead of complete failure.

• Static fallback: Return a cached or default response (e.g., approve low-risk transactions based on last-known-good rules)
• Stale data fallback: Serve slightly outdated enrichment data from a local cache
• Degraded mode: Skip non-critical checks like device fingerprinting while maintaining core velocity checks
• Essential for maintaining authorization flow continuity during downstream outages

A dedicated API endpoint that circuit breakers poll to determine if a downstream service has recovered and is ready to accept traffic again.

• Liveness check: 'Is the process running?'
• Readiness check: 'Can the service accept requests?' (database connected, caches warmed)
• Circuit breaker transitions to half-open state based on successful health check responses
• Should be lightweight and avoid cascading dependencies to prevent false negatives

A secondary message queue where unprocessable transactions are routed when all retry and circuit breaker policies are exhausted, ensuring no data is silently lost.

• Stores the original payload, error metadata, and failure timestamp
• Enables asynchronous manual investigation without blocking the real-time pipeline
• Critical for financial audit trails: every declined or failed transaction must be traceable
• Often paired with alerting to notify operations teams of DLQ buildup