Circuit Breaker

A circuit breaker operates as a finite state machine with three distinct states, transitioning based on failure counts and timeouts.

• CLOSED: The normal operational state. Requests flow to the dependency. Failures are counted.
• OPEN: The tripped state. Requests fail immediately without calling the dependency. A timer is set for a reset timeout.
• HALF-OPEN: A trial state entered after the timeout. A limited number of test requests are allowed. Success moves the breaker back to CLOSED; failure returns it to OPEN.

The transition from CLOSED to OPEN is governed by configurable thresholds measured within a time window.

• Failure Count/Percentage: The breaker trips after N consecutive failures or a percentage of failed calls (e.g., 50% failure rate).
• Sliding Time Window: Failures are counted within a recent time period (e.g., last 60 seconds), ensuring the breaker responds to current system health, not ancient history.
• Example: A common configuration is to open the circuit after 5 failures within a 10-second window.

The primary benefit of an OPEN circuit breaker is failing fast, which prevents cascading failures and resource exhaustion.

• Immediate Feedback: Downstream clients receive an error (e.g., 503 Service Unavailable) immediately, without waiting for a timeout from the unhealthy service.
• Graceful Fallbacks: Applications can implement fallback logic, such as returning cached data, default values, or a degraded user experience.
• Resource Protection: It prevents threads, connections, and memory from being tied up by calls to a failing service.

After tripping, the breaker doesn't stay open indefinitely. A reset timeout allows for periodic health checks.

• Reset Timeout: After entering the OPEN state, a timer is set (e.g., 30 seconds). No calls pass during this period.
• Half-Open Trial: When the timer expires, the breaker enters HALF-OPEN. A single, or small batch of, trial requests are permitted.
• State Resolution: If the trial succeeds, the breaker assumes recovery and resets to CLOSED. If it fails, the breaker re-opens and the timeout restarts.

Effective circuit breakers are heavily instrumented to provide visibility into system health.

• State Transition Metrics: Counters for transitions to OPEN, HALF-OPEN, and CLOSED.
• Request Metrics: Track calls permitted, blocked (while OPEN), and failed.
• Alerting: Alert on prolonged OPEN states, indicating a chronic dependency failure.
• Integration: Metrics should feed into broader data observability and pipeline monitoring dashboards.

Exponential backoff is a retry strategy where the waiting time between consecutive retry attempts for a failed operation increases exponentially (e.g., 1s, 2s, 4s, 8s). It is a common companion to a circuit breaker pattern.

• Purpose: Reduces load on a struggling dependency, gives it time to recover, and prevents retry storms that exacerbate failures.
• Integration: A circuit breaker often implements a retry logic with exponential backoff before tripping to the OPEN state. After a reset timeout, it may use a similar backoff for probe requests.

An idempotent consumer is a message processing component designed such that processing the same message multiple times has the exact same effect as processing it once. This is critical for recovery after failures and retries.

• Resilience Link: When a circuit breaker resets and allows traffic to resume, or when messages are replayed from a DLQ, duplicate deliveries are likely. Idempotency ensures system state remains correct.
• Implementation Techniques: Using unique message IDs with a processed-IDs store, or designing state updates to be naturally idempotent (e.g., SET status = 'processed').

A Service Level Objective (SLO) is a specific, measurable target for the reliability or performance of a service, such as a data freshness or latency threshold. Circuit breakers are a tactical tool to help meet SLOs.

• Relationship: A circuit breaker's configuration (failure thresholds, timeouts) is often derived from SLOs. For example, if an SLO mandates 99.9% availability for a database dependency, the circuit breaker's error threshold is tuned to trip before violating that SLO for dependent services.
• Error Budgets: SLOs define an acceptable error budget. A circuit breaker failing fast preserves the error budget of the calling service by avoiding long, resource-draining calls to a failing dependency.

Tail latency refers to the high-percentile latencies (e.g., P95, P99) in a distribution of response times. P99 latency is the value below which 99% of observations fall, representing the worst-case user experience.

• Circuit Breaker Impact: A poorly performing dependency often manifests as spiking tail latency before complete failure. A circuit breaker monitoring for slow calls (via timeouts) can trip based on degrading P99/P99.9 latency, not just binary failures.
• Proactive Protection: By rejecting requests that are likely to timeout, a circuit breaker prevents the calling service's own latency from blowing out, thereby protecting its SLOs.