Failure Rate

Failure rate is not a static number but a dynamic metric calculated over a rolling time window. This window, often configurable (e.g., 60 seconds), slides forward continuously, ensuring the metric reflects recent system performance rather than historical averages. For example, a service might calculate the rate as (failed requests in last 60s) / (total requests in last 60s). This provides a real-time view of health, crucial for triggering fail-fast mechanisms like a circuit breaker before a cascade begins.

The failure rate is the most common metric used to trip a circuit breaker from a closed to an open state. An error threshold (e.g., 50% failure rate over 10 seconds) is configured. When the calculated rate exceeds this threshold, the breaker opens, preventing further requests to the failing dependency. This protects the calling system from wasting resources and allows the failing service time to recover. The transition often involves moving through a half-open state to test recovery.

Failure rate is a normalized percentage, not a raw count. This is essential for meaningful thresholds in systems with variable load. An error count of 10 might be catastrophic for a service receiving 12 requests per minute but negligible for one handling 10,000. By using a rate, configurations become load-agnostic and more portable. It is often paired with a minimum request volume to prevent a single early failure from tripping the breaker during low-traffic periods.

In Site Reliability Engineering (SRE), failure rate is directly tied to Service Level Objectives (SLOs) and error budgets. An SLO might define that 99.9% of requests must succeed. The inverse (0.1%) is the allowable error budget. Monitoring the failure rate against this budget allows for SLO-based tripping, where a circuit breaker can be configured to open when error consumption threatens to exhaust the budget, proactively preserving service reliability.

Not all errors contribute equally to the failure rate in resilience logic. Systems often differentiate between:

• Transient faults (e.g., network timeouts): May trigger retry logic with exponential backoff but not immediately increase the failure rate for circuit breaking.
• Persistent business logic errors: Likely counted immediately.
• Caller-induced errors (e.g., 4xx client errors): Often excluded from the failure rate calculation, as they indicate a problem with the request, not the service's health. This classification prevents unnecessary breaker trips.

Advanced implementations move beyond static thresholding. An adaptive circuit breaker analyzes trends in the failure rate, latency, and system load to dynamically adjust its trip thresholds. For instance, during a known traffic spike, it might temporarily tolerate a higher failure rate. This requires continuous analysis of the rolling window data and aligns with chaos engineering principles, where systems are tested to respond gracefully to variable failure conditions.