Cascading Failure

When a single node or service fails, its workload is automatically redistributed to the remaining healthy nodes. This sudden, unplanned increase in load can push those nodes beyond their capacity, causing them to fail and further concentrating the load on an ever-shrinking pool of resources. This is a primary driver in cloud autoscaling failures and database overloads.

• Example: A primary database replica fails. All read/write traffic shifts to a secondary replica, which lacks the compute capacity for the doubled load, causing it to fail as well.

Failures in one layer can cause runaway consumption of shared resources in another, such as CPU, memory, network bandwidth, or database connections. A failing service that enters a retry loop without backpressure can spawn infinite threads or connections, starving other healthy services of the resources they need to operate.

• Critical resources: Database connection pools, thread pools, and memory are common targets. Exhaustion here creates a system-wide deadlock.

Modern microservices architectures create deep, synchronous dependency chains. If Service C fails, Service B—which calls it—may block waiting for a response. This blockage then propagates upstream to Service A. Without timeouts and circuit breakers, these blocked threads accumulate, leading to thread pool exhaustion and making Service A and B appear failed to their own callers, even though their core logic is intact.

A failure condition creates a signal that induces more of the same failure condition, accelerating the collapse. A classic example is a retry storm: clients, using aggressive retry logic without jitter, simultaneously retry failed requests. This synchronized bombardment prevents the recovering service from ever stabilizing, creating a self-sustaining failure state.

• Other examples: Cache stampedes and thundering herds.

The bulkhead pattern is inspired by ship compartments. Without it, a failure in one subsystem can drain resources (like threads or memory) from the entire application. A cascading failure occurs when there are no isolation boundaries; a fault in a non-critical feature (e.g., a reporting module) can consume all available resources, causing the failure of mission-critical transaction processing.

A resilient system is designed for graceful degradation, maintaining core functionality when non-essential dependencies fail. Cascading failures often occur when this principle is violated. If the system cannot isolate the failure of a non-critical component and instead allows that failure to block or corrupt core workflows, a limited outage escalates into a full-system collapse.

A flow control mechanism where a system receiving data faster than it can process signals the upstream sender to slow down. This prevents the receiver from being overwhelmed, which can lead to memory exhaustion, crashes, and failure propagation.

• Reactive Streams: Frameworks like Project Reactor or RxJava implement backpressure protocols.
• Cascade Prevention: Stops fast producers from flooding and crashing slow consumers, breaking a critical link in a failure chain.

The proactive strategy of rejecting or dropping non-critical requests when a system is under extreme load. This preserves finite resources (CPU, memory, I/O) for critical operations, preventing a total collapse and enabling graceful degradation.

• Triage Logic: Systems identify low-priority traffic (e.g., background jobs, non-essential API features) to reject first.
• Direct Cascade Mitigation: Prevents a traffic surge from consuming all resources and causing dependent services to fail due to timeouts or starvation.

The design principle where a system maintains partial or reduced functionality when some of its components fail, rather than failing completely. It is the intentional outcome of successful failure containment patterns.

• Example: An e-commerce site disables product recommendations (a non-critical feature) when the recommendation service is down, but the core cart and checkout remain operational.
• Contrasts with Cascading Failure: A cascade leads to total outage; graceful degradation is a controlled, partial failure state.