Failover

In this classic pattern, a primary active node handles all traffic while one or more passive nodes remain idle or in a read-only state. Upon failure detection, traffic is redirected to a standby node, which must load its state and become active.

• Cold Standby: The backup system is powered off or not running the application, leading to longer recovery times (Recovery Time Objective - RTO).
• Warm Standby: The backup system is running and has pre-loaded data/software, allowing for faster failover, often used with database replicas.
• Use Case: Traditional database clusters, disaster recovery sites.

Multiple nodes are active and simultaneously handle traffic, typically behind a load balancer. If one node fails, the load balancer simply stops routing requests to it, distributing the load among the remaining healthy nodes.

• Provides near-instantaneous failover and maximizes resource utilization.
• Requires application statelessness or a shared, consistent data layer (e.g., a distributed cache or database) to ensure all nodes operate on the same data.
• Introduces complexity for stateful services, often solved via sharding or externalized session stores.
• Use Case: Stateless web server fleets, microservices behind an API gateway.

A critical pattern for distributed systems where a single leader must be designated to coordinate actions (e.g., managing a replicated log). Upon leader failure, the remaining nodes run a consensus algorithm to elect a new leader.

• Algorithms: Raft and Paxos are standard protocols for achieving consensus in the presence of failures.
• Failure Detection: Uses heartbeats; a leader is presumed dead if heartbeats stop.
• State Transfer: The new leader must synchronize state with followers to ensure consistency.
• Use Case: Distributed databases (etcd, Consul), coordination services (Apache ZooKeeper).

A method for making a service fault-tolerant by replicating a deterministic service (the state machine) across multiple servers. All replicas start from the same state and process the same sequence of commands in the same order.

• Core Principle: If replicas are deterministic and start from the same state, they will produce identical outputs and state transitions.
• Log Replication: A consensus protocol (like Raft) is used to agree on the order of commands in a replicated log.
• Failover: If the primary replica fails, any other replica with the latest log can take over seamlessly.
• Use Case: Core infrastructure for strong consistency in systems like distributed databases and financial transaction processors.

A fail-fast pattern that prevents a system from repeatedly trying to execute an operation that's likely to fail, protecting downstream services and preventing cascading failures.

• Three States:
  • Closed: Requests flow normally.
  • Open: Requests fail immediately without attempting the operation.
  • Half-Open: A limited number of test requests are allowed to see if the underlying fault is resolved.
• Triggers: Failures exceed a defined threshold (e.g., 5 failures in 10 seconds).
• Fallback: When the circuit is open, a system can execute a predefined fallback strategy (e.g., return cached data, default response).
• Use Case: Inter-service communication in microservices, external API calls.

A design philosophy where a system maintains partial, reduced functionality when a non-critical component fails, rather than failing completely. This is implemented via fallback strategies.

• Hierarchy of Fallbacks:
  1. Retry with exponential backoff.
  2. Switch to a redundant backup service.
  3. Return stale but acceptable cached data.
  4. Provide a simplified, non-personalized experience.
  5. Display a user-friendly message while preserving core UI functionality.
• Goal: Maximize availability and user experience even during partial outages.
• Use Case: E-commerce sites showing cached product info if the recommendation engine fails, maps showing a basic grid if live traffic data is unavailable.

A design pattern that prevents a software component from repeatedly attempting an operation that is likely to fail, thereby stopping cascading failures and allowing the system to degrade gracefully. When failures exceed a threshold, the circuit "opens" and fails fast for a period before allowing a "half-open" test request. This is a proactive complement to reactive failover, protecting upstream services from downstream outages.

• Key Use: Protecting services from calling unhealthy dependencies.
• Example: An API gateway stops routing requests to a failed payment service for 30 seconds after 5 consecutive timeouts.

The duplication of critical components or functions of a system with the intention of increasing reliability. This is the prerequisite infrastructure that makes failover possible. Redundancy can be active (hot standby, ready to take over) or passive (cold standby, requires initialization).

• N+1 Redundancy: A system has one extra component beyond what is needed for operation.
• 2N Redundancy: A fully mirrored system where capacity is doubled.
• Geographic Redundancy: Systems replicated across different data centers or cloud regions to survive site-level disasters.

A dedicated API endpoint (e.g., /health or /ready) that returns the operational status of a service. This is the primary signal used by load balancers, service meshes, and orchestration platforms (like Kubernetes) to determine if a failover should be triggered.

• Liveness Probe: Indicates if the service process is running.
• Readiness Probe: Indicates if the service is ready to accept traffic (e.g., database connections are established).
• Startup Probe: Used for slow-starting containers to prevent premature failure marking.

A system design principle where functionality is reduced in a controlled manner when a component fails or resources are constrained, preserving core operations. While failover switches to a backup, graceful degradation manages the user experience when full functionality is unavailable.

• Example: A video streaming service reduces resolution during network congestion.
• Example: An e-commerce site disables product recommendations but keeps the shopping cart and checkout functional during a cache failure.

A distributed algorithm by which nodes in a cluster autonomously select a single node to act as the coordinator or leader. This is critical for stateful failover, ensuring that only one replica acts as the primary for a given shard or service at a time, maintaining data consistency.

• Used in: Databases (e.g., PostgreSQL with Patroni), distributed queues, coordination services (e.g., Apache ZooKeeper, etcd).
• Protocols: Raft and Paxos are common consensus algorithms used to implement robust leader election.

A design pattern that isolates elements of an application into independent pools, so if one fails, the others continue to function. Named after the watertight compartments on a ship, it prevents a single point of failure from cascading through the entire system. This pattern complements failover by containing the blast radius of a failure.

• Resource Pool Isolation: Using separate connection pools for different downstream services.
• Thread Pool Isolation: Dedicating specific threads to different tasks to prevent one slow task from starving all others.