Leader Election

A core guarantee of leader election is liveness: the system must eventually elect a leader if a majority of nodes are alive and can communicate, even after failures. This requires mechanisms to detect node crashes (e.g., via heartbeat signals) and trigger a new election. The algorithm must ensure progress is made and avoid indefinite deadlock, guaranteeing the cluster is never left without a coordinator for an extended period.

The primary safety property is that at most one leader is elected for a given term or epoch. Having two active leaders (a split-brain scenario) leads to data corruption and inconsistent system state. Algorithms like Raft and Paxos use mechanisms such as quorums and unique, monotonically increasing election terms to mathematically guarantee uniqueness, even in the presence of network delays and message reordering.

A leader maintains its authority by periodically sending heartbeat messages (empty AppendEntries in Raft). If followers stop receiving heartbeats within a timeout, they assume the leader has failed and initiate a new election. This concept is often extended with a leader lease, a time-bound guarantee where followers agree not to challenge the leader for a period, improving stability in volatile networks.

Most production leader election algorithms are quorum-based. A node must receive votes from a majority (or more) of the cluster to become leader. This ensures:

• Uniqueness: Two nodes cannot simultaneously achieve a majority.
• Fault Tolerance: The system can tolerate f failures with 2f + 1 nodes.
• Progress: A majority ensures at least one node has seen the most recent committed state, preserving consistency.

Elections are triggered by specific events, not run continuously:

• Follower Election Timeout: A randomized timer expires without hearing from a leader.
• Leader Failure: Detected via missed heartbeats.
• Cluster Startup.
  Randomized timeouts are critical to prevent split votes where multiple followers simultaneously become candidates and split the vote, requiring another round.

Leader election is rarely an isolated function; it's tightly coupled with log replication (as in Raft's State Machine Replication). The elected leader becomes the authority for sequencing client commands into a replicated log. The election process often ensures the new leader possesses the most up-to-date log, preventing data loss. This makes it a cornerstone for building strongly consistent distributed data stores like etcd and Consul.

A heartbeat is a periodic signal sent from a leader node to followers to assert its liveness and maintain its authority. Failure to receive heartbeats triggers a new election.

• Function: Prevents split-brain scenarios by allowing followers to detect a leader failure.
• Implementation: Often a simple empty AppendEntries RPC in Raft, or a periodic ping in custom protocols.
• Configurable Parameter: The heartbeat interval and election timeout are critical tuning parameters for balancing responsiveness against network flapping.

Failover is the automatic process of transferring responsibilities from a failed primary node (leader) to a designated standby node, ensuring continuous service availability.

• Triggered By: Leader election is the core logic that orchestrates failover in software-defined systems.
• Types: Can be stateful (requiring state transfer) or stateless. Leader election often manages stateless failover for coordination roles.
• Objective: Minimizes Mean Time To Recovery (MTTR) and is a key component of High Availability (HA) architectures.

The Bulkhead pattern is a fault isolation design that partitions system resources (like thread pools, connections, or node groups) to prevent a failure in one partition from cascading and exhausting all resources.

• Analogy: Like watertight compartments in a ship.
• Relation to Leader Election: Can be applied by running independent leader election processes for different service clusters or data shards. The failure of a leader in one bulkhead does not affect the operation of others.
• Benefit: Increases overall system resilience and limits blast radius.

Let-it-crash is a fault-tolerance philosophy, central to the Erlang/OTP and Actor model, where processes are allowed to fail and are restarted by a supervisor, rather than attempting complex internal error recovery.

• Relation to Leader Election: Complements leader election by simplifying node design. A node participating in election can crash and be restarted; the cluster's election protocol handles the node's absence and reintegration.
• Supervision Trees: Failed leader nodes are treated as ephemeral components within a larger, resilient process hierarchy.
• Result: Creates systems that are self-healing at the process level, while leader election provides healing at the coordination level.