Failover

All nodes in the cluster are active and simultaneously process requests, often behind a load balancer. This pattern provides both high availability and horizontal scalability. Failover is seamless: if one node fails, the load balancer simply stops routing traffic to it, and the remaining nodes absorb the workload. It requires more sophisticated state synchronization (e.g., using a shared database, CRDTs, or event sourcing) to ensure all nodes have a consistent view of the world. The key challenge is designing idempotent operations and managing distributed consensus for state changes to prevent conflicts during concurrent processing.

A specialized form of active-passive replication common in data systems (e.g., databases, Raft-based services). One node is elected as the leader, responsible for processing all write operations and replicating changes to follower nodes. Followers can often serve read requests. The system uses a consensus protocol like Raft or Paxos to manage leader election and log replication. If the leader fails, the followers automatically hold an election to choose a new leader. This pattern provides strong consistency guarantees and is fundamental to state machine replication. It is more complex to implement than basic heartbeat monitoring.

This pattern extends failover across geographically dispersed data centers or cloud regions to protect against site-wide disasters. It typically involves active-passive setups where the passive site is in a different region. Data replication across regions introduces significant latency, often leading to asynchronous replication and eventual consistency models. The failover decision is often manual or requires sophisticated automated triggers to avoid split-brain syndrome caused by network partitions. Recovery Point Objectives (RPO) and Recovery Time Objectives (RTO) are much higher than in single-data-center failover.

Resilience is implemented at the service client or API gateway level. The client maintains a list of available service endpoints (discovered via a service registry). When a request fails, the client logic includes:

• Retry Policies: Immediate retry or exponential backoff.
• Failover: Switching to the next endpoint in the list.
• Circuit Breaker: Tripping a circuit breaker after repeated failures to prevent cascading failures.
• Dead Letter Queues: Routing persistently failing messages to a DLQ for analysis. This pattern decentralizes failover logic, making the system more robust but requiring consistent implementation across all clients. It is often used in conjunction with server-side patterns.

The failover strategy is dictated by whether the agent is stateless or stateful.

• Stateless Agent Failover: Simpler. Any healthy replica can immediately take over. Requires externalized session state (in a database or cache) and relies on load balancers with health checks. Patterns like rolling updates and blue-green deployments are inherently failover-capable.
• Stateful Agent Failover: Complex. The agent's internal memory or context must be preserved. This necessitates:
  • Checkpointing: Periodic snapshots of state to persistent storage.
  • State Synchronization: Real-time replication to peers (e.g., hot standby).
  • Recovery Orchestration: A process to restart the agent on new hardware and reload its last known good state. The choice profoundly impacts system architecture and the orchestrator's complexity.

A fundamental fault tolerance technique where a deterministic service is replicated across multiple machines. Each replica processes the same sequence of requests in the same order to produce identical state transitions and outputs.

• Core Principle: Ensures all replicas have the same state, making any of them a viable candidate for failover.
• Requirement: The service must be deterministic; given the same input log, all replicas produce the same output.
• Use Case: The backbone of databases like etcd and Consul, which use consensus protocols (Raft) to maintain the replicated log.

A catastrophic failure condition in high-availability clusters where a network partition causes independent sub-clusters to believe they are the sole active group, leading to data corruption and conflicts.

• Cause: A faulty network switch or misconfigured heartbeat timeout can isolate nodes.
• Consequence: Both partitions may activate their own 'primary' agents, processing conflicting writes (e.g., double-spending a resource).
• Prevention: Mitigated by using a quorum-based consensus algorithm (like Raft) or a reliable fencing mechanism (STONITH - Shoot The Other Node In The Head) to ensure only one partition can remain active.

A design philosophy where a system maintains partial, core functionality when some of its components fail, providing a reduced but acceptable level of service instead of a complete outage.

• Contrast to Failover: While failover aims for full continuity, graceful degradation accepts a loss of features to preserve core service.
• Multi-Agent Example: If a specialized 'data-analysis' agent fails, the system might degrade by returning raw data instead of analyzed insights, while the 'user-interface' and 'data-fetching' agents continue to operate.
• Strategy: Often implemented alongside failover for non-critical subsystems.