High Availability (HA)

The duplication of critical components to increase system reliability. This is the foundational principle behind HA, ensuring no single component's failure causes a total system outage.

• Active-Active: Multiple identical components (servers, databases, agents) process requests simultaneously, distributing load and providing immediate failover.
• Active-Passive (Hot/Warm Standby): A primary component handles all traffic while one or more replicas remain synchronized and on standby, ready to take over if the primary fails.
• N+1, 2N Redundancy: Common models where 'N' components handle the normal load, with '+1' or a full duplicate set ('2N') providing backup capacity.

The automatic process of switching to a redundant or standby system upon the failure of the currently active component. This mechanism is what makes redundancy operational.

• Automatic Detection: Relies on health checks (e.g., watchdog timers, /health endpoints) to detect component failure.
• State Transfer: For stateful services, failover requires transferring or replicating session state to the standby node. Stateless services simplify this process.
• DNS, Load Balancer, or Cluster Manager: The entity responsible for redirecting traffic from the failed component to the healthy one.

Distributed algorithms that ensure a group of nodes can agree on a single coordinator (leader) or a shared state, maintaining consistency during failures.

• Leader Election: Critical for systems requiring a single decision-maker (e.g., for managing a distributed lock or sequencing writes). If the leader fails, the cluster elects a new one.
• Consensus Protocols: Algorithms like Raft and Paxos enable a distributed system to agree on a value or sequence of commands, even if some nodes fail. They are the backbone of Crash Fault Tolerant (CFT) and Byzantine Fault Tolerant (BFT) systems.

Techniques to ensure data and application state persist and remain consistent across failures, preventing data loss and enabling seamless recovery.

• State Machine Replication: A method where multiple replicas of a deterministic service process the same sequence of commands in the same order, producing identical state transitions.
• Checkpointing: Periodically saving a complete, consistent snapshot of an application's state to stable storage, allowing a failed process to restart from the last checkpoint.
• Event Sourcing: Storing all changes to application state as a sequence of immutable events. The current state can be reconstructed by replaying events, providing a robust audit trail and recovery mechanism.

Designing systems to maintain core functionality and user experience when under extreme load or during partial failures, rather than collapsing entirely.

• Load Shedding: Deliberately dropping non-critical requests (e.g., dropping low-priority API calls) when the system is overloaded to preserve capacity for critical operations.
• Rate Limiting & Backpressure: Controlling the flow of incoming requests or data to prevent downstream components from being overwhelmed, which can cascade into system-wide failure.
• Fallback Strategies: Providing simplified, cached, or default responses when a primary dependency (e.g., an external API, a ML model inference endpoint) is unavailable or slow.

Operational practices and testing methodologies that proactively validate and maintain a system's HA characteristics in production-like environments.

• Blue-Green & Canary Deployments: Release strategies that allow for instant rollback (Blue-Green) or gradual, monitored exposure to new versions (Canary), minimizing deployment-related downtime.
• Chaos Engineering: The disciplined practice of intentionally injecting failures (latency, crashes, network partitions) into a system to test its resilience and uncover hidden flaws.
• Circuit Breaker Pattern: A design pattern that wraps calls to external services, detects failures, and opens the circuit to prevent cascading failures, allowing the system to fail fast and recover.

The automatic process of switching operations to a redundant or standby system component upon the failure or degradation of the primary component. This is the core mechanism that enables High Availability.

• Active-Passive: A standby system remains idle until a failure triggers a switch.
• Active-Active: Multiple systems handle load simultaneously; if one fails, traffic is redistributed.
• Stateful vs. Stateless: Failover complexity increases with systems that maintain session or transaction state, often requiring shared storage or state replication.

The strategic duplication of critical components or functions to increase system reliability. It is the foundational design principle behind HA architectures.

• N+1 Redundancy: A system has one extra component beyond what is needed for basic operation.
• 2N Redundancy: A fully mirrored, independent duplicate of the entire system.
• Geographic Redundancy: Systems are duplicated across different data centers or regions to survive site-level disasters.
• Redundancy applies to hardware (servers, power supplies), data (replication), and networks (multiple paths).

A design pattern that prevents a component from repeatedly attempting an operation that is likely to fail, thereby stopping cascading failures and allowing the system to degrade gracefully. It is a software implementation of an electrical circuit breaker.

• States: Closed (normal operation), Open (failing fast), Half-Open (probing for recovery).
• Critical for microservices: Prevents a single failing downstream service from exhausting the threads or resources of all upstream callers.
• Enables fallback strategies: When the circuit is 'open,' the system can execute a predefined fallback response.

A dedicated API endpoint (e.g., /health or /ready) that returns the operational status of a service. It is the primary signal used by orchestration systems to make HA decisions.

• Liveness Probe: Indicates if the service process is running. Failure triggers a restart.
• Readiness Probe: Indicates if the service is ready to accept traffic (e.g., database connections established). Failure triggers removal from a load balancer.
• Composite Checks: A robust health endpoint may check internal dependencies, disk space, and memory usage.

A key reliability metric that measures the average time required to repair a failed component or system and restore it to normal operation. In HA design, the goal is to minimize MTTR.

• Components: Includes detection time, diagnostic time, repair time, and verification time.
• Automated Recovery: Systems with low MTTR use automated failover, rollback, and restart procedures.
• Business Impact: Directly correlates with the duration of service disruption and associated costs. It is often balanced against Mean Time Between Failures (MTBF).