etcd

etcd uses the Raft consensus algorithm to guarantee strong consistency across all nodes in the cluster. All write operations must be committed by a quorum (majority) of nodes before being acknowledged to the client, ensuring linearizable reads. This prevents split-brain scenarios and guarantees that a client reading from any node sees the most recent write.

• Leader Election: A single leader handles all client requests, simplifying the replication log.
• Log Replication: The leader replicates each state change to follower nodes.
• Fault Tolerance: The cluster can tolerate the failure of (N-1)/2 nodes, where N is the cluster size.

At its core, etcd is a key-value store where keys are hierarchically organized, similar to a filesystem. This simple data model is exposed via a gRPC API and a JSON-over-HTTP API. It supports standard operations like PUT, GET, DELETE, and WATCH.

• Hierarchical Keys: Keys are organized in a directory-like structure (e.g., /registry/pods/default/my-pod).
• Range Queries: Efficiently retrieve all keys under a prefix.
• Leases & TTLs: Keys can be associated with time-bound leases, automatically expiring for service discovery and leader liveness.

The watch API allows clients to subscribe to changes on specific keys or key prefixes. This is the foundation for the control loops in systems like Kubernetes. When a key's value is updated, all watching clients are notified with the new data, enabling reactive systems without inefficient polling.

• Event Streams: Clients receive a continuous stream of create, update, and delete events.
• Historical Events: Watches can be started from a specific revision in the past.
• Compact Notification: Efficiently handles scenarios where the event history is compacted.

etcd implements MVCC, storing every change to a key as a new revision. This creates a complete, immutable history of the datastore. Reads are performed at a specific revision, providing consistent snapshots and enabling features like watch from a past revision.

• Revision Numbers: Each transaction increments a global, monotonically increasing revision counter.
• Snapshot Isolation: Reads are never blocked by writes.
• History Compaction: Old revisions can be periodically compacted to reclaim disk space, while preserving a configurable window of history.

etcd provides robust security features for production clusters. Communication can be encrypted with Transport Layer Security (TLS), and access to keys is controlled via role-based access control (RBAC).

• Peer & Client TLS: Encrypts traffic between etcd members and between clients and the cluster.
• Client Certificate Authentication: Clients authenticate using X.509 certificates.
• Fine-Grained RBAC: Permissions (read, write, readwrite) can be assigned to users and roles for specific key ranges.

Designed for high availability, etcd clusters typically run with an odd number of nodes (3, 5, 7) to maintain a quorum during failures. Data durability is ensured by writing all committed entries to a persistent write-ahead log (WAL) on disk before applying them to the in-memory state machine. Snapshots of the state are also periodically written to disk.

• Quorum-Based Operations: Requires a majority of nodes to be healthy for writes.
• Persistent Storage: The WAL and snapshots ensure data survives process restarts.
• Member Management: Nodes can be added, removed, or replaced dynamically with minimal downtime.

In Edge AI Orchestration, etcd is used as the source of truth for the desired state of a distributed fleet of edge devices. An orchestration plane (like a lightweight Kubernetes distribution, e.g., K3s) uses etcd to store:

• Device inventory and status (online/offline, resource utilization).
• Model deployment manifests specifying which AI models should run on which edge nodes.
• Configuration profiles for sensors and local inference pipelines.
• Rollout policies for staged model updates across thousands of geographically dispersed devices. Its ability to operate in high-latency, occasionally connected environments (with proper tuning) makes it suitable for edge deployments.

A consensus protocol is a fault-tolerant mechanism that enables a group of distributed processes to agree on a single data value or system state, even when some participants fail. It is the core theoretical foundation for systems like etcd, ensuring linearizability and safety (no two correct processes decide differently) across the cluster.

• Purpose: Guarantees that all nodes in a distributed system see the same sequence of state changes.
• Key Property: Fault Tolerance, typically tolerating up to (N-1)/2 failures in a cluster of N nodes.
• Contrast with Locks: Provides agreement on state, not just mutual exclusion.

Raft is a consensus algorithm designed for understandability, which etcd uses to manage its replicated log and leader election. It decomposes consensus into three sub-problems: leader election, log replication, and safety. Raft ensures that all commands are replicated to a majority of nodes before being committed, making the system CP (Consistent and Partition-tolerant) in the CAP theorem.

• Leader-Based: A single elected leader handles all client requests for log entries.
• Log Commitment: An entry is committed once a majority of nodes have replicated it.
• Election Timeouts: Use randomized timers to prevent split votes and ensure liveness.

Leader election is a distributed systems pattern where multiple instances of a component coordinate to elect a single leader responsible for certain tasks, ensuring high availability. In etcd, the Raft protocol handles this: the leader serves all client write requests and manages log replication to followers. If the leader fails, followers initiate a new election.

• Prevents Split-Brain: Ensures only one node can coordinate writes at a time.
• Leases & Heartbeats: Leaders maintain authority by sending periodic heartbeats; failure to do so triggers a new election.
• Application Pattern: Used by Kubernetes controllers (e.g., kube-controller-manager) for HA deployments.

The control plane is the set of system components responsible for making global decisions about a cluster (e.g., scheduling, responding to events). In Kubernetes, etcd serves as the persistent backing store for the entire control plane, holding all cluster state like nodes, pods, and secrets. The API server is the only component that talks to etcd directly, acting as its gatekeeper.

• State Storage: etcd is the source of truth for the Kubernetes control plane.
• Decouples Logic from Storage: Controllers (data plane) watch for changes in etcd via the API server.
• High Availability: A clustered etcd is critical for a highly available Kubernetes control plane.

Declarative configuration is a paradigm where a user specifies the desired end state of a system, and the orchestration platform is responsible for continuously reconciling the actual state to match that specification. etcd enables this by storing the declared state (e.g., a Kubernetes Deployment manifest). Controllers watch for changes in etcd and drive the real world to match.

• Single Source of Truth: The desired state is authoritatively stored in etcd.
• Idempotency: The same declarative spec applied repeatedly results in the same final state.
• Contrast with Imperative: Focuses on what, not the step-by-step how of achieving it.

State reconciliation is the continuous control loop process by which an orchestration system observes the actual state of resources and takes corrective actions to drive them toward the declared desired state stored in etcd. This is the core mechanic of Kubernetes controllers.

• Watch & Diff: Controllers watch for changes to objects in etcd via the API server, compare actual state (from node reports) with desired state, and issue commands.
• Eventual Consistency: The system continuously works to minimize the difference between observed and desired states.
• Driven by etcd: The reconciliation loop is triggered by updates to the key-value store, making etcd's consistency guarantees fundamental to system stability.