Agent StatefulSet

An Agent DaemonSet is a Kubernetes workload controller that ensures a copy of a specific agent pod runs on all (or a subset of) nodes in a cluster. Unlike a StatefulSet, which manages a defined set of pods with unique identities, a DaemonSet is used for node-level agents that provide cluster-wide services.

• Primary Use Case: Deploying monitoring agents, log collectors, or network proxies that must run on every node.
• Key Characteristic: Pods are not numbered sequentially and are tightly bound to node lifecycle events.
• Contrast with StatefulSet: DaemonSets are for cluster infrastructure; StatefulSets are for stateful application workloads like databases or agents with persistent identity.

The Agent Operator Pattern is a method of packaging, deploying, and managing a complex stateful agent application using a custom Kubernetes controller. An Operator extends the Kubernetes API using Custom Resource Definitions (CRDs) to automate Day-2 operations.

• Automates Complex Tasks: Handles backups, updates, failure recovery, and configuration management specific to the agent.
• Manages StatefulSets: Often deploys and manages the underlying StatefulSet that runs the agent pods.
• Example: An operator for a database agent would handle scaling, failover, and point-in-time restores, abstracting complexity from the platform engineer.

Agent State Persistence is the mechanism by which an agent's volatile runtime state is saved to durable storage to survive pod restarts, failures, or migrations. This is a foundational requirement for any agent managed by a StatefulSet.

• Implementation: Typically uses PersistentVolumes (PVs) and PersistentVolumeClaims (PVCs) provisioned dynamically or statically.
• StatefulSet Guarantee: Each pod in a StatefulSet gets a unique, stable PVC that follows it during rescheduling.
• Types of State: Includes session data, learned model parameters, caches, and transaction logs.

An Agent Rolling Update is a deployment strategy where instances of an old agent version are incrementally replaced with a new version, one pod at a time (or in defined partitions). StatefulSets support controlled, ordered rolling updates to maintain application integrity.

• Ordered Pod Termination: Updates pods in reverse ordinal order (e.g., pod-2, then pod-1, then pod-0), waiting for each to be ready before proceeding.
• Zero-Downtime: Ensures the remaining pods continue to serve traffic during the update.
• Partitioned Updates: Allows canary-style updates by updating only pods with an ordinal greater than or equal to a specified partition.

A Pod Disruption Budget (PDB) is a Kubernetes policy that limits the number of pods from a StatefulSet (or other controller) that can be simultaneously unavailable during voluntary disruptions. It is critical for maintaining the availability of stateful agents during cluster maintenance.

• Voluntary Disruptions: Actions initiated by an administrator, such as node draining for an upgrade.
• Enforces High Availability: Guarantees a minimum number of available pods (.spec.minAvailable) or a maximum number of unavailable pods (.spec.maxUnavailable).
• Protects Quorum: For distributed agents, a PDB prevents too many replicas from being down at once, which could cause a loss of quorum and service outage.

An Agent Reconciliation Loop is the core control logic, often implemented within a custom controller or operator, that continuously observes the actual state of agent resources and takes action to drive it toward the declared desired state. It is the engine behind declarative configuration.

• Observe-Diff-Act Cycle: Constantly fetches the live state of pods, services, and PVCs, compares it to the spec defined in the StatefulSet YAML, and makes changes (create, update, delete) to align them.
• Self-Healing: If a pod crashes, the reconciliation loop detects the discrepancy and instructs the StatefulSet controller to create a replacement pod with the same identity.
• Manages Drift: Corrects configuration drift, ensuring the running system matches the version-controlled source of truth.