Agent DaemonSet

DaemonSet pods have a stable, predictable naming convention (<daemonset-name>-<random-suffix>) and are treated as immutable, cattle-like infrastructure. Updates are managed via a rolling update strategy (type: RollingUpdate), which is the default. The controller replaces pods on each node in a controlled sequence, respecting a configurable maxUnavailable count (e.g., 1) to maintain agent availability during upgrades. This is distinct from a recreate strategy, which would terminate all pods simultaneously, causing a service outage.

Agents deployed via DaemonSet are infrastructure-level software that consume node resources. It is critical to define resource requests and limits for these pods to prevent them from starving user applications. DaemonSet pods typically run with a high priority class to ensure they are scheduled before best-effort workloads. However, they must be configured carefully, as they are not automatically evicted under resource pressure if they have guaranteed QoS, which could impact node stability. Proper sizing ensures the agent's footprint is accounted for in cluster capacity planning.

DaemonSets are the standard method for deploying the data plane components of a service mesh (e.g., Istio's istio-proxy sidecar injector runs as a DaemonSet in some configurations) and observability collectors. For example:

• A OpenTelemetry Collector DaemonSet can be deployed on each node to receive traces and metrics from applications and forward them to a backend.
• This pattern provides a uniform telemetry layer with low latency, as the agent is local to the node, avoiding network hops for critical observability data.

It is essential to distinguish DaemonSets from other Kubernetes workloads:

• vs. Deployment: A Deployment manages a scalable set of identical pods with no inherent node affinity. A DaemonSet ensures one pod per node, scaling with the cluster's node count, not application demand.
• vs. StatefulSet: A StatefulSet is for stateful applications requiring stable network identities, ordered deployment/scaling, and persistent storage. A DaemonSet is for stateless, node-aware agents. While both provide stable pod names, DaemonSet pods do not have ordinal indexes or attached PersistentVolumeClaims by default. Choosing the wrong controller leads to incorrect scaling behavior and operational complexity.

An orchestration capability where the system automatically detects agent failures and takes corrective action. For an Agent DaemonSet, this is a core function managed by the Kubernetes control plane.

The self-healing loop involves:

1. Liveness Probes: Periodic checks (HTTP, TCP, or command execution) to determine if the agent process is running.
2. Failure Detection: The kubelet on the node identifies a failed probe.
3. Automatic Remediation: The kubelet restarts the agent container within the existing Pod.
4. Rescheduling: If the entire node fails, the DaemonSet controller ensures a new pod is scheduled on a replacement node.

This ensures a copy of the agent is always running on every targeted node.

A policy constraint that limits the aggregate amount of compute resources or object counts that a collection of agents within a Kubernetes namespace can consume. This is critical for DaemonSets, as they create pods on every node, which can lead to uncontrolled cluster-wide resource consumption.

Quotas can limit:

• Compute resources: Total CPU and memory requests/limits for all DaemonSet pods in a namespace.
• Object counts: The number of Pods, PersistentVolumeClaims, or Services.

Quotas enforce multi-tenancy and prevent a single DaemonSet from monopolizing cluster resources, ensuring fair sharing among all workloads.

Defines privilege and access control settings for an agent pod or container. For a DaemonSet that often requires deep node access (e.g., to read /dev devices, host network, or system logs), configuring the security context is a critical security task.

Settings include:

• RunAsUser / RunAsGroup: The Linux user and group ID for the container process.
• Privileged mode: Whether the container runs with elevated kernel privileges (often required for node agents).
• Capabilities: Adding or dropping specific Linux capabilities (e.g., NET_ADMIN, SYSLOG) as a more granular alternative to privileged: true.
• SELinux / AppArmor: Applying mandatory access control profiles.
• ReadOnlyRootFilesystem: Mounting the root filesystem as read-only for increased security.