Agent DaemonSet

The core function of a DaemonSet is to saturate nodes with a pod copy. When a new node joins the cluster, the DaemonSet controller automatically schedules the agent pod onto it. Conversely, when a node is removed, its pod is garbage collected. This ensures uniform agent coverage across the entire cluster infrastructure, which is critical for functions like:

• Host-level monitoring (e.g., Prometheus Node Exporter)
• Log collection (e.g., Fluentd, Filebeat)
• Security and compliance scanning
• Network plugin agents (e.g., Calico, Cilium)

DaemonSets use nodeSelectors and tolerations to control placement. A nodeSelector constrains pods to nodes with specific labels (e.g., disk: ssd). Tolerations allow pods to be scheduled onto nodes with matching taints, which are often applied to dedicated or special-purpose nodes like GPU workers or master/control-plane nodes. This enables precise targeting, such as:

• Running a GPU monitoring agent only on nodes with accelerator: nvidia.
• Deploying a networking agent on all nodes, including the control plane, by tolerating the node-role.kubernetes.io/control-plane:NoSchedule taint.

DaemonSet pods have a stable, predictable naming convention (<daemonset-name>-<random-suffix>) and are treated as immutable, cattle-like infrastructure. Updates are managed via a rolling update strategy (type: RollingUpdate), which is the default. The controller replaces pods on each node in a controlled sequence, respecting a configurable maxUnavailable count (e.g., 1) to maintain agent availability during upgrades. This is distinct from a recreate strategy, which would terminate all pods simultaneously, causing a service outage.

Agents deployed via DaemonSet are infrastructure-level software that consume node resources. It is critical to define resource requests and limits for these pods to prevent them from starving user applications. DaemonSet pods typically run with a high priority class to ensure they are scheduled before best-effort workloads. However, they must be configured carefully, as they are not automatically evicted under resource pressure if they have guaranteed QoS, which could impact node stability. Proper sizing ensures the agent's footprint is accounted for in cluster capacity planning.

DaemonSets are the standard method for deploying the data plane components of a service mesh (e.g., Istio's istio-proxy sidecar injector runs as a DaemonSet in some configurations) and observability collectors. For example:

• A OpenTelemetry Collector DaemonSet can be deployed on each node to receive traces and metrics from applications and forward them to a backend.
• This pattern provides a uniform telemetry layer with low latency, as the agent is local to the node, avoiding network hops for critical observability data.

It is essential to distinguish DaemonSets from other Kubernetes workloads:

• vs. Deployment: A Deployment manages a scalable set of identical pods with no inherent node affinity. A DaemonSet ensures one pod per node, scaling with the cluster's node count, not application demand.
• vs. StatefulSet: A StatefulSet is for stateful applications requiring stable network identities, ordered deployment/scaling, and persistent storage. A DaemonSet is for stateless, node-aware agents. While both provide stable pod names, DaemonSet pods do not have ordinal indexes or attached PersistentVolumeClaims by default. Choosing the wrong controller leads to incorrect scaling behavior and operational complexity.

A deployment model where a helper container (the sidecar) runs alongside the primary agent container in the same Kubernetes Pod. This pattern is often used in conjunction with a DaemonSet to augment node-level agents with auxiliary services without modifying the agent's core code.

Common sidecar functions include:

• Log collection and forwarding (e.g., Fluent Bit)
• Metrics scraping for Prometheus
• Network proxying or service mesh data plane (e.g., Envoy)
• Security auditing or secret injection

The sidecar shares the Pod's network namespace and storage volumes, enabling tight integration with the main agent.

An orchestration capability where the system automatically detects agent failures and takes corrective action. For an Agent DaemonSet, this is a core function managed by the Kubernetes control plane.

The self-healing loop involves:

1. Liveness Probes: Periodic checks (HTTP, TCP, or command execution) to determine if the agent process is running.
2. Failure Detection: The kubelet on the node identifies a failed probe.
3. Automatic Remediation: The kubelet restarts the agent container within the existing Pod.
4. Rescheduling: If the entire node fails, the DaemonSet controller ensures a new pod is scheduled on a replacement node.

This ensures a copy of the agent is always running on every targeted node.

A policy constraint that limits the aggregate amount of compute resources or object counts that a collection of agents within a Kubernetes namespace can consume. This is critical for DaemonSets, as they create pods on every node, which can lead to uncontrolled cluster-wide resource consumption.

Quotas can limit:

• Compute resources: Total CPU and memory requests/limits for all DaemonSet pods in a namespace.
• Object counts: The number of Pods, PersistentVolumeClaims, or Services.

Quotas enforce multi-tenancy and prevent a single DaemonSet from monopolizing cluster resources, ensuring fair sharing among all workloads.

Defines privilege and access control settings for an agent pod or container. For a DaemonSet that often requires deep node access (e.g., to read /dev devices, host network, or system logs), configuring the security context is a critical security task.

Settings include:

• RunAsUser / RunAsGroup: The Linux user and group ID for the container process.
• Privileged mode: Whether the container runs with elevated kernel privileges (often required for node agents).
• Capabilities: Adding or dropping specific Linux capabilities (e.g., NET_ADMIN, SYSLOG) as a more granular alternative to privileged: true.
• SELinux / AppArmor: Applying mandatory access control profiles.
• ReadOnlyRootFilesystem: Mounting the root filesystem as read-only for increased security.