DaemonSet

The core function of a DaemonSet is to guarantee that a Pod template is scheduled and runs on every eligible node in the cluster. When a new node joins the cluster, the DaemonSet controller automatically deploys the Pod to it. Conversely, when a node is removed, its Pod is garbage collected. This ensures a uniform, node-level presence for services like:

• Log collectors (e.g., Fluentd, Filebeat)
• Monitoring agents (e.g., Prometheus Node Exporter, Datadog Agent)
• Network plugins (e.g., Calico, Cilium)
• Storage daemons (e.g., Gluster, Ceph)

DaemonSets can target a subset of nodes using nodeSelectors and affinity/anti-affinity rules. This is critical for heterogeneous clusters where a service should only run on nodes with specific hardware (e.g., GPUs), roles (e.g., node-role.kubernetes.io/ingress), or labels (e.g., disk=ssd). For example, a GPU monitoring agent would use a node selector like accelerator: nvidia-tesla-v100 to deploy only to GPU-equipped nodes.

Unlike a Deployment, which manages a Pod lifecycle via an intermediate ReplicaSet to maintain a desired count of identical Pods, a DaemonSet directly manages Pods based on node inventory. There is no replicas field. The desired state is defined by the node selector and the cluster's node count. Pods are named with the template hash and the node name (e.g., fluentd-abc12-node-01), providing a clear 1:1 mapping to their host.

In Agentic Observability, DaemonSets are the standard method for deploying the data collection layer of a telemetry pipeline. A DaemonSet ensures every node runs an instance of a telemetry agent (e.g., OpenTelemetry Collector, Grafana Agent, Vector) that is responsible for:

• Receiving traces/spans from instrumented applications on the local node.
• Scraping node-level metrics (CPU, memory, disk).
• Collecting container logs from the node's container runtime.
• Enriching data with node-level attributes (e.g., AZ, instance type).
• Forwarding data reliably to a central OTel Collector or backend. This provides a scalable, resilient foundation for capturing signals from autonomous agents and their infrastructure.

DaemonSets support two update strategies for rolling out new Pod templates:

• RollingUpdate (Default): Updates Pods on nodes in a controlled, rolling fashion. You can configure maxUnavailable (e.g., 1) to limit how many nodes can be down during the update.
• OnDelete: The DaemonSet only creates new Pods with the updated template when you manually delete the old Pods on each node. This provides maximum control for stateful or sensitive node-level services. Updates are triggered by changes to the Pod template spec (spec.template).

DaemonSets often require tolerations to deploy Pods onto tainted nodes. Master/control plane nodes commonly have a taint like node-role.kubernetes.io/master:NoSchedule. To run a cluster-wide monitoring agent on these nodes, the DaemonSet's Pod spec must include a matching toleration. This is a key differentiator from standard Deployments and is essential for full cluster coverage. The DaemonSet controller itself does not add tolerations automatically; they must be explicitly defined in the Pod template.

A Deployment is a Kubernetes controller that manages the declarative updates and lifecycle of a set of identical pods, known as ReplicaSets. It is the primary workload for stateless applications.

• Key Purpose: Ensures a specified number of pod replicas are running and facilitates rolling updates and rollbacks.
• Contrast with DaemonSet: Deployments schedule pods based on resource availability, not node affinity. They are used for scalable application services, not node-level agents.
• Example: Running 10 replicas of a web API service across a cluster's worker nodes.

A StatefulSet is a Kubernetes workload controller for managing stateful applications, providing guarantees about the ordering and uniqueness of pods.

• Key Purpose: Manages pods with persistent storage, stable network identities, and ordered deployment/scaling.
• Contrast with DaemonSet: StatefulSets are for clustered databases (e.g., Cassandra, Kafka) where pod identity and storage matter. DaemonSets ignore pod identity, focusing on node coverage.
• Stable Identity: Each pod gets a persistent hostname based on its ordinal index (e.g., kafka-0, kafka-1).

A Job creates one or more pods to execute a finite task to completion. A CronJob manages time-scheduled Jobs.

• Key Purpose: For batch processing, one-off computations, or scheduled maintenance tasks.
• Contrast with DaemonSet: Jobs are ephemeral and terminate; DaemonSets are long-running. A CronJob for log rotation might trigger a Job, while the log collector itself runs via a DaemonSet.
• Example: A CronJob that runs a data backup Job every night at 2 AM.

These are mechanisms to control pod scheduling.

• Node Selector: A simple field in a pod spec that schedules pods only onto nodes with matching labels.
• Taints & Tolerations: A taint allows a node to repel pods. A pod must have a matching toleration to be scheduled on a tainted node.
• DaemonSet Usage: DaemonSets use tolerations to deploy pods on master nodes (which are typically tainted) and node selectors to target subsets of nodes (e.g., node-type: gpu).

An Operator is a method of packaging, deploying, and managing a Kubernetes application using custom resources and controllers.

• Key Purpose: Encodes human operational knowledge (like backups, upgrades) into software for complex stateful applications.
• Relation to DaemonSet: An Operator for a monitoring stack (e.g., Prometheus Operator) would likely manage the DaemonSet that deploys the node exporters. The Operator handles the DaemonSet's lifecycle, configuration, and updates.