Kubernetes DaemonSet Advanced Concepts: Scheduling, Updates, Resource Management, and Troubleshooting with Practical Examples

Kubernetes DaemonSets ensure that a copy of a Pod runs on all or some cluster nodes. This guide goes beyond the basics to cover advanced DaemonSet internals, scheduling nuances, update strategies, resource management, and operational troubleshooting. We will walk through a realistic example of running a node monitoring agent as a DaemonSet, with concrete commands, configurations, and verification steps.

Intro

DaemonSets are a core workload controller in Kubernetes. While many teams use them for log collection, monitoring, or storage plugins, few fully exploit their advanced capabilities. Understanding DaemonSet internals helps operators avoid common pitfalls such as Pods running on unintended nodes, update storms, or resource contention. This guide provides a deep dive into advanced DaemonSet concepts, including node affinity, rollout strategies, resource management, and diagnostics. By the end, you will be able to design DaemonSets that are robust, efficient, and easy to operate in production.

We will use a practical scenario: deploying a node monitoring agent (like a Prometheus node exporter) as a DaemonSet. The principles apply to any node-level service.

Version and Environment Inventory

Before implementing advanced DaemonSet features, verify your cluster version and environment. Some features, such as rollingUpdate with maxSurge and maxUnavailable, require Kubernetes 1.12+. Others, like podAffinity or topologySpreadConstraints, have evolved over releases. Check your API server version:

kubectl version --short

Expected output (example):

Client Version: v1.24.3
Server Version: v1.25.2

Ensure you have kubectl configured with cluster-admin or appropriate RBAC permissions for DaemonSets. Test connectivity:

kubectl get nodes

Example output:

NAME     STATUS   ROLES                  AGE   VERSION
node-1   Ready    control-plane,master   10d   v1.25.2
node-2   Ready    <none>                 10d   v1.25.2
node-3   Ready    <none>                 10d   v1.25.2

If your cluster has mixed node roles or taints, identify them because DaemonSets must tolerate taints to schedule on certain nodes. For example, control-plane nodes often have a node-role.kubernetes.io/control-plane taint. You can list taints:

kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

Example output:

NAME     TAINTS
node-1   [map[effect: NoSchedule key: node-role.kubernetes.io/control-plane]]
node-2   <none>
node-3   <none>

Finally, note the container runtime and any node-level resources you need to access. For our monitoring agent, we will mount host filesystem paths, so the Pod security settings must allow it.

Safe Configuration Path

Advanced DaemonSet configuration involves several key areas: node scheduling, update strategy, resource management, and security. We will implement each step incrementally, verifying along the way.

Step 1: Basic DaemonSet with Node Selector and Tolerations

Start with a minimal DaemonSet that targets all Linux nodes. We'll use a node selector based on kubernetes.io/os: linux and add tolerations for common taints to ensure the agent runs everywhere. Save the following manifest as daemonset-basic.yaml:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-agent
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: node-agent
  template:
    metadata:
      labels:
        app: node-agent
    spec:
      nodeSelector:
        kubernetes.io/os: linux
      tolerations:
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule
        - key: node-role.kubernetes.io/master
          operator: Exists
          effect: NoSchedule
      containers:
        - name: agent
          image: quay.io/prometheus/node-exporter:v1.3.1
          ports:
            - containerPort: 9100
              name: metrics
          resources:
            requests:
              cpu: 100m
              memory: 64Mi
            limits:
              cpu: 200m
              memory: 128Mi
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            runAsNonRoot: true
            runAsUser: 65534

Apply and verify:

kubectl create namespace monitoring
kubectl apply -f daemonset-basic.yaml
kubectl get daemonset node-agent -n monitoring

Expected output:

NAME          DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
node-agent    3         3         3       3            3           <none>          10s

Step 2: Advanced Scheduling with Node Affinity

Node selectors are simple but limited. Use nodeAffinity for more expressive rules, such as scheduling on nodes with specific labels or excluding others. For example, suppose we want to avoid nodes in a preemptible pool. First label nodes accordingly (if not already):

kubectl label node node-2 node-pool=regular
kubectl label node node-3 node-pool=regular
kubectl label node node-1 node-pool=control-plane

Then update the DaemonSet spec to include affinity.nodeAffinity:

spec:
  template:
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: node-pool
                    operator: In
                    values: [regular, control-plane]
      nodeSelector: {}
      tolerations: ...  # same as before

Apply the change:

kubectl apply -f daemonset-basic.yaml

Check the rollout status:

kubectl rollout status daemonset node-agent -n monitoring

Expected output:

daemon set "node-agent" successfully rolled out

Step 3: Rolling Update Strategy

By default, DaemonSets use RollingUpdate with maxUnavailable: 1 and maxSurge: 0. This means one node at a time is updated, ensuring availability. For large clusters, you may want faster updates. Configure the update strategy explicitly. In our DaemonSet, add:

spec:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 0

This is the default, but being explicit documents intent. If your DaemonSet is not critical and you have spare capacity, you could set maxUnavailable: 2 or maxSurge: 1 (requires K8s 1.22+ for surge). For now, keep one at a time. Apply and watch the rollout:

kubectl apply -f daemonset-basic.yaml
kubectl get pods -n monitoring -o wide --watch

Observe the controlled replacement: old Pods are terminated after new Pods become ready.

Step 4: Resource Management and Priority

DaemonSet Pods can be resource hogs. Set requests and limits carefully. In our example, requests are modest. Also consider setting a priorityClassName to ensure DaemonSet Pods are scheduled even under resource pressure. Create a PriorityClass if not present:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: node-agent-priority
value: 1000000
globalDefault: false
description: "Priority for node agents"

Then add priorityClassName: node-agent-priority to the Pod template. Apply both.

Step 5: Pod Security and Host Access

For accessing host metrics, our container may need to read /proc and /sys. The node-exporter typically runs as root or with capabilities. We used a non-root user, which may limit functionality. If you need hostPath mounts, add them and adjust securityContext. For demonstration, we keep it simple, but be aware of Pod Security Standards. In clusters with Pod Security Admission, you may need to set namespace labels to pod-security.kubernetes.io/enforce=privileged or use a custom policy.

Verification and Diagnostics

After deployment, verify that DaemonSet Pods are running correctly on each expected node.

• Check DaemonSet status:

kubectl get ds node-agent -n monitoring -o wide

Expected output shows DESIRED, CURRENT, READY, etc., and NODE SELECTOR/AFFINITY if set.

• Ensure every node has a Pod:

kubectl get pods -n monitoring -o wide --selector=app=node-agent

Example:

NAME              READY   STATUS    RESTARTS   AGE    IP          NODE
node-agent-abcde   1/1     Running   0          2m     10.0.1.10   node-1
node-agent-fghij   1/1     Running   0          2m     10.0.2.10   node-2
node-agent-klmno   1/1     Running   0          2m     10.0.3.10   node-3

Compare with node list: every Ready node should have a Pod (unless excluded by scheduling).

• Check logs for the agent:

kubectl logs daemonset/node-agent -n monitoring --tail=10

This returns logs from a random Pod. To check a specific Pod:

kubectl logs node-agent-abcde -n monitoring

• Verify metrics endpoint:

Port-forward to a Pod:

kubectl port-forward pod/node-agent-abcde 9100:9100 -n monitoring

Then in another terminal:

curl localhost:9100/metrics | head

Expected output includes lines like:

# HELP node_cpu_seconds_total Seconds the CPUs spent in each mode.
# TYPE node_cpu_seconds_total counter
node_cpu_seconds_total{cpu="0",mode="idle"} 12345.67

• Describe the DaemonSet to check events:

kubectl describe ds node-agent -n monitoring

Look for events indicating scheduling failures or other issues.

Failure Modes and Recovery

DaemonSets can fail in several ways. Common issues include:

• Scheduling failures: A node may not receive a Pod if it has taints not tolerated, or if nodeSelector/affinity rules exclude it. Check DaemonSet status: DESIRED > CURRENT indicates missing Pods.
• Image pull errors: If the image is wrong or registry inaccessible, Pods will be in ImagePullBackOff. Check kubectl describe pod for details.
• CrashLoopBackOff: The container starts but exits repeatedly. Inspect logs and adjust command or resources.
• Update stuck: A rolling update may stall if a new Pod never becomes Ready. This can happen due to readiness probe failure or resource constraints. Check kubectl rollout status and pod statuses.
• Node cordon/drain: If a node is cordoned, DaemonSet Pods may still be scheduled (unless node.kubernetes.io/unschedulable taint is added). Understand node lifecycle.

Rollback

To roll back a DaemonSet update, revert to the previous manifest or use kubectl rollout undo:

kubectl rollout undo daemonset node-agent -n monitoring

This works if you have rollout history enabled (DaemonSets support history since 1.15). Check rollout history:

kubectl rollout history daemonset node-agent -n monitoring

Then undo to a specific revision if needed:

kubectl rollout undo daemonset node-agent -n monitoring --to-revision=2

Forcing Deletion

If a DaemonSet is stuck, you may need to delete it and reapply. Use --cascade=orphan to keep Pods running temporarily, then delete Pods manually:

kubectl delete ds node-agent -n monitoring --cascade=orphan
kubectl delete pods -n monitoring -l app=node-agent

Then reapply the corrected manifest.

Recovery Verification

After rollback, verify the DaemonSet is healthy again:

kubectl rollout status daemonset node-agent -n monitoring
kubectl get ds node-agent -n monitoring

Expected output shows READY equals DESIRED.

Operations Checklist

Use this checklist for regular DaemonSet operations and reviews.

Perform this checklist after any cluster change, upgrade, or incident.

Common Pitfalls and How to Avoid Them

DaemonSet management often trips up teams in predictable ways. Here are the most frequent mistakes and how to steer clear of them.

Ignoring Taints on New Nodes

What goes wrong: A new node is joined with a taint that the DaemonSet does not tolerate. The DaemonSet never schedules a Pod there, leading to unmonitored infrastructure. This typically happens when teams add autoscaling groups or replace nodes without reviewing taints.

How to avoid it: Include broad tolerations for common node-role and hardware taints in your DaemonSet spec. Regularly audit nodes and taints with kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints. Set up an alert if DESIRED exceeds CURRENT in kubectl get ds.

Over-Aggressive Update Strategies

What goes wrong: Setting maxUnavailable: 100% or maxSurge: 100% to speed up updates can cause all Pods to be replaced simultaneously, leading to a brief outage of the node-level service. For critical agents like log shippers or CNI plugins, this disrupts observability or networking.

How to avoid it: Use the default maxUnavailable: 1 for critical DaemonSets. If you need faster updates, test with a canary node first (see checklist) and monitor rollout status.

Neglecting Resource Requests and Limits

What goes wrong: Without resource requests, the scheduler may place DaemonSet Pods on nodes already under pressure. Without limits, a runaway agent can consume node CPU or memory, affecting other workloads.

How to avoid it: Always set resources.requests and limits appropriate for the agent. Use kubectl top pods regularly to verify actual usage. Consider priorityClassName to guarantee scheduling.

Forgetting to Update Pod Security Policies

What goes wrong: In clusters with Pod Security Admission (PSA) or older PodSecurityPolicy (PSP), DaemonSet Pods that require privileged access (e.g., mounting host /proc) fail to start when namespace policies are enforced. This shows up as CreateContainerConfigError or admission denial.

How to avoid it: Review the required capabilities for your agent. Use the least privilege possible, but if host access is needed, label the namespace with the appropriate PSA level (e.g., pod-security.kubernetes.io/enforce=privileged for the monitoring namespace). Document exceptions clearly.

Overlooking Node Cordon or Drain Effects

What goes wrong: When a node is cordoned, many assume DaemonSet Pods won't be scheduled there, but DaemonSets bypass the unschedulable flag unless the node has the node.kubernetes.io/unschedulable taint. Conversely, during a drain, DaemonSet Pods are often ignored by kubectl drain unless --ignore-daemonsets is set, causing confusion about remaining Pods.

How to avoid it: Understand cordon vs. taint semantics. Use kubectl drain --ignore-daemonsets when evicting regular workloads, and manually delete DaemonSet Pods if necessary. Document node maintenance procedures in runbooks.

Advanced Scheduling with Topology Spread Constraints

Beyond node affinity, consider topologySpreadConstraints to evenly distribute DaemonSet Pods across failure domains like zones. While a DaemonSet already places one Pod per node, spread constraints become useful when you have multiple DaemonSets that should not co-locate, or when nodes are grouped into logical zones. For example, to spread our monitoring agent across availability zones:

spec:
  template:
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: node-agent

This ensures that if zones have uneven node counts, the scheduler tries to keep the number of Pods in each zone within one of each other. Apply and verify with kubectl get pods -n monitoring -o wide --sort-by=.spec.nodeName.

Monitoring and Alerting for DaemonSets

To keep DaemonSets healthy, integrate them into your monitoring stack. Here are key metrics to track (using Prometheus as an example):

• kube_daemonset_status_desired_number_scheduled vs kube_daemonset_status_current_number_scheduled: Alert if desired > current for more than 5 minutes.
• kube_daemonset_status_number_ready: Should equal desired.
• kube_daemonset_status_updated_number_scheduled: During updates, this temporarily differs; monitor for stuck rollouts.
• Pod restart rate: rate(kube_pod_container_status_restarts_total{namespace="monitoring"}[5m]) > 0 indicates instability.

Set up alerts with these expressions and route them to your on-call rotation. Review alert thresholds quarterly to avoid noise.

Security Hardening

Security for DaemonSets goes beyond Pod Security Admission. Consider these practices:

• Run as non-root: Always set runAsNonRoot: true unless absolutely required. Our node-exporter example uses UID 65534 (nobody), which works for basic metrics but may need root for certain collectors.
• Use read-only root filesystem: Set readOnlyRootFilesystem: true to prevent tampering.
• Drop capabilities: Use capabilities.drop: ["ALL"] and add only those needed. For node-exporter, you may need CAP_SYS_TIME for some collectors; test carefully.
• Limit hostPath usage: Prefer hostPath with type specified (e.g., Directory or File) and avoid mounting sensitive paths like /etc/kubernetes unless necessary.
• Network policies: Apply NetworkPolicies to the namespace to restrict traffic to and from DaemonSet Pods.

For example, a hardened security context for our agent:

securityContext:
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  runAsNonRoot: true
  runAsUser: 65534
  capabilities:
    drop: ["ALL"]
    add: ["CAP_SYS_TIME"]

Test thoroughly in a staging cluster before applying to production.

Conclusion

Advanced DaemonSet concepts enable precise control over node-level workloads in Kubernetes. By using node affinity, tolerations, explicit update strategies, and resource management, you can ensure reliable and efficient operation. This guide demonstrated a practical implementation with a monitoring agent, covering configuration, verification, failure recovery, and an operational checklist. Use these patterns as a foundation for your own DaemonSets, adapting to your environment and requirements.

For further reading, consult the official Kubernetes DaemonSet documentation and the API reference for details on all fields.

Additional Resources

• Kubernetes DaemonSet Documentation
• Kubernetes API Reference: DaemonSetSpec
• Pod Priority and Preemption
• Pod Topology Spread Constraints
• Pod Security Admission