Kubernetes DaemonSets ensure that a copy of a Pod runs on all or some cluster nodes. This guide goes beyond the basics to cover advanced DaemonSet internals, scheduling nuances, update strategies, resource management, and operational troubleshooting. We will walk through a realistic example of running a node monitoring agent as a DaemonSet, with concrete commands, configurations, and verification steps.

## Intro

DaemonSets are a core workload controller in Kubernetes. While many teams use them for log collection, monitoring, or storage plugins, few fully exploit their advanced capabilities. Understanding DaemonSet internals helps operators avoid common pitfalls such as Pods running on unintended nodes, update storms, or resource contention. This guide provides a deep dive into advanced DaemonSet concepts, including node affinity, rollout strategies, resource management, and diagnostics. By the end, you will be able to design DaemonSets that are robust, efficient, and easy to operate in production.

We will use a practical scenario: deploying a node monitoring agent (like a Prometheus node exporter) as a DaemonSet. The principles apply to any node-level service.

## Version and Environment Inventory

Before implementing advanced DaemonSet features, verify your cluster version and environment. Some features, such as `rollingUpdate` with `maxSurge` and `maxUnavailable`, require Kubernetes 1.12+. Others, like `podAffinity` or `topologySpreadConstraints`, have evolved over releases. Check your API server version:

```bash
kubectl version --short
```

Expected output (example):

```
Client Version: v1.24.3
Server Version: v1.25.2
```

Ensure you have `kubectl` configured with cluster-admin or appropriate RBAC permissions for DaemonSets. Test connectivity:

```bash
kubectl get nodes
```

Example output:

```
NAME     STATUS   ROLES                  AGE   VERSION
node-1   Ready    control-plane,master   10d   v1.25.2
node-2   Ready    <none>                 10d   v1.25.2
node-3   Ready    <none>                 10d   v1.25.2
```

If your cluster has mixed node roles or taints, identify them because DaemonSets must tolerate taints to schedule on certain nodes. For example, control-plane nodes often have a `node-role.kubernetes.io/control-plane` taint. You can list taints:

```bash
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints
```

Example output:

```
NAME     TAINTS
node-1   [map[effect: NoSchedule key: node-role.kubernetes.io/control-plane]]
node-2   <none>
node-3   <none>
```

Finally, note the container runtime and any node-level resources you need to access. For our monitoring agent, we will mount host filesystem paths, so the Pod security settings must allow it.

## Safe Configuration Path

Advanced DaemonSet configuration involves several key areas: node scheduling, update strategy, resource management, and security. We will implement each step incrementally, verifying along the way.

### Step 1: Basic DaemonSet with Node Selector and Tolerations

Start with a minimal DaemonSet that targets all Linux nodes. We'll use a node selector based on `kubernetes.io/os: linux` and add tolerations for common taints to ensure the agent runs everywhere. Save the following manifest as `daemonset-basic.yaml`:

```yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-agent
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: node-agent
  template:
    metadata:
      labels:
        app: node-agent
    spec:
      nodeSelector:
        kubernetes.io/os: linux
      tolerations:
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule
        - key: node-role.kubernetes.io/master
          operator: Exists
          effect: NoSchedule
      containers:
        - name: agent
          image: quay.io/prometheus/node-exporter:v1.3.1
          ports:
            - containerPort: 9100
              name: metrics
          resources:
            requests:
              cpu: 100m
              memory: 64Mi
            limits:
              cpu: 200m
              memory: 128Mi
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            runAsNonRoot: true
            runAsUser: 65534
```

Apply and verify:

```bash
kubectl create namespace monitoring
kubectl apply -f daemonset-basic.yaml
kubectl get daemonset node-agent -n monitoring
```

Expected output:

```
NAME          DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
node-agent    3         3         3       3            3           <none>          10s
```

### Step 2: Advanced Scheduling with Node Affinity

Node selectors are simple but limited. Use `nodeAffinity` for more expressive rules, such as scheduling on nodes with specific labels or excluding others. For example, suppose we want to avoid nodes in a preemptible pool. First label nodes accordingly (if not already):

```bash
kubectl label node node-2 node-pool=regular
kubectl label node node-3 node-pool=regular
kubectl label node node-1 node-pool=control-plane
```

Then update the DaemonSet spec to include `affinity.nodeAffinity`:

```yaml
spec:
  template:
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: node-pool
                    operator: In
                    values: [regular, control-plane]
      nodeSelector: {}
      tolerations: ...  # same as before
```

Apply the change:

```bash
kubectl apply -f daemonset-basic.yaml
```

Check the rollout status:

```bash
kubectl rollout status daemonset node-agent -n monitoring
```

Expected output:

```
daemon set "node-agent" successfully rolled out
```

### Step 3: Rolling Update Strategy

By default, DaemonSets use `RollingUpdate` with `maxUnavailable: 1` and `maxSurge: 0`. This means one node at a time is updated, ensuring availability. For large clusters, you may want faster updates. Configure the update strategy explicitly. In our DaemonSet, add:

```yaml
spec:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 0
```

This is the default, but being explicit documents intent. If your DaemonSet is not critical and you have spare capacity, you could set `maxUnavailable: 2` or `maxSurge: 1` (requires K8s 1.22+ for surge). For now, keep one at a time. Apply and watch the rollout:

```bash
kubectl apply -f daemonset-basic.yaml
kubectl get pods -n monitoring -o wide --watch
```

Observe the controlled replacement: old Pods are terminated after new Pods become ready.

### Step 4: Resource Management and Priority

DaemonSet Pods can be resource hogs. Set requests and limits carefully. In our example, requests are modest. Also consider setting a `priorityClassName` to ensure DaemonSet Pods are scheduled even under resource pressure. Create a PriorityClass if not present:

```yaml
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: node-agent-priority
value: 1000000
globalDefault: false
description: "Priority for node agents"
```

Then add `priorityClassName: node-agent-priority` to the Pod template. Apply both.

### Step 5: Pod Security and Host Access

For accessing host metrics, our container may need to read `/proc` and `/sys`. The node-exporter typically runs as root or with capabilities. We used a non-root user, which may limit functionality. If you need `hostPath` mounts, add them and adjust `securityContext`. For demonstration, we keep it simple, but be aware of Pod Security Standards. In clusters with Pod Security Admission, you may need to set namespace labels to `pod-security.kubernetes.io/enforce=privileged` or use a custom policy.

## Verification and Diagnostics

After deployment, verify that DaemonSet Pods are running correctly on each expected node.

- Check DaemonSet status:

```bash
kubectl get ds node-agent -n monitoring -o wide
```

Expected output shows DESIRED, CURRENT, READY, etc., and NODE SELECTOR/AFFINITY if set.

- Ensure every node has a Pod:

```bash
kubectl get pods -n monitoring -o wide --selector=app=node-agent
```

Example:

```
NAME              READY   STATUS    RESTARTS   AGE    IP          NODE
node-agent-abcde   1/1     Running   0          2m     10.0.1.10   node-1
node-agent-fghij   1/1     Running   0          2m     10.0.2.10   node-2
node-agent-klmno   1/1     Running   0          2m     10.0.3.10   node-3
```

Compare with node list: every Ready node should have a Pod (unless excluded by scheduling).

- Check logs for the agent:

```bash
kubectl logs daemonset/node-agent -n monitoring --tail=10
```

This returns logs from a random Pod. To check a specific Pod:

```bash
kubectl logs node-agent-abcde -n monitoring
```

- Verify metrics endpoint:

Port-forward to a Pod:

```bash
kubectl port-forward pod/node-agent-abcde 9100:9100 -n monitoring
```

Then in another terminal:

```bash
curl localhost:9100/metrics | head
```

Expected output includes lines like:

```
# HELP node_cpu_seconds_total Seconds the CPUs spent in each mode.
# TYPE node_cpu_seconds_total counter
node_cpu_seconds_total{cpu="0",mode="idle"} 12345.67
```

- Describe the DaemonSet to check events:

```bash
kubectl describe ds node-agent -n monitoring
```

Look for events indicating scheduling failures or other issues.

## Failure Modes and Recovery

DaemonSets can fail in several ways. Common issues include:

- **Scheduling failures**: A node may not receive a Pod if it has taints not tolerated, or if nodeSelector/affinity rules exclude it. Check DaemonSet status: DESIRED > CURRENT indicates missing Pods.
- **Image pull errors**: If the image is wrong or registry inaccessible, Pods will be in `ImagePullBackOff`. Check `kubectl describe pod` for details.
- **CrashLoopBackOff**: The container starts but exits repeatedly. Inspect logs and adjust command or resources.
- **Update stuck**: A rolling update may stall if a new Pod never becomes Ready. This can happen due to readiness probe failure or resource constraints. Check `kubectl rollout status` and pod statuses.
- **Node cordon/drain**: If a node is cordoned, DaemonSet Pods may still be scheduled (unless `node.kubernetes.io/unschedulable` taint is added). Understand node lifecycle.

### Rollback

To roll back a DaemonSet update, revert to the previous manifest or use `kubectl rollout undo`:

```bash
kubectl rollout undo daemonset node-agent -n monitoring
```

This works if you have rollout history enabled (DaemonSets support history since 1.15). Check rollout history:

```bash
kubectl rollout history daemonset node-agent -n monitoring
```

Then undo to a specific revision if needed:

```bash
kubectl rollout undo daemonset node-agent -n monitoring --to-revision=2
```

### Forcing Deletion

If a DaemonSet is stuck, you may need to delete it and reapply. Use `--cascade=orphan` to keep Pods running temporarily, then delete Pods manually:

```bash
kubectl delete ds node-agent -n monitoring --cascade=orphan
kubectl delete pods -n monitoring -l app=node-agent
```

Then reapply the corrected manifest.

### Recovery Verification

After rollback, verify the DaemonSet is healthy again:

```bash
kubectl rollout status daemonset node-agent -n monitoring
kubectl get ds node-agent -n monitoring
```

Expected output shows READY equals DESIRED.

## Operations Checklist

Use this checklist for regular DaemonSet operations and reviews.

| Task | Command/Action | Expected Result |
|------|----------------|-----------------|
| Check DaemonSet status | `kubectl get ds -n monitoring` | DESIRED equals CURRENT and READY |
| Verify Pod distribution | `kubectl get pods -n monitoring -o wide` | One Pod per eligible node |
| Check for Pending Pods | `kubectl get pods -n monitoring --field-selector=status.phase=Pending` | No Pods listed |
| Review recent events | `kubectl describe ds node-agent -n monitoring` | No warning events |
| Inspect logs for errors | `kubectl logs daemonset/node-agent -n monitoring --tail=50` | No repeated error patterns |
| Validate resource usage | `kubectl top pods -n monitoring -l app=node-agent` | CPU/Memory within limits |
| Check rollout history | `kubectl rollout history daemonset node-agent -n monitoring` | Multiple revisions available |
| Test update with canary | Temporarily add a node selector to target a single node, update, then expand | Successful rollout on canary node |

Perform this checklist after any cluster change, upgrade, or incident.

## Common Pitfalls and How to Avoid Them

DaemonSet management often trips up teams in predictable ways. Here are the most frequent mistakes and how to steer clear of them.

### Ignoring Taints on New Nodes

**What goes wrong**: A new node is joined with a taint that the DaemonSet does not tolerate. The DaemonSet never schedules a Pod there, leading to unmonitored infrastructure. This typically happens when teams add autoscaling groups or replace nodes without reviewing taints.

**How to avoid it**: Include broad tolerations for common node-role and hardware taints in your DaemonSet spec. Regularly audit nodes and taints with `kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints`. Set up an alert if DESIRED exceeds CURRENT in `kubectl get ds`.

### Over-Aggressive Update Strategies

**What goes wrong**: Setting `maxUnavailable: 100%` or `maxSurge: 100%` to speed up updates can cause all Pods to be replaced simultaneously, leading to a brief outage of the node-level service. For critical agents like log shippers or CNI plugins, this disrupts observability or networking.

**How to avoid it**: Use the default `maxUnavailable: 1` for critical DaemonSets. If you need faster updates, test with a canary node first (see checklist) and monitor rollout status.

### Neglecting Resource Requests and Limits

**What goes wrong**: Without resource requests, the scheduler may place DaemonSet Pods on nodes already under pressure. Without limits, a runaway agent can consume node CPU or memory, affecting other workloads.

**How to avoid it**: Always set `resources.requests` and `limits` appropriate for the agent. Use `kubectl top pods` regularly to verify actual usage. Consider `priorityClassName` to guarantee scheduling.

### Forgetting to Update Pod Security Policies

**What goes wrong**: In clusters with Pod Security Admission (PSA) or older PodSecurityPolicy (PSP), DaemonSet Pods that require privileged access (e.g., mounting host `/proc`) fail to start when namespace policies are enforced. This shows up as `CreateContainerConfigError` or admission denial.

**How to avoid it**: Review the required capabilities for your agent. Use the least privilege possible, but if host access is needed, label the namespace with the appropriate PSA level (e.g., `pod-security.kubernetes.io/enforce=privileged` for the monitoring namespace). Document exceptions clearly.

### Overlooking Node Cordon or Drain Effects

**What goes wrong**: When a node is cordoned, many assume DaemonSet Pods won't be scheduled there, but DaemonSets bypass the unschedulable flag unless the node has the `node.kubernetes.io/unschedulable` taint. Conversely, during a drain, DaemonSet Pods are often ignored by `kubectl drain` unless `--ignore-daemonsets` is set, causing confusion about remaining Pods.

**How to avoid it**: Understand cordon vs. taint semantics. Use `kubectl drain --ignore-daemonsets` when evicting regular workloads, and manually delete DaemonSet Pods if necessary. Document node maintenance procedures in runbooks.

## Advanced Scheduling with Topology Spread Constraints

Beyond node affinity, consider `topologySpreadConstraints` to evenly distribute DaemonSet Pods across failure domains like zones. While a DaemonSet already places one Pod per node, spread constraints become useful when you have multiple DaemonSets that should not co-locate, or when nodes are grouped into logical zones. For example, to spread our monitoring agent across availability zones:

```yaml
spec:
  template:
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: node-agent
```

This ensures that if zones have uneven node counts, the scheduler tries to keep the number of Pods in each zone within one of each other. Apply and verify with `kubectl get pods -n monitoring -o wide --sort-by=.spec.nodeName`.

## Monitoring and Alerting for DaemonSets

To keep DaemonSets healthy, integrate them into your monitoring stack. Here are key metrics to track (using Prometheus as an example):

- `kube_daemonset_status_desired_number_scheduled` vs `kube_daemonset_status_current_number_scheduled`: Alert if desired > current for more than 5 minutes.
- `kube_daemonset_status_number_ready`: Should equal desired.
- `kube_daemonset_status_updated_number_scheduled`: During updates, this temporarily differs; monitor for stuck rollouts.
- Pod restart rate: `rate(kube_pod_container_status_restarts_total{namespace="monitoring"}[5m]) > 0` indicates instability.

Set up alerts with these expressions and route them to your on-call rotation. Review alert thresholds quarterly to avoid noise.

## Security Hardening

Security for DaemonSets goes beyond Pod Security Admission. Consider these practices:

- **Run as non-root**: Always set `runAsNonRoot: true` unless absolutely required. Our node-exporter example uses UID 65534 (nobody), which works for basic metrics but may need root for certain collectors.
- **Use read-only root filesystem**: Set `readOnlyRootFilesystem: true` to prevent tampering.
- **Drop capabilities**: Use `capabilities.drop: ["ALL"]` and add only those needed. For node-exporter, you may need `CAP_SYS_TIME` for some collectors; test carefully.
- **Limit hostPath usage**: Prefer `hostPath` with `type` specified (e.g., `Directory` or `File`) and avoid mounting sensitive paths like `/etc/kubernetes` unless necessary.
- **Network policies**: Apply NetworkPolicies to the namespace to restrict traffic to and from DaemonSet Pods.

For example, a hardened security context for our agent:

```yaml
securityContext:
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  runAsNonRoot: true
  runAsUser: 65534
  capabilities:
    drop: ["ALL"]
    add: ["CAP_SYS_TIME"]
```

Test thoroughly in a staging cluster before applying to production.

## Conclusion

Advanced DaemonSet concepts enable precise control over node-level workloads in Kubernetes. By using node affinity, tolerations, explicit update strategies, and resource management, you can ensure reliable and efficient operation. This guide demonstrated a practical implementation with a monitoring agent, covering configuration, verification, failure recovery, and an operational checklist. Use these patterns as a foundation for your own DaemonSets, adapting to your environment and requirements.

For further reading, consult the official Kubernetes DaemonSet documentation and the API reference for details on all fields.

## Additional Resources

- [Kubernetes DaemonSet Documentation](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/)
- [Kubernetes API Reference: DaemonSetSpec](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.25/#daemonsetspec-v1-apps)
- [Pod Priority and Preemption](https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/)
- [Pod Topology Spread Constraints](https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/)
- [Pod Security Admission](https://kubernetes.io/docs/concepts/security/pod-security-admission/)