## Intro Kubernetes DaemonSets ensure that a copy of a specific Pod runs on all (or some) nodes in a cluster. They are commonly used for log collection agents, monitoring exporters, storage daemons, and node-level networking components. When a DaemonSet misbehaves, operators need a systematic approach to observe current state, identify deviations, make minimal changes, and verify recovery. This article provides practical kubectl commands for daily DaemonSet operations. For each scenario, you will see the exact command, a realistic expected output, failure signals, and recovery steps. The focus is on operational safety: always inspect before changing, use read-only commands for diagnosis, and apply changes only when their scope and rollback path are understood. After reading, you will be able to: - Confirm the DaemonSet version and environment with kubectl version and kubectl cluster-info . - List and describe a DaemonSet and its Pods with labeling and output filtering. - Check rollout history and status. - Update a DaemonSet using kubectl apply , kubectl edit , or kubectl set image , and roll back if needed. - Delete a DaemonSet cleanly, including its Pods. - Diagnose common failure modes such as unschedulable Pods, crash loops, and node selector mismatches. - Use a concise cheat sheet for quick reference. All examples assume kubectl is installed and configured for your cluster. Replace resource names and namespaces with your own values. ## Version and Environment Inventory Before interacting with a DaemonSet, verify that your client and server versions are compatible, and gather basic cluster information. Commands: kubectl version --short kubectl cluster-info kubectl get nodes Example output: Client Version: v1.28.2 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.28.2 Kubernetes control plane is running at https://192.168.49.2:8443 CoreDNS is running at https://192.168.49.2:8443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy NAME STATUS ROLES AGE VERSION minikube Ready control-plane 10d v1.28.2 Key points to check: - Client and server minor versions should be within one minor version of each other per Kubernetes version skew policy. - All nodes should be in Ready state. If any node is NotReady , DaemonSet Pods on it will not be scheduled or will be evicted. - For managed clusters (EKS, GKE, AKS), node status may be aggregated; use kubectl get nodes -o wide for details. Environment discovery for a DaemonSet: Use kubectl get daemonsets --all-namespaces to see all DaemonSets and their namespaces. kubectl get daemonsets --all-namespaces Example output: NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE kube-system kube-proxy 1 1 1 1 1 10d logging fluentd-agent 3 3 3 3 3 5d monitoring node-exporter 3 3 3 3 3 5d If the DaemonSet you expect is missing, check the namespace and the deployment method. For tools installed via Helm, use helm list -A to confirm the release exists. ## Listing and Describing DaemonSets ### List all DaemonSets in the current namespace kubectl get daemonsets Example output: NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE node-exporter 3 3 3 3 3 5d ### List DaemonSets across all namespaces kubectl get daemonsets --all-namespaces ### Describe a specific DaemonSet Describing a DaemonSet shows its configuration, recent events, and status. This is the first diagnostic step when something goes wrong. kubectl describe daemonset node-exporter -n monitoring Key sections in the output: - Selector: labels used to match Pods. - Node-Selector: nodes where Pods are scheduled. - Tolerations: allow scheduling on tainted nodes. - Update Strategy: RollingUpdate or OnDelete. - Events: recent scheduling or update events that can reveal problems. ### Show DaemonSet labels and selectors kubectl get daemonset node-exporter -n monitoring -o yaml kubectl get daemonset node-exporter -n monitoring -o jsonpath='{.spec.selector.matchLabels}' Example output: {"app":"node-exporter"} ### List Pods owned by a DaemonSet kubectl get pods -l app=node-exporter -n monitoring -o wide Example output: NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES node-exporter-abcde 1/1 Running 0 5d 10.244.1.5 worker-1 node-exporter-fghij 1/1 Running 0 5d 10.244.2.5 worker-2 node-exporter-klmno 1/1 Running 0 5d 10.244.3.5 worker-3 Observe whether a Pod exists for every intended node. If a node is missing a Pod, check node taints, labels, and DaemonSet tolerations. ## Viewing DaemonSet Configuration and YAML To understand exactly what a DaemonSet does, inspect its full YAML definition. kubectl get daemonset node-exporter -n monitoring -o yaml Save it to a file for editing or backup: kubectl get daemonset node-exporter -n monitoring -o yaml > node-exporter-ds.yaml Examine key fields: - spec.template.spec.containers[].image : the container image and tag. - spec.updateStrategy : RollingUpdate or OnDelete . - spec.minReadySeconds : minimum seconds a Pod must be ready before considered available. - spec.revisionHistoryLimit : number of old ReplicaSets to retain for rollback. ## Checking Rollout History and Status DaemonSets support rolling updates, similar to Deployments. To verify that an update completed successfully, use rollout commands. ### Check rollout status kubectl rollout status daemonset/node-exporter -n monitoring Successful output: daemonset "node-exporter" successfully rolled out If the rollout is stuck, the command may time out or print messages like: Waiting for daemon set "node-exporter" rollout to finish: 2 of 3 updated pods are available... ### View rollout history kubectl rollout history daemonset/node-exporter -n monitoring Example output: daemonset.apps/node-exporter REVISION CHANGE-CAUSE 1 2 kubectl set image daemonset/node-exporter node-exporter=prom/node-exporter:v1.6.1 --record=true Note: --record=true is deprecated in newer versions; change-cause is no longer stored automatically. To see details of a specific revision: kubectl rollout history daemonset/node-exporter -n monitoring --revision=2 ## Updating a DaemonSet Updates can be made via kubectl apply , kubectl edit , or kubectl set image . Always check the update strategy first. ### Check update strategy kubectl get daemonset node-exporter -n monitoring -o jsonpath='{.spec.updateStrategy.type}' Example output: RollingUpdate ### Apply a manifest change Edit the YAML file you saved earlier, or apply a new one: kubectl apply -f node-exporter-ds.yaml Then monitor the rollout with kubectl rollout status daemonset/node-exporter -n monitoring . ### Update container image with kubectl set image kubectl set image daemonset/node-exporter node-exporter=prom/node-exporter:v1.7.0 -n monitoring Example output: daemonset.apps/node-exporter image updated Verify the new image is used: kubectl get daemonset node-exporter -n monitoring -o jsonpath='{.spec.template.spec.containers[0].image}' ### Edit live object kubectl edit daemonset node-exporter -n monitoring This opens the object in your default editor. Changes are applied on save, and a rollout will start if the update strategy is RollingUpdate. ## Rolling Back a DaemonSet If an update introduces problems, roll back to a previous revision. ### Rollback to previous revision kubectl rollout undo daemonset/node-exporter -n monitoring Example output: daemonset.apps/node-exporter rolled back ### Rollback to a specific revision kubectl rollout undo daemonset/node-exporter -n monitoring --to-revision=1 After rollback, check status and Pod versions: kubectl rollout status daemonset/node-exporter -n monitoring kubectl get pods -l app=node-exporter -n monitoring -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\n"}{end}' ## Deleting a DaemonSet Deleting a DaemonSet removes the DaemonSet controller and, by default, all Pods it created. ### Delete the DaemonSet and its Pods kubectl delete daemonset node-exporter -n monitoring Example output: daemonset.apps "node-exporter" deleted ### Delete only the DaemonSet without deleting Pods Use --cascade=orphan to leave Pods running. This is rarely desired but can be useful during migration. kubectl delete daemonset node-exporter -n monitoring --cascade=orphan The orphaned Pods will continue to run, but they will no longer be managed by a DaemonSet. You can adopt them later with a new controller or delete them manually.

Quick check 2 of 2 What is the main difference between a Deployment and a DaemonSet according to the passage? - Deployments are used for node-level services, while DaemonSets are for stateless services - DaemonSets are used for control plane components, while Deployments are for worker nodes - Deployments are used for stateless services where scaling is more important, while DaemonSets are used when a copy of a Pod must run on all or certain hosts - There is no difference; they are interchangeable

The passage explains that Deployments are for stateless services where scaling matters, while DaemonSets are for node-level functionality that must run on all or certain nodes.

## Diagnostic Commands for DaemonSet Pods When Pods are not running correctly, use the following commands to gather information. ### Check Pod status kubectl get pods -l app=node-exporter -n monitoring -o wide Look for non- Running statuses: Pending , CrashLoopBackOff , ImagePullBackOff , Evicted , etc. ### Describe a problematic Pod kubectl describe pod node-exporter-abcde -n monitoring Key parts of the output: - Events: show recent scheduling and lifecycle events, such as failed image pulls or insufficient resources. - Conditions: indicate Pod readiness and other status. - Tolerations: show which taints the Pod tolerates. ### View logs kubectl logs node-exporter-abcde -n monitoring For a crash-looping Pod, get logs from the previous container instance: kubectl logs node-exporter-abcde -n monitoring --previous ### Execute commands inside the container kubectl exec -it node-exporter-abcde -n monitoring -- /bin/sh Then run diagnostic commands like ps aux , df -h , or cat /etc/hostname . ## Common Failure Modes and Recovery ### Failure 1: DaemonSet Pods not scheduled on some nodes Symptom: kubectl get pods -l app=node-exporter -o wide shows fewer Pods than nodes. Possible causes and checks: - Node selector mismatch: kubectl get ds node-exporter -o yaml | grep -A5 nodeSelector . - Taints and tolerations: kubectl describe node | grep Taints , and compare with DaemonSet tolerations . - Node is cordoned: kubectl get node shows SchedulingDisabled . - Resource insufficient: kubectl describe node | grep -A10 "Allocated resources" . Recovery: - Adjust node selector or tolerations in the DaemonSet spec and apply. - Uncordon the node with kubectl uncordon if cordoned. - Free up resources or adjust requests. ### Failure 2: Pods in CrashLoopBackOff Symptom: kubectl get pods shows CrashLoopBackOff or many restarts. Diagnostic: kubectl logs -n --previous kubectl describe pod -n Look for exit codes and error messages. Common causes: misconfiguration, missing dependencies, incorrect command arguments. Recovery: - Fix the configuration and update the DaemonSet. - If the image is bad, roll back to a previous version. ### Failure 3: ImagePullBackOff Symptom: Pods stuck in ImagePullBackOff or ErrImagePull . Diagnostic: kubectl describe pod -n | grep -A5 Events Check that the image name and tag are correct, and that the node can access the registry (network, credentials). Recovery: - Correct the image reference. - If using a private registry, ensure imagePullSecrets are set in the DaemonSet spec. - Roll back if necessary. ### Failure 4: Rolling update stuck Symptom: kubectl rollout status hangs, and not all Pods are updated. Diagnostic: - kubectl describe daemonset -n for events. - Check Pod status on individual nodes; often due to unschedulable new Pods (see Failure 1). - Verify maxUnavailable and minReadySeconds settings. Recovery: - Resolve the underlying scheduling or Pod issue. - If needed, pause the rollout: kubectl rollout pause daemonset/ -n . - Resume with kubectl rollout resume daemonset/ -n after fixing. ## Operations Checklist Use this checklist before and after DaemonSet operations to reduce risk. Before any change: - [ ] Verify cluster health: kubectl get nodes all Ready. - [ ] List current DaemonSets: kubectl get ds -n . - [ ] Describe target DaemonSet: kubectl describe ds -n . - [ ] Export current YAML to backup: kubectl get ds -n -o yaml > backup.yaml . - [ ] Check update strategy and rollout history: kubectl get ds -o jsonpath='{.spec.updateStrategy.type}' and kubectl rollout history ds/ . - [ ] Understand node selector and tolerations to ensure Pods will schedule as expected. After any change: - [ ] Run kubectl rollout status ds/ -n and confirm success. - [ ] Check Pod status across nodes: kubectl get pods -l -o wide -n . - [ ] View logs of new Pods for runtime errors. - [ ] Verify application behavior (e.g., metrics endpoint, logs collected). - [ ] If failure, rollback with kubectl rollout undo ds/ -n and verify again. ## DaemonSet Commands Cheat Sheet The following table summarizes essential commands for quick reference.

Operation	Command
List DaemonSets in current namespace	`kubectl get daemonsets`
List DaemonSets in all namespaces	`kubectl get daemonsets --all-namespaces`
Describe a DaemonSet	`kubectl describe daemonset <name> -n <namespace>`
Get YAML of a DaemonSet	`kubectl get daemonset <name> -n <namespace> -o yaml`
Check rollout status	`kubectl rollout status daemonset/<name> -n <namespace>`
View rollout history	`kubectl rollout history daemonset/<name> -n <namespace>`
Update image	`kubectl set image daemonset/<name> <container>=<new-image> -n <namespace>`
Rollback to previous	`kubectl rollout undo daemonset/<name> -n <namespace>`
Delete DaemonSet	`kubectl delete daemonset <name> -n <namespace>`
List Pods belonging to DaemonSet	`kubectl get pods -l <selector> -n <namespace>`
Get logs of a DaemonSet Pod	`kubectl logs <pod-name> -n <namespace>`
Execute command in Pod	`kubectl exec -it <pod-name> -n <namespace> -- /bin/sh`

## Conclusion Mastering DaemonSet commands is essential for anyone operating Kubernetes clusters. By following the structured approach in this article, you can confidently inspect, update, and recover DaemonSets while minimizing risk. Always start with read-only commands to observe the current state, make one scoped change at a time, and verify the result with rollout status and Pod checks. Keep backups of YAML definitions and understand rollback procedures. With these practices, you can keep node-level services running smoothly across your cluster.