## Intro Kubernetes device plugins enable pods to access specialized hardware such as GPUs, FPGAs, and NICs. However, misconfigurations can lead to failed pod scheduling, resource deadlocks, or even node crashes. This practical guide targets developers, DevOps consultants, and technical startup teams that need to configure device plugins safely and resolve issues quickly. We will walk through a structured approach: first, take a complete version and environment inventory; then apply safe configuration changes; next, verify and diagnose problems; and finally, handle failure modes with recovery procedures. Each section includes concrete commands, expected outputs, and real-world examples. By following this operational safety framework—observe before changing, limit blast radius, avoid hardcoded secrets, verify results, and document rollback paths—you will reduce risk and downtime. ## Version and Environment Inventory Before changing any configuration, you must understand your Kubernetes cluster and the device plugin ecosystem. Capturing exact versions, deployment topology, and prerequisites prevents avoidable mistakes. Start with read-only observations, record timestamps, and protect credentials. ### Step 1: Collect Cluster and Node Information Run the following commands to snapshot the current state: ```bash kubectl version --short kubectl get nodes -o wide kubectl get pods -n kube-system -l component=kubelet -o wide ``` Expected output looks like: ```text Client Version: v1.27.3 Server Version: v1.27.3 NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME node-1 Ready control-plane 10d v1.27.3 10.0.0.10 Ubuntu 22.04.3 LTS 5.15.0-91-generic containerd://1.7.2 node-2 Ready 10d v1.27.3 10.0.0.11 Ubuntu 22.04.3 LTS 5.15.0-91-generic containerd://1.7.2 ``` **Key checks:** - Kubernetes version: Device plugin API `v1beta1` was deprecated in 1.20 and removed in 1.25; only `v1` is supported in modern clusters. Confirm the plugin is built against the correct API. - Node OS/kernel: Some plugins require specific kernel modules or drivers (e.g., NVIDIA driver for GPU). - Container runtime: Plugins interact with the runtime through the kubelet's device plugin registration mechanism; mismatch can cause registration failures. ### Step 2: Inspect Existing Device Plugin Deployments List all pods in `kube-system` and filter for device plugin names: ```bash kubectl get pods -n kube-system | grep -E 'device-plugin|nvidia|intel|amd|fpga|nic' ``` Example output: ```text nvidia-device-plugin-daemonset-6xz7m 1/1 Running 0 3d ``` If none appear, check the DaemonSet: ```bash kubectl get ds -n kube-system | grep -i device ``` Expected output might show: ```text NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE nvidia-device-plugin 2 2 2 2 2 3d ``` Use `kubectl describe ds -n kube-system` to view the pod template, resource requests, and tolerations. ### Step 3: Verify Node Allocatable Resources and Device Counts Check if the plugin has advertised devices to kubelet: ```bash kubectl describe node | grep -A 10 'Allocatable' ``` Look for extended resources like `nvidia.com/gpu: 2` or `amd.com/gpu: 1`. If absent, the plugin is not healthy or not registered. ### Step 4: Review Plugin Logs for Startup Errors ```bash kubectl logs -n kube-system --tail=50 ``` Watch for messages like: ```text I0628 10:00:00.123456 1 main.go:120] Starting FS watcher. I0628 10:00:00.234567 1 main.go:130] Starting OS watcher. E0628 10:00:00.345678 1 main.go:140] Failed to initialize NVML: could not load NVML library. ``` The last line indicates a missing driver—a common mistake. ## Safe Configuration Path When you need to change the device plugin configuration, adopt a step-by-step approach that minimizes risk. Apply one change at a time, validate in a test environment if possible, and always have a rollback plan. ### Principle: Least Privilege and Minimal Change A frequent mistake is rolling out a plugin update to all nodes without testing. Instead: 1. Choose a single node or a non-production node pool. 2. Label it for testing: `kubectl label node node-1 test-device-plugin=true`. 3. Adjust the DaemonSet's node selector temporarily using a patch or a separate manifest. Example patch to target only nodes with label `test-device-plugin=true`: ```yaml spec: template: spec: nodeSelector: test-device-plugin: "true" ``` Apply with: ```bash kubectl patch ds nvidia-device-plugin -n kube-system --patch '{"spec":{"template":{"spec":{"nodeSelector":{"test-device-plugin":"true"}}}}}' ``` Wait for rollout and check pod status: ```bash kubectl rollout status ds/nvidia-device-plugin -n kube-system ``` Expected output: ```text daemon set "nvidia-device-plugin" successfully rolled out ``` If successful, gradually expand the label to other nodes. ### Placeholders Instead of Secrets Never put credentials directly in the plugin configuration. Use Kubernetes Secrets and reference them via environment variables or mounted volumes. For example, if a plugin needs a license key, create a Secret: ```bash kubectl create secret generic device-plugin-license --from-literal=license-key='abc123' -n kube-system ``` Then in the DaemonSet manifest: ```yaml env: - name: LICENSE_KEY valueFrom: secretKeyRef: name: device-plugin-license key: license-key ``` Avoid hardcoding the key in the container spec; it appears in `kubectl describe` and logs. ### Example: Updating the NVIDIA Device Plugin Version Safely Suppose you need to upgrade from version `v0.12.2` to `v0.14.0`. Steps: 1. Save the current manifest: ```bash kubectl get ds nvidia-device-plugin -n kube-system -o yaml > nvidia-device-plugin-v0.12.2-backup.yaml ``` 2. Apply the new manifest with the new image, but only to the test node (using the label selector). 3. Monitor logs and node resource status: ```bash kubectl logs -n kube-system -l name=nvidia-device-plugin-ds --tail=20 kubectl describe node node-1 | grep nvidia.com/gpu ``` 4. If the new version fails, rollback by reapplying the backup manifest. 5. Once satisfied, remove the node selector to deploy cluster-wide. ## Verification and Diagnostics After configuration changes, verify that the plugin is healthy and devices are available. Diagnosing issues early saves hours of troubleshooting. ### Verification Commands **1. Check DaemonSet Status** ```bash kubectl get ds -n kube-system | grep nvidia ``` Expected: `DESIRED` equals `READY` and `AVAILABLE`. **2. Confirm Pod Health** ```bash kubectl get pods -n kube-system -l name=nvidia-device-plugin-ds -o wide ``` Look for `Running` and `Ready 1/1`. If a pod is not ready, use `kubectl describe pod -n kube-system` to view events. Look for `FailedScheduling`, `CrashLoopBackOff`, or `FailedMount`. **3. Validate Device Registration on Node** ```bash kubectl get node node-1 -o json | jq '.status.allocatable' ``` If `nvidia.com/gpu` appears with a value greater than 0, the plugin is advertising devices. **4. Test Pod Scheduling with GPU Request** Create a test pod that requests a GPU: ```yaml apiVersion: v1 kind: Pod metadata: name: gpu-test-pod spec: restartPolicy: Never containers: - name: cuda-vector-add image: k8s.gcr.io/cuda-vector-add:v0.1 resources: limits: nvidia.com/gpu: 1 ``` Apply and check status: ```bash kubectl apply -f gpu-test-pod.yaml kubectl get pod gpu-test-pod ``` If it remains `Pending`, inspect events: ```bash kubectl describe pod gpu-test-pod ``` Look for error messages like `Insufficient nvidia.com/gpu` (no devices available) or `Failed to allocate` (plugin issue). ### Diagnostics for Common Issues **Issue 1: Pod Pending with "Insufficient nvidia.com/gpu"** - Cause: Node does not have GPUs or plugin not running. - Diagnose: `kubectl get ds -n kube-system`, `kubectl logs -n kube-system `. - Fix: Ensure plugin DaemonSet is running on the node and driver is installed. **Issue 2: Plugin Pod CrashLoopBackOff** - Cause: Missing library, permission error, or incompatible API. - Logs might show: `Fatal: failed to start device plugin: rpc error: code = Unimplemented desc = unknown service v1beta1.DevicePlugin` (if using old API on new kubelet). - Fix: Use a plugin version compatible with your Kubernetes version, or update `--device-plugin-version` (if supported). **Issue 3: Device Not Visible in Allocatable** - Cause: Plugin not registering due to socket path issue. - The plugin typically creates a Unix socket at `/var/lib/kubelet/device-plugins/`; ensure the DaemonSet mounts that directory with `hostPath`. - Check pod spec for volume mounts: ```yaml volumeMounts: - name: device-plugin mountPath: /var/lib/kubelet/device-plugins volumes: - name: device-plugin hostPath: path: /var/lib/kubelet/device-plugins ``` ## Failure Modes and Recovery Even with careful planning, failures happen. Knowing common failure modes and having a recovery plan reduces mean time to recovery (MTTR). ### Failure Mode 1: Plugin Upgrade Breaks Device Discovery **Scenario:** You updated the plugin from v1 to v2, but v2 requires a different driver version, causing the plugin to crash on startup. **Recovery:** 1. Immediately rollback the DaemonSet to the previous version: ```bash kubectl rollout undo ds/nvidia-device-plugin -n kube-system ``` Or apply the backup manifest. 2. Verify that pods return to Running and devices reappear: ```bash kubectl get pods -n kube-system -l name=nvidia-device-plugin-ds kubectl describe node node-1 | grep nvidia.com/gpu ``` ### Failure Mode 2: Misconfigured Resource Limits Cause Scheduling Deadlock **Scenario:** You set the plugin to advertise a higher number of devices than physically present by editing the `--device-count` flag. Pods are scheduled but fail when starting. **Recovery:** 1. Correct the flag in the DaemonSet spec and roll out: ```bash kubectl edit ds nvidia-device-plugin -n kube-system # change --device-count=N to actual number ``` 2. Delete the stuck pods or let them be evicted; they will be rescheduled with correct counts. ### Failure Mode 3: Node Cordon and Drain for Plugin Maintenance **Scenario:** You need to update the host driver (e.g., NVIDIA driver) and must drain the node. **Steps:** 1. Cordon the node: ```bash kubectl cordon node-1 ``` 2. Drain pods (but not daemonsets by default; include `--ignore-daemonsets`): ```bash kubectl drain node-1 --ignore-daemonsets --delete-emptydir-data ``` 3. Perform maintenance (driver update, reboot). 4. Uncordon: ```bash kubectl uncordon node-1 ``` 5. Verify plugin pod restarts and devices register. **Important:** Always have a recent backup of the DaemonSet manifest and know how to roll back. ## Operations Checklist Use this checklist before and after any device plugin configuration change. It encapsulates best practices to avoid common mistakes. ### Pre-Change Checklist | Item | Command / Check | Expected Result | |------|-----------------|-----------------| | 1. Confirm cluster version | `kubectl version --short` | Server >= 1.25 for v1 API only | | 2. Confirm plugin version compatibility | Read plugin docs or `kubectl describe ds -n kube-system` | Plugin version supports cluster version | | 3. Check node resource status | `kubectl describe node \| grep -A5 Allocatable` | Extended resources visible if plugin working | | 4. Backup current manifest | `kubectl get ds -n kube-system -o yaml > backup.yaml` | Backup file created | | 5. Label test node | `kubectl label node test-device-plugin=true` | Label applied | | 6. Ensure no hardcoded secrets | `kubectl get ds -n kube-system -o yaml \| grep -i secret` | Only references to Secret objects | ### Post-Change Checklist | Item | Command / Check | Expected Result | |------|-----------------|-----------------| | 1. DaemonSet rollout status | `kubectl rollout status ds/ -n kube-system` | "successfully rolled out" | | 2. Pod health | `kubectl get pods -n kube-system -l name=` | Running and Ready 1/1 | | 3. Device registration | `kubectl describe node \| grep ` | Extended resource count >0 | | 4. Test pod scheduling | Apply test pod requesting device; check status | Pod becomes Running or completes | | 5. Logs check | `kubectl logs -n kube-system --tail=50` | No fatal errors | | 6. Rollback procedure documented | Have `kubectl rollout undo` command ready | Able to revert quickly | Example filled checklist for a fictional NVIDIA plugin update: - Test node label: `node-1` labeled `test-device-plugin=true`. - Backup file: `nvidia-device-plugin-v0.12.2-backup.yaml` saved. - Post-change: `nvidia.com/gpu` shows `2` on node-1. - Rollback: `kubectl rollout undo ds/nvidia-device-plugin -n kube-system` verified. ## Conclusion Kubernetes device plugin configuration mistakes can be avoided with a disciplined approach. Always start by inventorying versions, environment, and current state. Make small, reversible changes with secrets properly handled. Verify thoroughly using the commands and expected outputs provided. When failures occur, rely on rollback procedures and documented recovery steps. By adopting these practices, you reduce the risk of downtime and keep your GPU-accelerated workloads running smoothly. Begin by implementing the version inventory and safe configuration path in your next plugin update, and expand from there. Remember: a reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery before an incident forces the decision.