## Intro Kubernetes device plugins bridge the gap between specialized hardware and your workloads. They advertise resources such as GPUs, FPGAs, NICs, or RDMA capabilities to the kubelet, which then schedules pods requiring those resources. Automating their deployment and lifecycle through CI/CD is challenging due to their close coupling with node hardware, kernel drivers, and cluster versions. This article provides a practical, step-by-step approach to integrating device plugins into your CI/CD pipeline, ensuring safe rollouts, immediate verification, and reliable rollback. We focus on common device plugins like NVIDIA GPU, Intel FPGA, and generic sample plugins, demonstrating how to observe cluster state, apply changes in a controlled manner, and verify functionality. Whether you run on-prem or in cloud GPU instances, the principles remain the same. By the end, you will have a concrete workflow: starting from version inventory to automated deployment, health checks, failure detection, and rollback strategies, all backed by command-line examples and configuration snippets. ## Version and Environment Inventory Before automating anything, you must know your current landscape. Device plugins are version-sensitive: a mismatch between the plugin, Kubernetes API, container runtime, and hardware drivers can cause silent failures. Begin by gathering read-only information about your cluster and nodes. Start with cluster-level details: ```bash kubectl version --short kubectl get nodes -o wide kubectl get pods -n kube-system -l app.kubernetes.io/component=device-plugin ``` Expected output example: ``` Client Version: v1.27.3 Server Version: v1.27.3 NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME gpu-node1 Ready 30d v1.27.3 10.10.0.4 Ubuntu 22.04.3 LTS 5.15.0-91-generic containerd://1.7.2 gpu-node2 Ready 30d v1.27.3 10.10.0.5 Ubuntu 22.04.3 LTS 5.15.0-91-generic containerd://1.7.2 ``` For each node hosting device plugins, check the kubelet configuration and device plugin directory: ```bash kubectl describe node gpu-node1 | grep -A5 "Capacity:" ls -la /var/lib/kubelet/device-plugins/ # on the node ``` The kubelet's device plugin registration path defaults to `/var/lib/kubelet/device-plugins/`, but this can vary. Verify the `--device-plugin-enabled` flag and the registered resources: ```bash kubectl describe node gpu-node1 | grep -A20 "Allocated resources:" ``` For NVIDIA GPUs, confirm driver and runtime versions: ```bash # On node nvidia-smi nvidia-container-cli info ``` Example output: ``` +-----------------------------------------------------------------------------+ | NVIDIA-SMI 535.104.05 Driver Version: 535.104.05 CUDA Version: 12.2 | |-------------------------------+----------------------+----------------------+ ... ``` Check if the device plugin daemonset is already running: ```bash kubectl get ds -n kube-system | grep nvidia kubectl get pods -n kube-system -l name=nvidia-device-plugin-ds ``` Record all findings in your inventory sheet. This observation step protects against incompatible upgrades. ## Safe Configuration Path The safest way to introduce changes is gradually: first in a staging environment, then a canary node, and finally rollout to all nodes. Device plugins run as DaemonSets, so changes affect all matching nodes. Control this via node selectors and tolerations. Common device plugin manifests include a DaemonSet with a hostPath mount for `/var/lib/kubelet/device-plugins`. Here is a minimal example for the NVIDIA device plugin: ```yaml apiVersion: apps/v1 kind: DaemonSet metadata: name: nvidia-device-plugin-daemonset namespace: kube-system spec: selector: matchLabels: name: nvidia-device-plugin-ds template: metadata: labels: name: nvidia-device-plugin-ds spec: priorityClassName: system-node-critical tolerations: - key: nvidia.com/gpu operator: Exists effect: NoSchedule containers: - name: nvidia-device-plugin-ctr image: nvcr.io/nvidia/k8s-device-plugin:v0.14.3 env: - name: FAIL_ON_INIT_ERROR value: "false" securityContext: allowPrivilegeEscalation: false capabilities: drop: ["ALL"] volumeMounts: - name: device-plugin mountPath: /var/lib/kubelet/device-plugins volumes: - name: device-plugin hostPath: path: /var/lib/kubelet/device-plugins ``` First, deploy Node Feature Discovery (NFD) to automatically label GPU-enabled nodes. NFD discovers hardware features and applies labels such as `feature.node.kubernetes.io/pci-10de.present=true` to nodes with NVIDIA GPUs, removing the need for manual `kubectl label` commands. ```bash kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/node-feature-discovery/master/deployment/base/nfd.yaml kubectl get nodes -l feature.node.kubernetes.io/pci-10de.present=true ``` To apply the device plugin to a single canary node, patch the DaemonSet to target that node by hostname: ```bash kubectl patch ds nvidia-device-plugin-daemonset -n kube-system -p '{"spec":{"template":{"spec":{"nodeSelector":{"kubernetes.io/hostname":"gpu-node1"}}}}}' ``` Verify the plugin registers on that node only: ```bash kubectl describe node gpu-node1 | grep nvidia.com/gpu ``` Expected output: ``` nvidia.com/gpu: 1 nvidia.com/gpu: 1 ``` If successful, promote to all GPU nodes by patching the DaemonSet nodeSelector to the NFD-generated label: ```bash kubectl patch ds nvidia-device-plugin-daemonset -n kube-system -p '{"spec":{"template":{"spec":{"nodeSelector":{"feature.node.kubernetes.io/pci-10de.present":"true"}}}}}' ``` Keep change sets small and reversible. ## Verification and Diagnostics After deployment, verify not just that pods are running, but that the plugin advertised resources correctly and that workloads can consume them. First, confirm the daemonset pods are healthy: ```bash kubectl get pods -n kube-system -l name=nvidia-device-plugin-ds -o wide kubectl rollout status ds/nvidia-device-plugin-daemonset -n kube-system ``` Check logs for registration success: ```bash kubectl logs -n kube-system -l name=nvidia-device-plugin-ds --tail=50 ``` Look for lines like: ``` I0912 10:00:01.234567 1 main.go:154] Starting FS watcher. I0912 10:00:01.234567 1 main.go:161] Starting OS watcher. I0912 10:00:01.234567 1 main.go:176] Starting to serve on /var/lib/kubelet/device-plugins/nvidia-gpu.sock I0912 10:00:01.234567 1 main.go:183] Registered device plugin with Kubelet ``` To test actual resource allocation, create a pod requesting a GPU: ```yaml apiVersion: v1 kind: Pod metadata: name: gpu-test spec: restartPolicy: Never containers: - name: cuda-vector-add image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda11.7.1-ubi8 resources: limits: nvidia.com/gpu: 1 ``` Apply and check logs: ```bash kubectl apply -f gpu-test.yaml kubectl logs gpu-test ``` Expected output: ``` [Vector addition of 50000 elements] Copy input data from the host memory to the CUDA device CUDA kernel launch with 196 blocks of 256 threads Copy output data from the CUDA device to the host memory Test PASSED Done ``` If the pod fails to schedule, inspect events: ```bash kubectl describe pod gpu-test | grep -A10 Events ``` Watch for messages like: ``` 0/2 nodes are available: 2 Insufficient nvidia.com/gpu. ``` This indicates the plugin did not register resources or the node selector excluded all nodes. ## Failure Modes and Recovery Device plugin failures can be subtle. Common scenarios include: 1. **Registration socket not created**: The plugin container runs but cannot create the Unix socket due to permission issues or wrong mount path. Diagnosis: ```bash kubectl logs -n kube-system -l name=nvidia-device-plugin-ds | grep -i error ``` Example error: ``` E0912 10:05:00.000000 1 main.go:123] failed to create device plugin socket: mkdir /var/lib/kubelet/device-plugins: permission denied ``` Recovery: Ensure the container runs as root or with appropriate security context, and the hostPath directory has correct permissions (typically 0755). 2. **Kubelet restart disconnects plugins**: When kubelet restarts, plugins must re-register. If the plugin does not handle reconnection, resources disappear until pod restart. Diagnosis: ```bash kubectl get node gpu-node1 -o json | jq '.status.capacity' ``` Compare capacity before and after kubelet restart. If `nvidia.com/gpu` is missing, plugin may not have re-registered. Recovery: Restart the plugin daemonset. ```bash kubectl rollout restart ds/nvidia-device-plugin-daemonset -n kube-system ``` 3. **Driver or runtime update breaks compatibility**: After a node kernel or driver update, the plugin may fail to initialize. Diagnosis: Check plugin logs for `FAIL_ON_INIT_ERROR` messages. If you set `FAIL_ON_INIT_ERROR: false`, the plugin may run but not register. Set it to `true` to crash and make the failure visible. Recovery: Roll back the driver or update the plugin to a compatible version. 4. **Resource leak or stale socket**: Old device plugin sockets can persist and confuse kubelet. Diagnosis: ```bash ls -la /var/lib/kubelet/device-plugins/ ``` If you see duplicate sockets or outdated ones, clean them manually after stopping the plugin. Recovery: ```bash systemctl stop kubelet rm -f /var/lib/kubelet/device-plugins/*.sock systemctl start kubelet kubectl delete pod -n kube-system -l name=nvidia-device-plugin-ds ``` For rollbacks in a CI/CD context, maintain versioned manifests and use `kubectl apply` with the previous version. For instance, if you upgraded from v0.14.2 to v0.14.3 and face issues, revert: ```bash kubectl set image ds/nvidia-device-plugin-daemonset -n kube-system nvidia-device-plugin-ctr=nvcr.io/nvidia/k8s-device-plugin:v0.14.2 ``` Monitor rolling update status: ```bash kubectl rollout status ds/nvidia-device-plugin-daemonset -n kube-system ``` ## Operations Checklist Use this checklist before and after applying device plugin changes in production. Assign a single accountable owner for each step and specify review frequency. | # | Action | Command / Verification | Owner | Frequency | |---|--------|------------------------|-------|-----------| | 1 | Verify node hardware and driver compatibility | Run `nvidia-smi` on all target nodes and compare with plugin supported matrix | Infrastructure Lead (e.g., Priya Shah) | Before each rollout | | 2 | Apply plugin manifest to canary node | `kubectl apply -f nvidia-device-plugin-canary.yaml` (with node selector) | DevOps Engineer (e.g., Alex Chen) | Per deployment | | 3 | Validate plugin registration | `kubectl describe node | grep nvidia.com/gpu` | DevOps Engineer | Per deployment | | 4 | Run GPU test pod | `kubectl apply -f gpu-test.yaml && kubectl logs gpu-test` | QA Lead (e.g., Maria Gomez) | Per deployment | | 5 | Promote to all nodes | Patch DaemonSet nodeSelector to match all GPU nodes | DevOps Engineer | After canary verification | | 6 | Monitor node capacity and pod scheduling | `kubectl get nodes -o custom-columns=NAME:.metadata.name,GPU:.status.capacity.nvidia\\.com/gpu` | SRE (e.g., John Lee) | Daily for first week | | 7 | Check for leaked sockets or stale processes | `ls -la /var/lib/kubelet/device-plugins/` on nodes | Node Admin (e.g., Sam Patel) | Weekly | | 8 | Review plugin version and known CVEs | Consult vendor release notes and CVE databases | Security Officer (e.g., Emily Brown) | Monthly | Each item should be logged in your CI/CD system with a timestamp and outcome. If any check fails, halt the pipeline and trigger the rollback procedure. ## Common Pitfalls and How to Avoid Them Despite best efforts, mistakes happen. Here are frequent pitfalls in device plugin automation: ### Ignoring Node Heterogeneity **Why it happens**: Not all nodes have the same hardware or drivers. Applying a plugin uniformly can cause failures on incompatible nodes. **How to avoid**: Use NFD-discovered labels and node selectors to target only nodes with the specific hardware. Maintain separate DaemonSets for different hardware types if needed. **Recovery**: If a plugin crashes on a node, isolate it by cordoning the node and deleting the plugin pod. Example: ```bash kubectl cordon gpu-node2 kubectl delete pod -n kube-system -l name=nvidia-device-plugin-ds --field-selector spec.nodeName=gpu-node2 ``` Then fix the driver or replace hardware. ### Misconfigured Mount Paths **Why it happens**: Custom kubelet configurations may use a different device plugin directory. The default is `/var/lib/kubelet/device-plugins`, but some clusters use `/var/lib/kubelet/device-plugins/kubelet.sock` or a custom path. **How to avoid**: Check kubelet flags on nodes: ```bash ps aux | grep kubelet | grep device-plugins ``` Look for `--device-plugin-path` or similar. Ensure your DaemonSet mounts match. **Recovery**: Update the DaemonSet volume mount to the correct path and restart pods. ### Not Handling Plugin Downtime **Why it happens**: Rolling updates of device plugins may briefly remove resources from nodes, causing pending pods. If workloads are sensitive, this can cause disruptions. **How to avoid**: Use `maxUnavailable: 0` in DaemonSet update strategy (if supported) or perform updates during maintenance windows. For critical workloads, consider node draining before plugin update. **Recovery**: If pods get stuck, restart them after plugin is healthy. ### Overlooking Security Contexts **Why it happens**: Device plugins often need privileged access or specific capabilities. Running with too many permissions is risky, but too restrictive can break registration. **How to avoid**: Follow least privilege. For NVIDIA plugin, drop all capabilities and set `allowPrivilegeEscalation: false` as shown earlier. Avoid running as root if possible, but some plugins require it. **Recovery**: If plugin fails due to security context, adjust and redeploy, then test with a GPU pod. ## Conclusion Automating Kubernetes device plugins in CI/CD requires careful observation, controlled changes, and rigorous verification. By following the structured approach outlined—from inventory and safe configuration to diagnostics and rollback—you can reduce downtime and ensure hardware resources are reliably available to your workloads. Start by implementing the Operations Checklist in your pipeline for the next device plugin update. Assign owners, run the canary deployment, and verify with a sample workload. Over time, you will build confidence and catch issues before they affect production. Remember: device plugins are a critical bridge between hardware and Kubernetes. Treat their lifecycle with the same rigor as any other infrastructure component.