Kubernetes Device Plugins: CI/CD Automation in Practice

Intro

Kubernetes device plugins bridge the gap between specialized hardware and your workloads. They advertise resources such as GPUs, FPGAs, NICs, or RDMA capabilities to the kubelet, which then schedules pods requiring those resources. Automating their deployment and lifecycle through CI/CD is challenging due to their close coupling with node hardware, kernel drivers, and cluster versions. This article provides a practical, step-by-step approach to integrating device plugins into your CI/CD pipeline, ensuring safe rollouts, immediate verification, and reliable rollback.

We focus on common device plugins like NVIDIA GPU, Intel FPGA, and generic sample plugins, demonstrating how to observe cluster state, apply changes in a controlled manner, and verify functionality. Whether you run on-prem or in cloud GPU instances, the principles remain the same.

By the end, you will have a concrete workflow: starting from version inventory to automated deployment, health checks, failure detection, and rollback strategies, all backed by command-line examples and configuration snippets.

Version and Environment Inventory

Before automating anything, you must know your current landscape. Device plugins are version-sensitive: a mismatch between the plugin, Kubernetes API, container runtime, and hardware drivers can cause silent failures. Begin by gathering read-only information about your cluster and nodes.

Start with cluster-level details:

kubectl version --short
kubectl get nodes -o wide
kubectl get pods -n kube-system -l app.kubernetes.io/component=device-plugin

Expected output example:

Client Version: v1.27.3
Server Version: v1.27.3
NAME       STATUS   ROLES           AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
gpu-node1  Ready    <none>          30d   v1.27.3   10.10.0.4     <none>        Ubuntu 22.04.3 LTS   5.15.0-91-generic   containerd://1.7.2
gpu-node2  Ready    <none>          30d   v1.27.3   10.10.0.5     <none>        Ubuntu 22.04.3 LTS   5.15.0-91-generic   containerd://1.7.2

For each node hosting device plugins, check the kubelet configuration and device plugin directory:

kubectl describe node gpu-node1 | grep -A5 "Capacity:"
ls -la /var/lib/kubelet/device-plugins/   # on the node

The kubelet's device plugin registration path defaults to /var/lib/kubelet/device-plugins/, but this can vary. Verify the --device-plugin-enabled flag and the registered resources:

kubectl describe node gpu-node1 | grep -A20 "Allocated resources:"

For NVIDIA GPUs, confirm driver and runtime versions:

# On node
nvidia-smi
nvidia-container-cli info

Example output:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05   Driver Version: 535.104.05   CUDA Version: 12.2     |
|-------------------------------+----------------------+----------------------+
...

Check if the device plugin daemonset is already running:

kubectl get ds -n kube-system | grep nvidia
kubectl get pods -n kube-system -l name=nvidia-device-plugin-ds

Record all findings in your inventory sheet. This observation step protects against incompatible upgrades.

Safe Configuration Path

The safest way to introduce changes is gradually: first in a staging environment, then a canary node, and finally rollout to all nodes. Device plugins run as DaemonSets, so changes affect all matching nodes. Control this via node selectors and tolerations.

Common device plugin manifests include a DaemonSet with a hostPath mount for /var/lib/kubelet/device-plugins. Here is a minimal example for the NVIDIA device plugin:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nvidia-device-plugin-daemonset
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: nvidia-device-plugin-ds
  template:
    metadata:
      labels:
        name: nvidia-device-plugin-ds
    spec:
      priorityClassName: system-node-critical
      tolerations:
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule
      containers:
      - name: nvidia-device-plugin-ctr
        image: nvcr.io/nvidia/k8s-device-plugin:v0.14.3
        env:
        - name: FAIL_ON_INIT_ERROR
          value: "false"
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop: ["ALL"]
        volumeMounts:
        - name: device-plugin
          mountPath: /var/lib/kubelet/device-plugins
      volumes:
      - name: device-plugin
        hostPath:
          path: /var/lib/kubelet/device-plugins

First, deploy Node Feature Discovery (NFD) to automatically label GPU-enabled nodes. NFD discovers hardware features and applies labels such as feature.node.kubernetes.io/pci-10de.present=true to nodes with NVIDIA GPUs, removing the need for manual kubectl label commands.

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/node-feature-discovery/master/deployment/base/nfd.yaml
kubectl get nodes -l feature.node.kubernetes.io/pci-10de.present=true

To apply the device plugin to a single canary node, patch the DaemonSet to target that node by hostname:

kubectl patch ds nvidia-device-plugin-daemonset -n kube-system -p '{"spec":{"template":{"spec":{"nodeSelector":{"kubernetes.io/hostname":"gpu-node1"}}}}}'

Verify the plugin registers on that node only:

kubectl describe node gpu-node1 | grep nvidia.com/gpu

Expected output:

nvidia.com/gpu:     1
nvidia.com/gpu:     1

If successful, promote to all GPU nodes by patching the DaemonSet nodeSelector to the NFD-generated label:

kubectl patch ds nvidia-device-plugin-daemonset -n kube-system -p '{"spec":{"template":{"spec":{"nodeSelector":{"feature.node.kubernetes.io/pci-10de.present":"true"}}}}}'

Keep change sets small and reversible.

Verification and Diagnostics

After deployment, verify not just that pods are running, but that the plugin advertised resources correctly and that workloads can consume them.

First, confirm the daemonset pods are healthy:

kubectl get pods -n kube-system -l name=nvidia-device-plugin-ds -o wide
kubectl rollout status ds/nvidia-device-plugin-daemonset -n kube-system

Check logs for registration success:

kubectl logs -n kube-system -l name=nvidia-device-plugin-ds --tail=50

Look for lines like:

I0912 10:00:01.234567       1 main.go:154] Starting FS watcher.
I0912 10:00:01.234567       1 main.go:161] Starting OS watcher.
I0912 10:00:01.234567       1 main.go:176] Starting to serve on /var/lib/kubelet/device-plugins/nvidia-gpu.sock
I0912 10:00:01.234567       1 main.go:183] Registered device plugin with Kubelet

To test actual resource allocation, create a pod requesting a GPU:

apiVersion: v1
kind: Pod
metadata:
  name: gpu-test
spec:
  restartPolicy: Never
  containers:
  - name: cuda-vector-add
    image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda11.7.1-ubi8
    resources:
      limits:
        nvidia.com/gpu: 1

Apply and check logs:

kubectl apply -f gpu-test.yaml
kubectl logs gpu-test

Expected output:

[Vector addition of 50000 elements]
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
Copy output data from the CUDA device to the host memory
Test PASSED
Done

If the pod fails to schedule, inspect events:

kubectl describe pod gpu-test | grep -A10 Events

Watch for messages like:

0/2 nodes are available: 2 Insufficient nvidia.com/gpu.

This indicates the plugin did not register resources or the node selector excluded all nodes.

Failure Modes and Recovery

Device plugin failures can be subtle. Common scenarios include:

1. Registration socket not created: The plugin container runs but cannot create the Unix socket due to permission issues or wrong mount path.

Diagnosis:

   kubectl logs -n kube-system -l name=nvidia-device-plugin-ds | grep -i error

Example error:

   E0912 10:05:00.000000       1 main.go:123] failed to create device plugin socket: mkdir /var/lib/kubelet/device-plugins: permission denied

Recovery: Ensure the container runs as root or with appropriate security context, and the hostPath directory has correct permissions (typically 0755).

1. Kubelet restart disconnects plugins: When kubelet restarts, plugins must re-register. If the plugin does not handle reconnection, resources disappear until pod restart.

Diagnosis:

   kubectl get node gpu-node1 -o json | jq '.status.capacity'

Compare capacity before and after kubelet restart. If nvidia.com/gpu is missing, plugin may not have re-registered.

Recovery: Restart the plugin daemonset.

   kubectl rollout restart ds/nvidia-device-plugin-daemonset -n kube-system

1. Driver or runtime update breaks compatibility: After a node kernel or driver update, the plugin may fail to initialize.

Diagnosis: Check plugin logs for FAIL_ON_INIT_ERROR messages. If you set FAIL_ON_INIT_ERROR: false, the plugin may run but not register. Set it to true to crash and make the failure visible.

Recovery: Roll back the driver or update the plugin to a compatible version.

1. Resource leak or stale socket: Old device plugin sockets can persist and confuse kubelet.

Diagnosis:

   ls -la /var/lib/kubelet/device-plugins/

If you see duplicate sockets or outdated ones, clean them manually after stopping the plugin.

Recovery:

   systemctl stop kubelet
   rm -f /var/lib/kubelet/device-plugins/*.sock
   systemctl start kubelet
   kubectl delete pod -n kube-system -l name=nvidia-device-plugin-ds

For rollbacks in a CI/CD context, maintain versioned manifests and use kubectl apply with the previous version. For instance, if you upgraded from v0.14.2 to v0.14.3 and face issues, revert:

kubectl set image ds/nvidia-device-plugin-daemonset -n kube-system nvidia-device-plugin-ctr=nvcr.io/nvidia/k8s-device-plugin:v0.14.2

Monitor rolling update status:

kubectl rollout status ds/nvidia-device-plugin-daemonset -n kube-system

Operations Checklist

Use this checklist before and after applying device plugin changes in production. Assign a single accountable owner for each step and specify review frequency.

Each item should be logged in your CI/CD system with a timestamp and outcome. If any check fails, halt the pipeline and trigger the rollback procedure.

Common Pitfalls and How to Avoid Them

Despite best efforts, mistakes happen. Here are frequent pitfalls in device plugin automation:

Ignoring Node Heterogeneity

Why it happens: Not all nodes have the same hardware or drivers. Applying a plugin uniformly can cause failures on incompatible nodes.

How to avoid: Use NFD-discovered labels and node selectors to target only nodes with the specific hardware. Maintain separate DaemonSets for different hardware types if needed.

Recovery: If a plugin crashes on a node, isolate it by cordoning the node and deleting the plugin pod. Example:

kubectl cordon gpu-node2
kubectl delete pod -n kube-system -l name=nvidia-device-plugin-ds --field-selector spec.nodeName=gpu-node2

Then fix the driver or replace hardware.

Misconfigured Mount Paths

Why it happens: Custom kubelet configurations may use a different device plugin directory. The default is /var/lib/kubelet/device-plugins, but some clusters use /var/lib/kubelet/device-plugins/kubelet.sock or a custom path.

How to avoid: Check kubelet flags on nodes:

ps aux | grep kubelet | grep device-plugins

Look for --device-plugin-path or similar. Ensure your DaemonSet mounts match.

Recovery: Update the DaemonSet volume mount to the correct path and restart pods.

Not Handling Plugin Downtime

Why it happens: Rolling updates of device plugins may briefly remove resources from nodes, causing pending pods. If workloads are sensitive, this can cause disruptions.

How to avoid: Use maxUnavailable: 0 in DaemonSet update strategy (if supported) or perform updates during maintenance windows. For critical workloads, consider node draining before plugin update.

Recovery: If pods get stuck, restart them after plugin is healthy.

Overlooking Security Contexts

Why it happens: Device plugins often need privileged access or specific capabilities. Running with too many permissions is risky, but too restrictive can break registration.

How to avoid: Follow least privilege. For NVIDIA plugin, drop all capabilities and set allowPrivilegeEscalation: false as shown earlier. Avoid running as root if possible, but some plugins require it.

Recovery: If plugin fails due to security context, adjust and redeploy, then test with a GPU pod.

Conclusion

Automating Kubernetes device plugins in CI/CD requires careful observation, controlled changes, and rigorous verification. By following the structured approach outlined—from inventory and safe configuration to diagnostics and rollback—you can reduce downtime and ensure hardware resources are reliably available to your workloads.

Start by implementing the Operations Checklist in your pipeline for the next device plugin update. Assign owners, run the canary deployment, and verify with a sample workload. Over time, you will build confidence and catch issues before they affect production.

Remember: device plugins are a critical bridge between hardware and Kubernetes. Treat their lifecycle with the same rigor as any other infrastructure component.