Kubernetes CSI Driver CI/CD: A Practical Automation Guide

Intro

Kubernetes CSI Driver CI/CD automation with practical examples should help operators move from an observed problem to a verified result. This article focuses on Kubernetes CSI Driver CI/CD for developers, DevOps consultants, and technical startup teams. It connects Kubernetes CSI Driver automation, deployment, pipeline, and rollback to concrete commands, expected output, failure signals, and recovery decisions.

The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached. Throughout, we use a consistent example: an AWS EBS CSI driver running in a cluster on AWS, with a StorageClass named ebs-sc and a PVC named data-pvc. Replace these with your environment specifics.

Version and Environment Inventory

Before any CI/CD change, capture the current state of the CSI driver and its dependencies. This inventory prevents upgrades or configuration changes from breaking existing volumes or applications.

Component Identification

Identify the CSI driver name, namespace, version, and the Kubernetes version it supports. For the AWS EBS CSI driver, the controller and node pods typically run in the kube-system namespace. Check the installed version with:

kubectl get deployment ebs-csi-controller -n kube-system -o jsonpath='{.spec.template.spec.containers[0].image}'

Expected output example:

public.ecr.aws/ebs-csi-driver/aws-ebs-csi-driver:v1.35.0

Note the version v1.35.0. The driver release notes will specify minimum Kubernetes versions and compatibility. For example, v1.35.0 requires Kubernetes 1.21 or later.

Read-Only Observation

Run these commands to observe the system without changing anything:

kubectl get pods -n kube-system -l app=ebs-csi-controller -o wide
kubectl get pods -n kube-system -l app=ebs-csi-node -o wide
kubectl get csidriver
kubectl get storageclass ebs-sc -o yaml

Check that all controller and node pods are Running and READY. The csidriver object shows whether the driver is registered; for EBS, it shows ebs.csi.aws.com with attachRequired: true and podInfoOnMount: false. This tells you that the driver requires volume attachment before mounting.

Dependencies and Prerequisites

Verify that required secrets and service accounts exist. For EBS CSI, the controller needs IAM permissions via IRSA or a secret. Check:

kubectl get sa ebs-csi-controller-sa -n kube-system
kubectl get secret aws-secret -n kube-system -o yaml

If using IRSA, confirm the annotation on the service account points to the correct IAM role ARN. Record these before changes, as CI/CD pipelines often overwrite them.

Smallest Justified Change

Once the current state is documented, define the smallest change. For a driver upgrade, this might be updating the image tag in the controller deployment. But before changing, record the current deployment manifest for rollback:

kubectl get deployment ebs-csi-controller -n kube-system -o yaml > ebs-csi-controller-backup-$(date +%Y%m%d-%H%M%S).yaml

This backup is your rollback artifact. Store it on a safe location accessible to your pipeline.

Safe Configuration Path

This section shows how to implement a configuration change safely, using a CI/CD pipeline with a canary deployment strategy.

Step 1: Build and Test the New Driver Configuration

Assume you need to update a driver setting, such as enabling volume expansion or changing the number of controller replicas. In your pipeline, create a new branch and modify the Helm values or YAML manifest. For example, to increase controller replicas from 2 to 3, edit the deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ebs-csi-controller
  namespace: kube-system
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: ebs-plugin
        image: public.ecr.aws/ebs-csi-driver/aws-ebs-csi-driver:v1.35.0
        args:
          - --endpoint=$(CSI_ENDPOINT)
          - --logtostderr
          - --v=4
          - --enable-volume-scheduling=true
          - --enable-volume-resizing=true
          - --enable-volume-snapshots=true

Here --enable-volume-resizing=true allows PVC expansion. Verify the args are supported by the driver version; check the driver docs for flags.

Step 2: Validate Manifests in CI

Before applying to the cluster, run static validation. Use kubeconform or kubectl apply --dry-run=client:

kubectl apply -f ebs-csi-controller.yaml --dry-run=client

Expected output:

deployment.apps/ebs-csi-controller configured (dry run)

This catches syntax errors. Also run kubectl diff -f ebs-csi-controller.yaml to see changes that would be applied.

Step 3: Deploy to a Canary Namespace or Node Group

Instead of rolling out to all nodes at once, use a canary approach. For CSI node plugins, which run as DaemonSets, you can use a node selector to target a subset of nodes. For the controller, you can create a separate deployment with a different name in the same namespace, but that can conflict. Better: use a Helm release with a staging namespace. For simplicity, we deploy to a test namespace csi-test:

kubectl create namespace csi-test
helm upgrade --install ebs-csi-driver-test aws-ebs-csi-driver \
  --namespace csi-test \
  --set controller.replicaCount=1 \
  --set node.enable=false

This deploys only the controller component in test, not the node plugin. Run integration tests against a test PVC in that namespace:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-pvc
  namespace: csi-test
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: ebs-sc
  resources:
    requests:
      storage: 1Gi

Apply and wait for binding:

kubectl apply -f test-pvc.yaml
kubectl get pvc test-pvc -n csi-test --watch

Expected output after a few seconds:

NAME       STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
test-pvc   Bound    pvc-1234abcd-5678-efgh-ijkl-mnopqrstuvwx   1Gi        RWO            ebs-sc         10s

If it stays Pending, check events:

kubectl describe pvc test-pvc -n csi-test

Step 4: Promote to Production

After test passes, promote the change to the production namespace (kube-system or your dedicated one). Use your CD tool (Argo CD, Flux, Jenkins). For example, with kubectl set image:

kubectl set image deployment/ebs-csi-controller -n kube-system ebs-plugin=public.ecr.aws/ebs-csi-driver/aws-ebs-csi-driver:v1.35.0

Check rollout status:

kubectl rollout status deployment/ebs-csi-controller -n kube-system

Expected output:

deployment "ebs-csi-controller" successfully rolled out

For the node plugin (DaemonSet), use kubectl rollout status daemonset/ebs-csi-node -n kube-system.

Step 5: Verify Persistent Volumes Still Work

After the rollout, create or check an existing PVC. Use a placeholder secret, never real credentials, in pipeline logs. If using AWS, ensure IAM role permissions are unchanged. Verify by creating a test pod that mounts the volume and writes a file.

Verification and Diagnostics

After deploying or changing the CSI driver, verify that storage operations work end-to-end.

Pod and Driver Health

Run the following:

kubectl get pods -n kube-system -l app=ebs-csi-controller
kubectl get pods -n kube-system -l app=ebs-csi-node
kubectl logs deployment/ebs-csi-controller -n kube-system --tail=50

Look for errors like failed to provision volume, unauthorized, or context deadline exceeded. If controller logs show IAM errors, check the service account IAM role binding.

Test Volume Provisioning

Create a PVC and a pod that uses it. Use a simple busybox pod:

apiVersion: v1
kind: Pod
metadata:
  name: test-ebs-pod
spec:
  containers:
  - name: app
    image: busybox
    command: ["/bin/sh"]
    args: ["-c", "echo hello > /data/test.txt && sleep 3600"]
    volumeMounts:
    - name: data
      mountPath: /data
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: test-pvc

Apply and check:

kubectl apply -f test-pod.yaml
kubectl get pod test-ebs-pod

If the pod is Running, exec to verify the file:

kubectl exec test-ebs-pod -- cat /data/test.txt

Expected output: hello

If the pod is stuck in ContainerCreating, run kubectl describe pod test-ebs-pod and look for events like FailedAttachVolume or FailedMount. Check the node plugin logs on the relevant node:

kubectl logs -n kube-system daemonset/ebs-csi-node -c ebs-plugin --tail=50

CSI Driver Specific Checks

For EBS CSI, verify the CSI driver object exists:

kubectl get csidriver ebs.csi.aws.com -o yaml

Check attachRequired: true, podInfoOnMount: false, and volumeLifecycleModes includes Persistent. If these are incorrect, volumes may not attach or mount.

Use csi-sanity tests if available, or run the driver's own validation. Many CSI drivers include a --check mode or a kubectl csi plugin (from cert-manager csi or similar) to test gRPC endpoints.

Failure Modes and Recovery

CI/CD for CSI drivers must include rollback and recovery procedures for common failure scenarios.

Scenario 1: Driver Upgrade Breaks Existing PVCs

If after upgrading, existing PVCs fail to mount (pods stuck in ContainerCreating with Volume not attached), first check if the driver version is incompatible with the Kubernetes version. The mitigation is to rollback to the previous image.

Use the backup YAML you created earlier or use your git history. For example:

kubectl apply -f ebs-csi-controller-backup-20250101-120000.yaml
kubectl rollout status deployment/ebs-csi-controller -n kube-system --timeout=60s

If the deployment rollback fails, use kubectl rollout undo:

kubectl rollout undo deployment/ebs-csi-controller -n kube-system

Then check pods and PVCs. If volumes were incorrectly modified, you may need to restore from snapshots. Ensure you have volume snapshots enabled and tested.

Scenario 2: Node Plugin DaemonSet Fails on New Nodes

A common issue after adding new nodes is that the CSI node plugin pod is not running on the new node. Check with:

kubectl get pods -n kube-system -l app=ebs-csi-node -o wide | grep <new-node-name>

If missing, verify node taints and tolerations. The DaemonSet must tolerate node taints. Check and update tolerations in the DaemonSet manifest. For example, add a toleration for node.kubernetes.io/not-ready:

tolerations:
- key: node.kubernetes.io/not-ready
  operator: Exists
  effect: NoExecute

Apply and verify. Also check that the node has necessary IAM permissions if using instance profiles.

Scenario 3: StorageClass Parameters Incompatible

If a new StorageClass parameter causes provisioning failure, e.g., setting type: io2 when not available in the region, you will see errors in controller logs:

Failed to create volume: InvalidParameterValue: The parameter type is invalid

Rollback the StorageClass change:

kubectl apply -f storageclass-backup.yaml

Then delete the failing PVC and recreate, or wait for the controller to retry.

Scenario 4: Secrets or IAM Roles Misconfigured

If controller logs show AccessDenied or NoCredentialProviders, check the secret or IRSA configuration. For IRSA, verify the service account annotation and IAM trust policy. For secrets, restore from backup or recreate. Example to check secret keys:

kubectl get secret aws-secret -n kube-system -o jsonpath='{.data.key_id}' | base64 -d

Never print full secret values in CI logs; use masked output.

Recovery Verification

After any recovery, run the verification steps from the previous section. Create a new test PVC and pod to ensure the driver works. Document the incident and update runbooks.

Operations Checklist

Use this checklist before, during, and after any Kubernetes CSI Driver CI/CD change. Each item includes a concrete example.

Pre-Change Checklist

• [ ] Record current driver version and image: kubectl get deployment ebs-csi-controller -n kube-system -o jsonpath='{.spec.template.spec.containers[0].image}' (e.g., v1.34.0)
• [ ] Backup current deployment manifest: kubectl get deployment ebs-csi-controller -n kube-system -o yaml > backup-$(date +%s).yaml
• [ ] Verify cluster version compatibility: kubectl version --short (e.g., Server Version: v1.28.5)
• [ ] Check existing PVCs and StorageClasses: kubectl get pvc --all-namespaces and kubectl get sc
• [ ] Confirm IAM roles or secrets are valid and tested: kubectl get sa ebs-csi-controller-sa -n kube-system -o yaml | grep eks.amazonaws.com/role-arn
• [ ] Define rollback criteria: e.g., "If more than 5% of PVCs fail to bind within 2 minutes, rollback."

During Deployment Checklist

• [ ] Apply change using declarative manifests or Helm, not imperative commands where possible.
• [ ] Monitor rollout: kubectl rollout status deployment/ebs-csi-controller -n kube-system --timeout=120s
• [ ] Watch logs in real time: kubectl logs -f deployment/ebs-csi-controller -n kube-system
• [ ] If using canary, verify test namespace before production.
• [ ] Check for unexpected pod restarts: kubectl get pods -n kube-system -l app=ebs-csi-controller -w

Post-Deployment Verification Checklist

• [ ] Create test PVC and verify it binds: kubectl apply -f test-pvc.yaml && kubectl get pvc test-pvc -n default -> should show Bound
• [ ] Create test pod and write/read data: kubectl exec test-ebs-pod -- cat /data/test.txt -> should return hello
• [ ] Validate existing PVCs are still healthy: kubectl get pvc --all-namespaces | grep -v Bound (should be empty)
• [ ] Check CSI driver object: kubectl get csidriver ebs.csi.aws.com -o yaml
• [ ] Review events for errors: kubectl get events -n kube-system --sort-by=.lastTimestamp | tail -20

Rollback and Recovery Checklist

• [ ] If rollback needed, apply backup YAML: kubectl apply -f backup-<timestamp>.yaml
• [ ] For Helm releases, helm rollback <release> <revision>
• [ ] Verify rollback: kubectl rollout status deployment/ebs-csi-controller -n kube-system
• [ ] Run verification tests again.
• [ ] Update incident documentation and label the failed change in your CI/CD system.

Conclusion

Kubernetes CSI Driver CI/CD automation with practical examples is useful only when each recommendation is version-scoped, observable, and reversible where the technology permits. Copying a command without checking prerequisites and expected output is not an operations procedure.

As a next step, choose one low-risk verification for Kubernetes CSI Driver CI/CD, record the current state, run the documented check, compare the result with the expected signal, and review dependencies such as Storage Class, Persistent Volume, and Persistent Volume Claim.

A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. Implement these practices in your pipeline, and you will deploy CSI driver changes with confidence.