Kubernetes Volumes Upgrade and Migration: A Practical Implementation Guide

Intro

Upgrading and migrating Kubernetes volumes is a high-stakes operation. A single misstep can leave applications unable to start, data inaccessible, or storage resources orphaned. This guide provides a practical, command-driven approach to upgrading and migrating volumes in a Kubernetes cluster, with a focus on safety, observability, and recovery.

You will learn how to:

• Inventory the current volume configuration and storage classes.
• Plan and execute a volume migration with minimal downtime.
• Upgrade storage provisioners or volume plugins.
• Validate that data is intact and applications are healthy.
• Roll back if something goes wrong.

Every step includes real kubectl commands, manifest snippets, and expected outputs. The goal is to give you a repeatable playbook that reduces risk and builds confidence.

This guide is intended for developers, DevOps engineers, and platform teams who already have a working knowledge of Kubernetes core concepts like Pods, Deployments, PersistentVolumes (PVs), PersistentVolumeClaims (PVCs), and StorageClasses. If you are new to these concepts, review the official Kubernetes documentation before proceeding.

We will use a fictional e-commerce application called shop-app as the running example. It consists of a PostgreSQL database (with its data on a PersistentVolume) and a stateless web frontend. All commands assume you have kubectl configured with access to a test cluster first and that you have already backed up critical data.

Version and Environment Inventory

Before touching anything, you must know exactly what you have. This section explains how to gather a complete inventory of your current storage configuration, including the Kubernetes version, StorageClasses, PVs, and PVCs. This information is essential for planning a safe migration and for troubleshooting if problems arise.

Check Kubernetes Version and Storage Capabilities

Start by checking the version of your Kubernetes control plane and nodes. Some storage features, like CSI migration or volume expansion, are version-dependent.

kubectl version --short
kubectl get nodes -o wide

Example output:

Client Version: v1.27.3
Server Version: v1.27.3
NAME       STATUS   ROLES           AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
node-1     Ready    control-plane   10d   v1.27.3   192.168.1.10   <none>        Ubuntu 22.04.2 LTS   5.15.0-76-generic   containerd://1.7.1
node-2     Ready    <none>          10d   v1.27.3   192.168.1.11   <none>        Ubuntu 22.04.2 LTS   5.15.0-76-generic   containerd://1.7.1

Record the server version. If you plan to upgrade volumes that depend on a specific CSI driver, check that driver's compatibility matrix against this version.

List StorageClasses

StorageClasses define the types of storage available in your cluster. You need to know which ones are in use and their provisioners.

kubectl get storageclass

Example output:

NAME                 PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
standard (default)   kubernetes.io/gce-pd    Delete          Immediate           false                  30d
fast                 kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer true                   30d
csi-example          example.csi.driver.io   Retain          Immediate           true                   20d

Note the provisioner (in-tree vs CSI), reclaim policy, and whether volume expansion is allowed. These properties affect migration options.

List PersistentVolumes and Claims

Get a detailed view of all PVs and their binding status.

kubectl get pv -o wide

Example output:

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                      STORAGECLASS   REASON   AGE
pvc-1234abcd-56ef-7890-1234-567890abcdef   10Gi       RWO            Delete           Bound    default/shop-db-data       standard                30d
pvc-5678efgh-90ij-1234-5678-901234ghijkl   5Gi        RWO            Delete           Bound    default/shop-db-backup     fast                    10d

Now list all PVCs across namespaces:

kubectl get pvc --all-namespaces

Example output:

NAMESPACE   NAME             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
default     shop-db-data     Bound    pvc-1234abcd-56ef-7890-1234-567890abcdef   10Gi       RWO            standard       30d
default     shop-db-backup   Bound    pvc-5678efgh-90ij-1234-5678-901234ghijkl   5Gi        RWO            fast           10d

Identify Which Pods Use Which Volumes

Find out which running Pods mount these PVCs. This tells you which workloads will be affected by a migration.

kubectl get pods -o wide --all-namespaces

Then, for a specific Pod, describe it to see volume mounts:

kubectl describe pod shop-db-0 -n default

Look for the Volumes and Mounts sections in the output. You can also query directly with JSONPath:

kubectl get pods -n default -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.volumes[*].persistentVolumeClaim.claimName}{"\n"}{end}'

Example output:

shop-db-0    shop-db-data
shop-web-xyz shop-web-uploads

Record which Pods use each PVC. This is critical for scheduling a maintenance window and for verifying that applications are healthy after the migration.

Prerequisites and Compatibility Check

Before proceeding, ensure you have:

• Backups: Take a snapshot or backup of every volume you plan to migrate. For cloud volumes, use the provider's snapshot feature. For on-premises, use your storage system's backup tools.
• kubectl access with appropriate permissions to get, create, and delete PVs, PVCs, and Pods.
• Enough storage capacity in the target storage class to hold the data plus temporary migration space.
• A test environment that mirrors production as closely as possible.

If you are upgrading a CSI driver, check the driver's release notes for supported Kubernetes versions and upgrade paths. Some CSI drivers can be upgraded in place, while others require a new driver instance and a rolling migration of volumes.

Safe Configuration Path

A safe migration or upgrade follows a controlled path: plan, test, backup, execute in stages, and validate at each stage. This section outlines a general safe path and then applies it to a concrete volume migration scenario from an in-tree provider to a CSI driver.

General Safe Migration Path

1. Backup: Snapshot the source volume or use application-level backups.
2. Test: Perform the migration in a non-production namespace or cluster with a similar configuration.
3. Plan the cutover: Decide whether to use a blue/green or canary approach. For stateful workloads, a blue/green deployment of the application with a new volume is often simpler.
4. Execute: Create the new volume, copy data, switch the application to the new volume, and verify.
5. Clean up: Remove the old volume only after a successful validation period.

Example: Migrating from an In-Tree Provider to a CSI Driver

Many Kubernetes distributions are deprecating in-tree volume plugins in favor of CSI drivers. A common migration is from the kubernetes.io/gce-pd in-tree provisioner to the GCE PD CSI driver. The process involves creating a new StorageClass, migrating PVs, and testing.

Step 1: Install the CSI driver (if not already installed). For GCE PD, you can apply the driver manifests from the official repository. Check that the driver pods are running:

kubectl get pods -n kube-system | grep csi

Expected output (similar):

gce-pd-csi-driver-controller-0   4/4     Running   0          5m
gce-pd-csi-driver-node-xxxxx     2/2     Running   0          5m

Step 2: Create a new StorageClass using the CSI provisioner. Save the following YAML as csi-standard.yaml:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-standard
provisioner: pd.csi.storage.gke.io
parameters:
  type: pd-standard
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

Apply it:

kubectl apply -f csi-standard.yaml

Verify:

kubectl get storageclass csi-standard

Step 3: Migrate an existing volume. The recommended way is to create a new PVC with the new StorageClass and copy data. We will use a simple kubectl cp method for moderate data sizes. For very large volumes, consider using a job with rsync or a tool like Velero.

First, create a new PVC for the database data:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: shop-db-data-csi
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: csi-standard
  resources:
    requests:
      storage: 10Gi

Apply and wait for it to be bound:

kubectl apply -f shop-db-data-csi.yaml
kubectl get pvc shop-db-data-csi

You will see the STATUS become Bound once a PV is provisioned.

Next, you need a temporary Pod to copy data. Since the database is running, you should stop writes to ensure consistency. For a production migration, you would either stop the application or use a database replication method. For this example, we will assume you can temporarily scale down the database StatefulSet.

Scale down the database:

kubectl scale statefulset shop-db --replicas=0

Create a temporary Pod that mounts both the old and new PVCs:

apiVersion: v1
kind: Pod
metadata:
  name: data-migrator
spec:
  containers:
  - name: migrator
    image: busybox
    command: ["/bin/sh", "-c"]
    args:
      - |
        echo "Copying data..."
        cp -a /source/* /destination/
        echo "Copy complete"
        sleep 3600
    volumeMounts:
    - name: source
      mountPath: /source
    - name: destination
      mountPath: /destination
  volumes:
  - name: source
    persistentVolumeClaim:
      claimName: shop-db-data
  - name: destination
    persistentVolumeClaim:
      claimName: shop-db-data-csi
  restartPolicy: Never

Apply and wait for the copy to complete:

kubectl apply -f data-migrator.yaml
kubectl logs data-migrator

Expected output:

Copying data...
Copy complete

Once the copy is done, delete the migrator Pod.

Step 4: Update the application to use the new PVC. For a StatefulSet, you would update the volumeClaimTemplates. However, because PersistentVolumeClaims created from templates are immutable, you cannot simply change the storageClassName. The common approach is to create a new StatefulSet with a different name, or to use a tool like kubectl patch if the PVC allows. For simplicity, here we will assume the database is a single Pod managed by a Deployment.

If it were a Deployment, you could patch the volume claim name:

kubectl patch deployment shop-db --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/volumes/0/persistentVolumeClaim/claimName", "value": "shop-db-data-csi"}]'

Then scale back up:

kubectl scale deployment shop-db --replicas=1

Step 5: Verify the application is running and data is intact. Check logs and run a query.

kubectl logs deployment/shop-db --tail=20
kubectl exec -it deployment/shop-db -- psql -U postgres -c "SELECT count(*) FROM orders;"

Expected output (example):

 count
-------
  1234
(1 row)

If everything works, you can delete the old PVC after a certain monitoring period.

Verification and Diagnostics

After any migration or upgrade, you must verify that the system is functioning correctly. This section provides commands and techniques to diagnose storage issues and confirm data integrity.

Verify Volume Binding and Status

Check that PVCs are bound and PVs are in the expected state.

kubectl get pvc -n default
kubectl get pv

For a specific PVC, describe it to see events and binding details:

kubectl describe pvc shop-db-data-csi

Look for warnings such as FailedBinding or ProvisioningFailed. These indicate problems with the storage provisioner or capacity.

Check Pod Health and Volume Mounts

Ensure the Pod is running and the volume is mounted correctly.

kubectl get pods
kubectl describe pod shop-db-xxxxx

In the describe output, under Volumes, you should see the PVC name and the mount path. Under Conditions, Ready should be True. If the Pod is stuck in ContainerCreating, check events for volume-related errors.

Common errors:

• FailedMount: The volume could not be mounted due to missing device or permission issues.
• FailedAttachVolume: The node could not attach the volume, often due to zone mismatch or driver issues.

Use kubectl logs to see application logs that might indicate data read/write problems.

Test Read/Write Performance

To ensure the new volume performs adequately, you can run a simple I/O test inside the Pod. For example, if the Pod has a shell, write and read a test file:

kubectl exec -it deployment/shop-db -- bash
# Inside container
dd if=/dev/zero of=/var/lib/postgresql/data/testfile bs=1M count=100
dd if=/var/lib/postgresql/data/testfile of=/dev/null bs=1M count=100
rm /var/lib/postgresql/data/testfile

This writes a 100MB file and reads it back. Check for errors or unusually slow performance.

Validate Filesystem Consistency

For database volumes, use the database's built-in consistency checks. For PostgreSQL, you can run pg_dump to a temporary file or use psql to run a few queries. For other applications, compare file counts or checksums before and after migration.

If you have a backup, you can compare the number of files or total size:

kubectl exec -it deployment/shop-db -- du -sh /var/lib/postgresql/data

Compare this to the source volume before deletion.

Monitor Storage Metrics

If your cluster has monitoring (Prometheus, Grafana), look at volume-related metrics such as kubelet_volume_stats_used_bytes, kubelet_volume_stats_capacity_bytes, and volume operation errors. Set up alerts for low disk space or high latency.

Failure Modes and Recovery

Even with careful planning, failures can occur. This section describes common failure scenarios during volume migration and upgrade, along with recovery steps.

Insufficient Capacity in New StorageClass

Failure: The new PVC remains in Pending state because the storage provisioner cannot create a volume of the requested size (e.g., the storage class has a limit or the backend is out of space).

Diagnosis:

kubectl describe pvc shop-db-data-csi

Look for events like:

Warning  ProvisioningFailed  2m (x5 over 5m)  persistentvolume-controller  failed to provision volume with StorageClass "csi-standard": rpc error: code = ResourceExhausted desc = Insufficient capacity

Recovery:

• Check available capacity in your storage backend.
• Reduce the requested size if possible (but only if data fits).
• Use a different StorageClass with more capacity.

Data Copy Incomplete or Corrupted

Failure: The data migrator Pod completes with exit code 0 but the application fails to start or data is missing.

Diagnosis:

• Compare the size of source and destination volumes using du -sh.
• Check logs of the migrator Pod for errors that may have been ignored.
• Run application-level integrity checks (e.g., database consistency).

Recovery:

• If corruption is suspected, do not delete the source volume. Re-run the copy with a more reliable method (e.g., rsync with checksum verification).
• For databases, consider using native backup/restore instead of file copy.

Pod Fails to Mount New Volume

Failure: After switching the PVC, the Pod is stuck in ContainerCreating with FailedMount events.

Diagnosis:

kubectl describe pod shop-db-xxxxx

Look for mount errors:

Warning  FailedMount  2m   kubelet  MountVolume.MountDevice failed for volume "pvc-..." : driver name pd.csi.storage.gke.io not found in the list of registered CSI drivers

Recovery:

• Ensure the CSI driver is installed and running on the node.
• Check that the StorageClass references the correct provisioner.
• If the volume was created by a different driver, you may need to manually set the PV's csi.driver field and restart the kubelet.

Rollback Strategy

If the migration fails and you cannot resolve the issue quickly, roll back to the original volume.

Steps:

1. Scale down the application.
2. Revert the PVC reference in the Pod or StatefulSet to the old PVC name.
3. Scale up the application.
4. Verify that the old volume is still intact and the application works.
5. Investigate the failure before attempting again.

Always keep the old volume until you are confident the new setup is stable. Set a monitoring period (e.g., one week) before deleting the old PVC.

Upgrading CSI Driver Failure

If you are upgrading the CSI driver itself, and the new version fails to start, you can roll back the driver to the previous version by reapplying the old manifests or using a version-controlled deployment method (e.g., Helm rollback).

helm rollback my-csi-driver 1

Then verify that the driver pods are healthy:

kubectl get pods -n kube-system | grep csi

Operations Checklist

Use this checklist before, during, and after a volume migration or upgrade to ensure nothing is missed.

Pre-Migration

• [ ] Back up all critical data from volumes to be migrated.
• [ ] Verify backups are restorable by performing a test restore.
• [ ] Record current Kubernetes version, StorageClasses, PVs, PVCs, and Pod-volume mappings.
• [ ] Identify all applications using the volumes and their downtime tolerance.
• [ ] Check capacity in target storage system.
• [ ] Test the migration process in a staging environment.
• [ ] Schedule a maintenance window and notify stakeholders.

During Migration

• [ ] Scale down applications that write to the source volumes (or stop writes).
• [ ] Create new PVCs with target StorageClass.
• [ ] Start data copy job and monitor progress.
• [ ] Verify data copy completed successfully (size, checksums if possible).
• [ ] Update application manifests to use new PVCs.
• [ ] Scale up applications.
• [ ] Run smoke tests and application-level checks.

Post-Migration

• [ ] Verify Pod health and volume mounts (kubectl get pods, kubectl describe pod).
• [ ] Check application logs for errors.
• [ ] Run database integrity checks or equivalent.
• [ ] Monitor performance metrics for a defined period.
• [ ] Keep old volumes for rollback until validation period ends.
• [ ] Delete old volumes and clean up temporary resources.
• [ ] Document the new configuration and update runbooks.

Emergency Rollback Checklist

• [ ] Scale down affected applications.
• [ ] Revert to old PVC references.
• [ ] Scale up applications.
• [ ] Verify old volumes are intact and applications function.
• [ ] Investigate root cause before retrying.

Conclusion

Upgrading and migrating Kubernetes volumes requires careful planning and execution. By following the structured approach in this guide, you can minimize downtime and data loss risks. The key principles are:

• Inventory first: Know your current storage layout and dependencies.
• Backup everything: Never migrate without a verified backup.
• Test in staging: Rehearse the migration before production.
• Go step by step: Use a safe path with rollback points.
• Verify rigorously: Use commands and checks to confirm data integrity and application health.
• Keep rollback ready: Retain old volumes until the new system proves stable.

With these practices, you can handle volume migrations and upgrades as routine operations rather than high-risk events. Start with a low-risk volume, document your process, and refine it over time. The command examples and checklists in this article provide a solid foundation for building your own playbook tailored to your environment.