## Intro Upgrading and migrating Kubernetes volumes is a high-stakes operation. A single misstep can leave applications unable to start, data inaccessible, or storage resources orphaned. This guide provides a practical, command-driven approach to upgrading and migrating volumes in a Kubernetes cluster, with a focus on safety, observability, and recovery. You will learn how to: - Inventory the current volume configuration and storage classes. - Plan and execute a volume migration with minimal downtime. - Upgrade storage provisioners or volume plugins. - Validate that data is intact and applications are healthy. - Roll back if something goes wrong. Every step includes real `kubectl` commands, manifest snippets, and expected outputs. The goal is to give you a repeatable playbook that reduces risk and builds confidence. This guide is intended for developers, DevOps engineers, and platform teams who already have a working knowledge of Kubernetes core concepts like Pods, Deployments, PersistentVolumes (PVs), PersistentVolumeClaims (PVCs), and StorageClasses. If you are new to these concepts, review the official Kubernetes documentation before proceeding. We will use a fictional e-commerce application called `shop-app` as the running example. It consists of a PostgreSQL database (with its data on a PersistentVolume) and a stateless web frontend. All commands assume you have `kubectl` configured with access to a test cluster first and that you have already backed up critical data. ## Version and Environment Inventory Before touching anything, you must know exactly what you have. This section explains how to gather a complete inventory of your current storage configuration, including the Kubernetes version, StorageClasses, PVs, and PVCs. This information is essential for planning a safe migration and for troubleshooting if problems arise. ### Check Kubernetes Version and Storage Capabilities Start by checking the version of your Kubernetes control plane and nodes. Some storage features, like CSI migration or volume expansion, are version-dependent. ```bash kubectl version --short kubectl get nodes -o wide ``` Example output: ```text Client Version: v1.27.3 Server Version: v1.27.3 NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME node-1 Ready control-plane 10d v1.27.3 192.168.1.10 Ubuntu 22.04.2 LTS 5.15.0-76-generic containerd://1.7.1 node-2 Ready 10d v1.27.3 192.168.1.11 Ubuntu 22.04.2 LTS 5.15.0-76-generic containerd://1.7.1 ``` Record the server version. If you plan to upgrade volumes that depend on a specific CSI driver, check that driver's compatibility matrix against this version. ### List StorageClasses StorageClasses define the types of storage available in your cluster. You need to know which ones are in use and their provisioners. ```bash kubectl get storageclass ``` Example output: ```text NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE standard (default) kubernetes.io/gce-pd Delete Immediate false 30d fast kubernetes.io/aws-ebs Delete WaitForFirstConsumer true 30d csi-example example.csi.driver.io Retain Immediate true 20d ``` Note the provisioner (in-tree vs CSI), reclaim policy, and whether volume expansion is allowed. These properties affect migration options. ### List PersistentVolumes and Claims Get a detailed view of all PVs and their binding status. ```bash kubectl get pv -o wide ``` Example output: ```text NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-1234abcd-56ef-7890-1234-567890abcdef 10Gi RWO Delete Bound default/shop-db-data standard 30d pvc-5678efgh-90ij-1234-5678-901234ghijkl 5Gi RWO Delete Bound default/shop-db-backup fast 10d ``` Now list all PVCs across namespaces: ```bash kubectl get pvc --all-namespaces ``` Example output: ```text NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE default shop-db-data Bound pvc-1234abcd-56ef-7890-1234-567890abcdef 10Gi RWO standard 30d default shop-db-backup Bound pvc-5678efgh-90ij-1234-5678-901234ghijkl 5Gi RWO fast 10d ``` ### Identify Which Pods Use Which Volumes Find out which running Pods mount these PVCs. This tells you which workloads will be affected by a migration. ```bash kubectl get pods -o wide --all-namespaces ``` Then, for a specific Pod, describe it to see volume mounts: ```bash kubectl describe pod shop-db-0 -n default ``` Look for the `Volumes` and `Mounts` sections in the output. You can also query directly with JSONPath: ```bash kubectl get pods -n default -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.volumes[*].persistentVolumeClaim.claimName}{"\n"}{end}' ``` Example output: ```text shop-db-0 shop-db-data shop-web-xyz shop-web-uploads ``` Record which Pods use each PVC. This is critical for scheduling a maintenance window and for verifying that applications are healthy after the migration. ### Prerequisites and Compatibility Check Before proceeding, ensure you have: - Backups: Take a snapshot or backup of every volume you plan to migrate. For cloud volumes, use the provider's snapshot feature. For on-premises, use your storage system's backup tools. - `kubectl` access with appropriate permissions to get, create, and delete PVs, PVCs, and Pods. - Enough storage capacity in the target storage class to hold the data plus temporary migration space. - A test environment that mirrors production as closely as possible. If you are upgrading a CSI driver, check the driver's release notes for supported Kubernetes versions and upgrade paths. Some CSI drivers can be upgraded in place, while others require a new driver instance and a rolling migration of volumes. ## Safe Configuration Path A safe migration or upgrade follows a controlled path: plan, test, backup, execute in stages, and validate at each stage. This section outlines a general safe path and then applies it to a concrete volume migration scenario from an in-tree provider to a CSI driver. ### General Safe Migration Path 1. **Backup**: Snapshot the source volume or use application-level backups. 2. **Test**: Perform the migration in a non-production namespace or cluster with a similar configuration. 3. **Plan the cutover**: Decide whether to use a blue/green or canary approach. For stateful workloads, a blue/green deployment of the application with a new volume is often simpler. 4. **Execute**: Create the new volume, copy data, switch the application to the new volume, and verify. 5. **Clean up**: Remove the old volume only after a successful validation period. ### Example: Migrating from an In-Tree Provider to a CSI Driver Many Kubernetes distributions are deprecating in-tree volume plugins in favor of CSI drivers. A common migration is from the `kubernetes.io/gce-pd` in-tree provisioner to the GCE PD CSI driver. The process involves creating a new StorageClass, migrating PVs, and testing. **Step 1: Install the CSI driver** (if not already installed). For GCE PD, you can apply the driver manifests from the official repository. Check that the driver pods are running: ```bash kubectl get pods -n kube-system | grep csi ``` Expected output (similar): ```text gce-pd-csi-driver-controller-0 4/4 Running 0 5m gce-pd-csi-driver-node-xxxxx 2/2 Running 0 5m ``` **Step 2: Create a new StorageClass** using the CSI provisioner. Save the following YAML as `csi-standard.yaml`: ```yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: csi-standard provisioner: pd.csi.storage.gke.io parameters: type: pd-standard reclaimPolicy: Delete volumeBindingMode: WaitForFirstConsumer allowVolumeExpansion: true ``` Apply it: ```bash kubectl apply -f csi-standard.yaml ``` Verify: ```bash kubectl get storageclass csi-standard ``` **Step 3: Migrate an existing volume**. The recommended way is to create a new PVC with the new StorageClass and copy data. We will use a simple `kubectl cp` method for moderate data sizes. For very large volumes, consider using a job with `rsync` or a tool like Velero. First, create a new PVC for the database data: ```yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: shop-db-data-csi spec: accessModes: - ReadWriteOnce storageClassName: csi-standard resources: requests: storage: 10Gi ``` Apply and wait for it to be bound: ```bash kubectl apply -f shop-db-data-csi.yaml kubectl get pvc shop-db-data-csi ``` You will see the STATUS become `Bound` once a PV is provisioned. Next, you need a temporary Pod to copy data. Since the database is running, you should stop writes to ensure consistency. For a production migration, you would either stop the application or use a database replication method. For this example, we will assume you can temporarily scale down the database StatefulSet. Scale down the database: ```bash kubectl scale statefulset shop-db --replicas=0 ``` Create a temporary Pod that mounts both the old and new PVCs: ```yaml apiVersion: v1 kind: Pod metadata: name: data-migrator spec: containers: - name: migrator image: busybox command: ["/bin/sh", "-c"] args: - | echo "Copying data..." cp -a /source/* /destination/ echo "Copy complete" sleep 3600 volumeMounts: - name: source mountPath: /source - name: destination mountPath: /destination volumes: - name: source persistentVolumeClaim: claimName: shop-db-data - name: destination persistentVolumeClaim: claimName: shop-db-data-csi restartPolicy: Never ``` Apply and wait for the copy to complete: ```bash kubectl apply -f data-migrator.yaml kubectl logs data-migrator ``` Expected output: ```text Copying data... Copy complete ``` Once the copy is done, delete the migrator Pod. **Step 4: Update the application to use the new PVC**. For a StatefulSet, you would update the volumeClaimTemplates. However, because PersistentVolumeClaims created from templates are immutable, you cannot simply change the storageClassName. The common approach is to create a new StatefulSet with a different name, or to use a tool like `kubectl patch` if the PVC allows. For simplicity, here we will assume the database is a single Pod managed by a Deployment. If it were a Deployment, you could patch the volume claim name: ```bash kubectl patch deployment shop-db --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/volumes/0/persistentVolumeClaim/claimName", "value": "shop-db-data-csi"}]' ``` Then scale back up: ```bash kubectl scale deployment shop-db --replicas=1 ``` **Step 5: Verify** the application is running and data is intact. Check logs and run a query. ```bash kubectl logs deployment/shop-db --tail=20 kubectl exec -it deployment/shop-db -- psql -U postgres -c "SELECT count(*) FROM orders;" ``` Expected output (example): ```text count ------- 1234 (1 row) ``` If everything works, you can delete the old PVC after a certain monitoring period. ## Verification and Diagnostics After any migration or upgrade, you must verify that the system is functioning correctly. This section provides commands and techniques to diagnose storage issues and confirm data integrity. ### Verify Volume Binding and Status Check that PVCs are bound and PVs are in the expected state. ```bash kubectl get pvc -n default kubectl get pv ``` For a specific PVC, describe it to see events and binding details: ```bash kubectl describe pvc shop-db-data-csi ``` Look for warnings such as `FailedBinding` or `ProvisioningFailed`. These indicate problems with the storage provisioner or capacity. ### Check Pod Health and Volume Mounts Ensure the Pod is running and the volume is mounted correctly. ```bash kubectl get pods kubectl describe pod shop-db-xxxxx ``` In the describe output, under `Volumes`, you should see the PVC name and the mount path. Under `Conditions`, `Ready` should be `True`. If the Pod is stuck in `ContainerCreating`, check events for volume-related errors. Common errors: - `FailedMount`: The volume could not be mounted due to missing device or permission issues. - `FailedAttachVolume`: The node could not attach the volume, often due to zone mismatch or driver issues. Use `kubectl logs` to see application logs that might indicate data read/write problems. ### Test Read/Write Performance To ensure the new volume performs adequately, you can run a simple I/O test inside the Pod. For example, if the Pod has a shell, write and read a test file: ```bash kubectl exec -it deployment/shop-db -- bash # Inside container dd if=/dev/zero of=/var/lib/postgresql/data/testfile bs=1M count=100 dd if=/var/lib/postgresql/data/testfile of=/dev/null bs=1M count=100 rm /var/lib/postgresql/data/testfile ``` This writes a 100MB file and reads it back. Check for errors or unusually slow performance. ### Validate Filesystem Consistency For database volumes, use the database's built-in consistency checks. For PostgreSQL, you can run `pg_dump` to a temporary file or use `psql` to run a few queries. For other applications, compare file counts or checksums before and after migration. If you have a backup, you can compare the number of files or total size: ```bash kubectl exec -it deployment/shop-db -- du -sh /var/lib/postgresql/data ``` Compare this to the source volume before deletion. ### Monitor Storage Metrics If your cluster has monitoring (Prometheus, Grafana), look at volume-related metrics such as `kubelet_volume_stats_used_bytes`, `kubelet_volume_stats_capacity_bytes`, and volume operation errors. Set up alerts for low disk space or high latency. ## Failure Modes and Recovery Even with careful planning, failures can occur. This section describes common failure scenarios during volume migration and upgrade, along with recovery steps. ### Insufficient Capacity in New StorageClass Failure: The new PVC remains in `Pending` state because the storage provisioner cannot create a volume of the requested size (e.g., the storage class has a limit or the backend is out of space). Diagnosis: ```bash kubectl describe pvc shop-db-data-csi ``` Look for events like: ```text Warning ProvisioningFailed 2m (x5 over 5m) persistentvolume-controller failed to provision volume with StorageClass "csi-standard": rpc error: code = ResourceExhausted desc = Insufficient capacity ``` Recovery: - Check available capacity in your storage backend. - Reduce the requested size if possible (but only if data fits). - Use a different StorageClass with more capacity. ### Data Copy Incomplete or Corrupted Failure: The data migrator Pod completes with exit code 0 but the application fails to start or data is missing. Diagnosis: - Compare the size of source and destination volumes using `du -sh`. - Check logs of the migrator Pod for errors that may have been ignored. - Run application-level integrity checks (e.g., database consistency). Recovery: - If corruption is suspected, do not delete the source volume. Re-run the copy with a more reliable method (e.g., `rsync` with checksum verification). - For databases, consider using native backup/restore instead of file copy. ### Pod Fails to Mount New Volume Failure: After switching the PVC, the Pod is stuck in `ContainerCreating` with `FailedMount` events. Diagnosis: ```bash kubectl describe pod shop-db-xxxxx ``` Look for mount errors: ```text Warning FailedMount 2m kubelet MountVolume.MountDevice failed for volume "pvc-..." : driver name pd.csi.storage.gke.io not found in the list of registered CSI drivers ``` Recovery: - Ensure the CSI driver is installed and running on the node. - Check that the StorageClass references the correct provisioner. - If the volume was created by a different driver, you may need to manually set the PV's `csi.driver` field and restart the kubelet. ### Rollback Strategy If the migration fails and you cannot resolve the issue quickly, roll back to the original volume. Steps: 1. Scale down the application. 2. Revert the PVC reference in the Pod or StatefulSet to the old PVC name. 3. Scale up the application. 4. Verify that the old volume is still intact and the application works. 5. Investigate the failure before attempting again. Always keep the old volume until you are confident the new setup is stable. Set a monitoring period (e.g., one week) before deleting the old PVC. ### Upgrading CSI Driver Failure If you are upgrading the CSI driver itself, and the new version fails to start, you can roll back the driver to the previous version by reapplying the old manifests or using a version-controlled deployment method (e.g., Helm rollback). ```bash helm rollback my-csi-driver 1 ``` Then verify that the driver pods are healthy: ```bash kubectl get pods -n kube-system | grep csi ``` ## Operations Checklist Use this checklist before, during, and after a volume migration or upgrade to ensure nothing is missed. ### Pre-Migration - [ ] Back up all critical data from volumes to be migrated. - [ ] Verify backups are restorable by performing a test restore. - [ ] Record current Kubernetes version, StorageClasses, PVs, PVCs, and Pod-volume mappings. - [ ] Identify all applications using the volumes and their downtime tolerance. - [ ] Check capacity in target storage system. - [ ] Test the migration process in a staging environment. - [ ] Schedule a maintenance window and notify stakeholders. ### During Migration - [ ] Scale down applications that write to the source volumes (or stop writes). - [ ] Create new PVCs with target StorageClass. - [ ] Start data copy job and monitor progress. - [ ] Verify data copy completed successfully (size, checksums if possible). - [ ] Update application manifests to use new PVCs. - [ ] Scale up applications. - [ ] Run smoke tests and application-level checks. ### Post-Migration - [ ] Verify Pod health and volume mounts (`kubectl get pods`, `kubectl describe pod`). - [ ] Check application logs for errors. - [ ] Run database integrity checks or equivalent. - [ ] Monitor performance metrics for a defined period. - [ ] Keep old volumes for rollback until validation period ends. - [ ] Delete old volumes and clean up temporary resources. - [ ] Document the new configuration and update runbooks. ### Emergency Rollback Checklist - [ ] Scale down affected applications. - [ ] Revert to old PVC references. - [ ] Scale up applications. - [ ] Verify old volumes are intact and applications function. - [ ] Investigate root cause before retrying. ## Conclusion Upgrading and migrating Kubernetes volumes requires careful planning and execution. By following the structured approach in this guide, you can minimize downtime and data loss risks. The key principles are: - **Inventory first**: Know your current storage layout and dependencies. - **Backup everything**: Never migrate without a verified backup. - **Test in staging**: Rehearse the migration before production. - **Go step by step**: Use a safe path with rollback points. - **Verify rigorously**: Use commands and checks to confirm data integrity and application health. - **Keep rollback ready**: Retain old volumes until the new system proves stable. With these practices, you can handle volume migrations and upgrades as routine operations rather than high-risk events. Start with a low-risk volume, document your process, and refine it over time. The command examples and checklists in this article provide a solid foundation for building your own playbook tailored to your environment.