## Intro When Kubernetes workloads lose access to persistent data, the impact can be severe: application crashes, data corruption, or even complete service outages. Persistent Volumes (PVs) and Persistent Volume Claims (PVCs) are abstractions that decouple storage from pods, but they also introduce a layer of complexity with many potential failure points. This guide helps developers, DevOps consultants, and technical startup teams diagnose and fix the most common Kubernetes Persistent Volume errors using practical, reproducible steps. We focus on operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document recovery paths. Every fix includes the exact command or manifest needed, the expected output, and the signal that confirms the issue is resolved. The examples assume a working Kubernetes cluster with `kubectl` configured, and we note any cloud-specific variations. ## Version and Environment Inventory Before troubleshooting any Persistent Volume issue, confirm your environment. PV behavior has changed across Kubernetes releases, especially around storage classes, volume expansion, and CSI drivers. Run these read-only commands and record the outputs: ```bash kubectl version --short kubectl get nodes -o wide kubectl get sc # storage classes kubectl get pv,pvc -A ``` Example output: ``` Client Version: v1.27.3 Server Version: v1.27.3 NAME STATUS AGE ip-10-0-1-23 Ready 45d NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE standard (default) kubernetes.io/aws-ebs Delete Immediate true 45d NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-abc123 10Gi RWO Delete Bound default/myclaim standard 30d ``` Key items to verify: - The cluster version and the CSI driver or in-tree plugin versions. Mismatched versions can cause mount failures. - The storage classes available and their provisioners. If your PVC references a storage class that does not exist, the PVC remains Pending. - Existing PVs and PVCs and their status. A PVC in `Lost` state indicates the underlying PV is unavailable. For cloud environments (AWS, GCP, Azure), ensure the required cloud controller manager and CSI driver pods are running: ```bash kubectl get pods -n kube-system | grep -E 'ebs|csi|cloud-controller' ``` All should be Running and Ready. In a self-managed cluster, verify that the storage provisioner (e.g., NFS, Ceph, local) is installed and healthy. ## Safe Configuration Path The most common PV errors stem from misconfigured PVCs, StorageClasses, or node selectors. Always start with a minimal manifest and increase complexity gradually to isolate the problem. ### Example: PVC Pending due to missing StorageClass Given this PVC: ```yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: myclaim spec: accessModes: - ReadWriteOnce resources: requests: storage: 5Gi storageClassName: fast-ssd ``` If no StorageClass named `fast-ssd` exists, the PVC stays Pending. Check with: ```bash kubectl get pvc myclaim kubectl describe pvc myclaim ``` Look for events: ``` Warning ProvisioningFailed 5s (x3 over 15s) persistentvolume-controller storageclass.storage.k8s.io "fast-ssd" not found ``` Fix: Create the StorageClass or change the PVC to use an existing one. Here is a minimal local-provisioner StorageClass for on-prem clusters: ```yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: fast-ssd provisioner: kubernetes.io/no-provisioner volumeBindingMode: WaitForFirstConsumer ``` After applying, the PVC still remains Pending until a pod uses it because of `WaitForFirstConsumer`. This is expected. Deploy a pod that references the PVC, then it should bind. ### Safe rollback example If a configuration change breaks existing workloads, revert immediately. For instance, if you changed a StorageClass parameter and PVCs fail to provision, restore the previous manifest: ```bash kubectl apply -f previous-storageclass.yaml kubectl get pvc -w ``` Always keep version-controlled manifests and avoid editing live resources directly with `kubectl edit` for storage-related objects. ## Verification and Diagnostics Suppose a pod using a PVC is stuck in `ContainerCreating` or `CrashLoopBackOff`. Use these commands to pinpoint the volume issue. ### Step 1: Inspect pod events ```bash kubectl describe pod mypod ``` Typical volume errors: - `MountVolume.SetUp failed for volume "pvc-..." : mount failed` - `Unable to attach or mount volumes: timed out waiting for the condition` - `FailedMount` with details about the driver or device. ### Step 2: Check PVC status ```bash kubectl get pvc ``` If status is `Bound`, the PV is attached. If `Pending`, follow the storage class and provisioner logs. ### Step 3: View CSI driver logs (if applicable) ```bash kubectl logs -n kube-system deployment/ebs-csi-controller -c csi-provisioner --tail=50 kubectl logs -n kube-system daemonset/ebs-csi-node -c ebs-csi-node --tail=50 ``` For in-tree plugins, check kube-controller-manager and kubelet logs. ### Step 4: Test with a debug pod Mount the same PVC into a simple pod (e.g., busybox) and try writing a file: ```bash kubectl run -it --rm debug --image=busybox --restart=Never -- sh # Inside pod: mount | grep /mnt # If the PVC is mounted at /mnt, test write: echo test > /mnt/testfile ls -l /mnt ``` If the write fails, it could be a permissions issue or a read-only mount. Check the PV's `persistentVolumeReclaimPolicy` and the mount options in your pod spec. ## Common Pitfalls and How to Avoid Them ### Pitfall 1: PVC stuck in Pending due to no matching PV When using static provisioning, a PVC will only bind if there is an available PV with matching capacity, access modes, and storage class (if specified). Many users expect automatic provisioning, but without a dynamic provisioner, nothing happens. **Why it happens:** The cluster has no StorageClass with a provisioner, or the PVC explicitly requests a storage class that doesn't exist. **How to avoid/recover:** Check `kubectl get pv` and `kubectl get storageclass`. If you intended static binding, create a PV manually with matching specs. Example: ```yaml apiVersion: v1 kind: PersistentVolume metadata: name: local-pv spec: capacity: storage: 5Gi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain storageClassName: manual hostPath: path: /mnt/data ``` Then reference `storageClassName: manual` in the PVC. The PVC should bind quickly. ### Pitfall 2: PVC stuck in Terminating When deleting a PVC, it may hang in Terminating state if the pod using it is still running, or if the finalizer can't complete. **Why it happens:** The PVC has a `kubernetes.io/pvc-protection` finalizer, and the system waits for no pods to be using it. Sometimes the pod is already deleted but the finalizer remains due to a kubelet issue. **How to avoid/recover:** First, ensure no pods are using the PVC: ```bash kubectl get pods -A -o json | jq '.items[] | select(.spec.volumes[]?.persistentVolumeClaim.claimName=="myclaim") | .metadata.name' ``` If none, you can force delete by removing the finalizer: ```bash kubectl patch pvc myclaim -p '{"metadata":{"finalizers":null}}' ``` Be careful: this bypasses safety checks and can lead to data loss if the volume is still attached. ### Pitfall 3: Pod stuck due to node affinity and volume zone mismatch On cloud platforms, a PV is created in a specific availability zone. If the pod is scheduled to a node in a different zone, the pod fails to mount. **Why it happens:** The StorageClass may use `volumeBindingMode: Immediate`, causing the PV to be created in the zone of the first available node, but the pod later gets scheduled elsewhere. Or the pod has node selectors that conflict with the PV's zone. **How to avoid/recover:** Use `volumeBindingMode: WaitForFirstConsumer` in the StorageClass so the PV is created in the same zone as the pod. Additionally, if your pod needs to run on specific nodes, ensure those nodes are in the same zone as the PV, or use zone-aware scheduling with `topologySpreadConstraints`. To check a node's zone: ```bash kubectl get node -o jsonpath='{.metadata.labels.topology\.kubernetes\.io/zone}' ``` And the PV's zone (if applicable): ```bash kubectl get pv -o jsonpath='{.spec.nodeAffinity}' ``` ## Failure Modes and Recovery This section covers severe failure scenarios and recovery steps. ### PV Status Lost A PV can become `Lost` when the underlying storage is deleted or unreachable. Check: ```bash kubectl get pv kubectl describe pv ``` Recovery: If the storage is recoverable (e.g., an EBS volume that still exists but is detached), re-attach it manually via cloud console or recreate the PV with the same volume ID. For dynamic PVs, the reclaim policy `Delete` means the PV is deleted automatically when PVC is deleted; if the PVC was deleted accidentally, the data is gone unless you have backups. To prevent this, set `persistentVolumeReclaimPolicy: Retain` for critical data, then manually clean up later. ### PVC Bound to Wrong PV If a PVC binds to a smaller or wrong PV, you need to unbind. Delete the PVC, edit the PV's `claimRef` to remove the binding, then recreate the PVC. However, this is risky and can cause data loss. Better approach: use labels and selectors in the PVC to target specific PVs if using static provisioning. ### File System Corruption or Read-Only Mount If a pod reports `read-only file system` when trying to write, check the pod's `securityContext` and the volume's mount options. The PV may be mounted read-only due to an error. Get pod details: ```bash kubectl describe pod ``` Look for `readOnly: true` in the volume definition. If not, check the persistent volume's mode. Also, verify that the filesystem is not corrupted by running `fsck` (if accessible) or by mounting the volume to a rescue pod. ## Operations Checklist Use this checklist before and after making changes to Persistent Volumes. **Before changes:** - [ ] List all PVs and PVCs and note their status: `kubectl get pv,pvc -A` - [ ] Identify which pods use the volumes: `kubectl get pods -A -o json | jq '.items[] | select(.spec.volumes[]?.persistentVolumeClaim.claimName!="") | .metadata.name'` - [ ] Check storage class configuration and provisioner health. - [ ] Note the reclaim policy of each PV: `Retain` or `Delete`. - [ ] Ensure you have a backup or snapshot of critical volumes. - [ ] If using cloud volumes, note the volume IDs and zones. - [ ] Confirm you have permissions to modify PV/PVC and storage classes. - [ ] For any deletion, first ensure no active pods are mounting the volume. - [ ] Test changes in a staging environment if possible. **After changes:** - [ ] Verify PVC status becomes `Bound` (if creating new). - [ ] Confirm pods restart successfully and can read/write. - [ ] Run a write test: `kubectl exec -- sh -c "echo test > /data/testfile && cat /data/testfile"` - [ ] Monitor events for any `FailedMount` or `FailedAttachVolume` warnings. - [ ] Check storage provisioner logs for errors. - [ ] Document the change and any new PVC/PV names. **Ownership and review cadence:** Assign a single storage administrator or platform engineer (e.g., "Priya Shah, Engineering Lead") to approve any changes to StorageClasses, PV reclaim policies, or CSI driver upgrades. Review the PV/PVC inventory and health weekly in a short ops sync, focusing on any `Pending`, `Lost`, or `Terminating` resources older than 24 hours. ## Conclusion Kubernetes Persistent Volume errors can be daunting, but a systematic approach with the right commands and understanding of the underlying abstractions resolves most issues quickly. This guide provided an inventory of essential diagnostics, safe configuration practices, common pitfalls, and recovery steps. Remember to always observe first, change one variable at a time, and verify with concrete tests. For further learning, explore the official Kubernetes documentation on persistent volumes, storage classes, and CSI drivers. Practice these steps in a test cluster to build confidence. A reliable workflow makes failure visible, protects sensitive data, limits blast radius, and defines recovery before an incident forces your hand.