Kubernetes Persistent Volume Common Errors and Fixes: A Practical Troubleshooting Guide

Intro

When Kubernetes workloads lose access to persistent data, the impact can be severe: application crashes, data corruption, or even complete service outages. Persistent Volumes (PVs) and Persistent Volume Claims (PVCs) are abstractions that decouple storage from pods, but they also introduce a layer of complexity with many potential failure points. This guide helps developers, DevOps consultants, and technical startup teams diagnose and fix the most common Kubernetes Persistent Volume errors using practical, reproducible steps.

We focus on operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document recovery paths. Every fix includes the exact command or manifest needed, the expected output, and the signal that confirms the issue is resolved. The examples assume a working Kubernetes cluster with kubectl configured, and we note any cloud-specific variations.

Version and Environment Inventory

Before troubleshooting any Persistent Volume issue, confirm your environment. PV behavior has changed across Kubernetes releases, especially around storage classes, volume expansion, and CSI drivers. Run these read-only commands and record the outputs:

kubectl version --short
kubectl get nodes -o wide
kubectl get sc   # storage classes
kubectl get pv,pvc -A

Example output:

Client Version: v1.27.3
Server Version: v1.27.3
NAME                 STATUS   AGE
ip-10-0-1-23         Ready    45d
NAME                 PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
standard (default)   kubernetes.io/aws-ebs   Delete          Immediate           true                   45d
NAME                     CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM             STORAGECLASS   REASON   AGE
pvc-abc123              10Gi       RWO            Delete           Bound    default/myclaim   standard                30d

Key items to verify:

• The cluster version and the CSI driver or in-tree plugin versions. Mismatched versions can cause mount failures.
• The storage classes available and their provisioners. If your PVC references a storage class that does not exist, the PVC remains Pending.
• Existing PVs and PVCs and their status. A PVC in Lost state indicates the underlying PV is unavailable.

For cloud environments (AWS, GCP, Azure), ensure the required cloud controller manager and CSI driver pods are running:

kubectl get pods -n kube-system | grep -E 'ebs|csi|cloud-controller'

All should be Running and Ready. In a self-managed cluster, verify that the storage provisioner (e.g., NFS, Ceph, local) is installed and healthy.

Safe Configuration Path

The most common PV errors stem from misconfigured PVCs, StorageClasses, or node selectors. Always start with a minimal manifest and increase complexity gradually to isolate the problem.

Example: PVC Pending due to missing StorageClass

Given this PVC:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: myclaim
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  storageClassName: fast-ssd

If no StorageClass named fast-ssd exists, the PVC stays Pending. Check with:

kubectl get pvc myclaim
kubectl describe pvc myclaim

Look for events:

Warning  ProvisioningFailed  5s (x3 over 15s)  persistentvolume-controller  storageclass.storage.k8s.io "fast-ssd" not found

Fix: Create the StorageClass or change the PVC to use an existing one. Here is a minimal local-provisioner StorageClass for on-prem clusters:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

After applying, the PVC still remains Pending until a pod uses it because of WaitForFirstConsumer. This is expected. Deploy a pod that references the PVC, then it should bind.

Safe rollback example

If a configuration change breaks existing workloads, revert immediately. For instance, if you changed a StorageClass parameter and PVCs fail to provision, restore the previous manifest:

kubectl apply -f previous-storageclass.yaml
kubectl get pvc -w

Always keep version-controlled manifests and avoid editing live resources directly with kubectl edit for storage-related objects.

Verification and Diagnostics

Suppose a pod using a PVC is stuck in ContainerCreating or CrashLoopBackOff. Use these commands to pinpoint the volume issue.

Step 1: Inspect pod events

kubectl describe pod mypod

Typical volume errors:

• MountVolume.SetUp failed for volume "pvc-..." : mount failed
• Unable to attach or mount volumes: timed out waiting for the condition
• FailedMount with details about the driver or device.

Step 2: Check PVC status

kubectl get pvc

If status is Bound, the PV is attached. If Pending, follow the storage class and provisioner logs.

Step 3: View CSI driver logs (if applicable)

kubectl logs -n kube-system deployment/ebs-csi-controller -c csi-provisioner --tail=50
kubectl logs -n kube-system daemonset/ebs-csi-node -c ebs-csi-node --tail=50

For in-tree plugins, check kube-controller-manager and kubelet logs.

Step 4: Test with a debug pod

Mount the same PVC into a simple pod (e.g., busybox) and try writing a file:

kubectl run -it --rm debug --image=busybox --restart=Never -- sh
# Inside pod:
mount | grep /mnt
# If the PVC is mounted at /mnt, test write:
echo test > /mnt/testfile
ls -l /mnt

If the write fails, it could be a permissions issue or a read-only mount. Check the PV's persistentVolumeReclaimPolicy and the mount options in your pod spec.

Common Pitfalls and How to Avoid Them

Pitfall 1: PVC stuck in Pending due to no matching PV

When using static provisioning, a PVC will only bind if there is an available PV with matching capacity, access modes, and storage class (if specified). Many users expect automatic provisioning, but without a dynamic provisioner, nothing happens.

Why it happens: The cluster has no StorageClass with a provisioner, or the PVC explicitly requests a storage class that doesn't exist.

How to avoid/recover: Check kubectl get pv and kubectl get storageclass. If you intended static binding, create a PV manually with matching specs. Example:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: local-pv
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: manual
  hostPath:
    path: /mnt/data

Then reference storageClassName: manual in the PVC. The PVC should bind quickly.

Pitfall 2: PVC stuck in Terminating

When deleting a PVC, it may hang in Terminating state if the pod using it is still running, or if the finalizer can't complete.

Why it happens: The PVC has a kubernetes.io/pvc-protection finalizer, and the system waits for no pods to be using it. Sometimes the pod is already deleted but the finalizer remains due to a kubelet issue.

How to avoid/recover: First, ensure no pods are using the PVC:

kubectl get pods -A -o json | jq '.items[] | select(.spec.volumes[]?.persistentVolumeClaim.claimName=="myclaim") | .metadata.name'

If none, you can force delete by removing the finalizer:

kubectl patch pvc myclaim -p '{"metadata":{"finalizers":null}}'

Be careful: this bypasses safety checks and can lead to data loss if the volume is still attached.

Pitfall 3: Pod stuck due to node affinity and volume zone mismatch

On cloud platforms, a PV is created in a specific availability zone. If the pod is scheduled to a node in a different zone, the pod fails to mount.

Why it happens: The StorageClass may use volumeBindingMode: Immediate, causing the PV to be created in the zone of the first available node, but the pod later gets scheduled elsewhere. Or the pod has node selectors that conflict with the PV's zone.

How to avoid/recover: Use volumeBindingMode: WaitForFirstConsumer in the StorageClass so the PV is created in the same zone as the pod. Additionally, if your pod needs to run on specific nodes, ensure those nodes are in the same zone as the PV, or use zone-aware scheduling with topologySpreadConstraints.

To check a node's zone:

kubectl get node <node-name> -o jsonpath='{.metadata.labels.topology\.kubernetes\.io/zone}'

And the PV's zone (if applicable):

kubectl get pv <pv-name> -o jsonpath='{.spec.nodeAffinity}'

Failure Modes and Recovery

This section covers severe failure scenarios and recovery steps.

PV Status Lost

A PV can become Lost when the underlying storage is deleted or unreachable. Check:

kubectl get pv
kubectl describe pv <pv-name>

Recovery: If the storage is recoverable (e.g., an EBS volume that still exists but is detached), re-attach it manually via cloud console or recreate the PV with the same volume ID. For dynamic PVs, the reclaim policy Delete means the PV is deleted automatically when PVC is deleted; if the PVC was deleted accidentally, the data is gone unless you have backups. To prevent this, set persistentVolumeReclaimPolicy: Retain for critical data, then manually clean up later.

PVC Bound to Wrong PV

If a PVC binds to a smaller or wrong PV, you need to unbind. Delete the PVC, edit the PV's claimRef to remove the binding, then recreate the PVC. However, this is risky and can cause data loss. Better approach: use labels and selectors in the PVC to target specific PVs if using static provisioning.

File System Corruption or Read-Only Mount

If a pod reports read-only file system when trying to write, check the pod's securityContext and the volume's mount options. The PV may be mounted read-only due to an error. Get pod details:

kubectl describe pod <pod>

Look for readOnly: true in the volume definition. If not, check the persistent volume's mode. Also, verify that the filesystem is not corrupted by running fsck (if accessible) or by mounting the volume to a rescue pod.

Operations Checklist

Use this checklist before and after making changes to Persistent Volumes.

Before changes:

• [ ] List all PVs and PVCs and note their status: kubectl get pv,pvc -A
• [ ] Identify which pods use the volumes: kubectl get pods -A -o json | jq '.items[] | select(.spec.volumes[]?.persistentVolumeClaim.claimName!="") | .metadata.name'
• [ ] Check storage class configuration and provisioner health.
• [ ] Note the reclaim policy of each PV: Retain or Delete.
• [ ] Ensure you have a backup or snapshot of critical volumes.
• [ ] If using cloud volumes, note the volume IDs and zones.
• [ ] Confirm you have permissions to modify PV/PVC and storage classes.
• [ ] For any deletion, first ensure no active pods are mounting the volume.
• [ ] Test changes in a staging environment if possible.

After changes:

• [ ] Verify PVC status becomes Bound (if creating new).
• [ ] Confirm pods restart successfully and can read/write.
• [ ] Run a write test: kubectl exec <pod> -- sh -c "echo test > /data/testfile && cat /data/testfile"
• [ ] Monitor events for any FailedMount or FailedAttachVolume warnings.
• [ ] Check storage provisioner logs for errors.
• [ ] Document the change and any new PVC/PV names.

Ownership and review cadence: Assign a single storage administrator or platform engineer (e.g., "Priya Shah, Engineering Lead") to approve any changes to StorageClasses, PV reclaim policies, or CSI driver upgrades. Review the PV/PVC inventory and health weekly in a short ops sync, focusing on any Pending, Lost, or Terminating resources older than 24 hours.

Conclusion

Kubernetes Persistent Volume errors can be daunting, but a systematic approach with the right commands and understanding of the underlying abstractions resolves most issues quickly. This guide provided an inventory of essential diagnostics, safe configuration practices, common pitfalls, and recovery steps. Remember to always observe first, change one variable at a time, and verify with concrete tests.

For further learning, explore the official Kubernetes documentation on persistent volumes, storage classes, and CSI drivers. Practice these steps in a test cluster to build confidence. A reliable workflow makes failure visible, protects sensitive data, limits blast radius, and defines recovery before an incident forces your hand.