--previous
Look for permission denied errors like:
AccessDeniedException: User: arn:aws:sts::123456789012:assumed-role/eks-node-role/i-0abc123 is not authorized to perform: ec2:AttachVolume
Fix the IAM role or instance profile used by the node.
### Validate File Permissions and Ownership
A volume mounts, but the application cannot write to it. Exec into the pod:
kubectl exec -it mysql-0 -n production -- ls -ld /var/lib/mysql
Output may show:
drwxr-xr-x 3 root root 4096 Jul 10 12:34 /var/lib/mysql
If the container runs as non-root (e.g., user 999), it lacks write permission. Fix by either using a securityContext with fsGroup in the pod spec, or by using an init container to adjust ownership. For example, add to the pod spec:
securityContext:
fsGroup: 999
Apply the pod change and confirm ownership:
kubectl exec -it mysql-0 -- ls -ld /var/lib/mysql
Should now show group write and correct group ID.
### Monitor Capacity and Expansion
Check current PVC capacity and usage:
kubectl get pvc mysql-data -n production -o wide
To see actual filesystem usage, exec into the pod:
kubectl exec -it mysql-0 -n production -- df -h /var/lib/mysql
Example:
Filesystem Size Used Avail Use% Mounted on
/dev/nvme1n1 20G 14G 5.2G 73% /var/lib/mysql
If the storage class supports expansion ( allowVolumeExpansion: true ), you can increase the PVC size:
kubectl patch pvc mysql-data -n production -p '{"spec":{"resources":{"requests":{"storage":"30Gi"}}}}'
Watch the PVC condition:
kubectl get pvc mysql-data -n production -w
Expected event:
Normal Resizing 10s external-resizer External resizer is resizing volume pvc-...
If expansion is not enabled, you will see an error like:
Warning ExternalExpanding 5s volume_expand Ignoring the PVC: didn't find a plugin capable of expanding the volume
In that case, you must provision a new larger PVC and migrate data.
## Failure Modes and Recovery
Several distinct failure modes affect PVCs. Knowing the symptoms and recovery path for each reduces downtime.
### Failure Mode 1: PVC Stuck in Pending - No Storage Class Match
Symptom: kubectl get pvc shows Pending indefinitely. Describe shows no events or a ProvisioningFailed with "storageclass not found".
Recovery:
- Verify storage class exists: kubectl get storageclass
- If missing, create it or edit PVC's storageClassName field.
- To edit: kubectl patch pvc -p '{"spec":{"storageClassName":"standard"}}'
- Note: You can only change storage class if the PVC is not yet bound and the new class exists.
- If the storage class exists but no provisioner is running, check the CSI driver pods in kube-system .
### Failure Mode 2: PVC Bound but Pod Fails to Mount
Symptom: Pod describe shows FailedMount with Unable to attach or mount volumes . The PVC status is Bound.
Recovery:
- Check if the pod is scheduled on a node in the same availability zone as the volume (for zonal storage like EBS). If not, the pod may need to be rescheduled or volume recreated in the correct zone.
- Verify the CSI driver node plugin is running on the node: kubectl get pods -n kube-system -o wide | grep csi-node
- Inspect CSI controller logs for attach errors.
- Check node's volume attachment limits.
Example check:
kubectl get volumeattachments
If the volume attachment is stuck in attached: false or not present, force delete the attachment after confirming the pod is no longer using it:
kubectl patch volumeattachment -p '{"metadata":{"finalizers":[]}}' --type=merge
kubectl delete volumeattachment
This should only be done when the attached pod is terminated or the cluster is in a known state.
### Failure Mode 3: Data Corruption or Lost Filesystem
Symptom: Application logs show input/output errors, or files are missing.
Recovery:
- Take a filesystem snapshot if the storage provider supports it. For AWS EBS, you can create a snapshot from the PV's underlying volume. Identify the PV:
kubectl get pv pvc- -o yaml
Note the volumeID under spec.csi.volumeHandle , e.g., vol-0abcd1234ef567890 .
- Create a snapshot via cloud CLI or use Kubernetes VolumeSnapshot if CSI driver supports it.
- Restore from snapshot into a new PVC and point the pod to it.
- For minor corruption, attempt fsck on a detached volume (not possible on attached volumes).
### Failure Mode 4: PVC Deleted Accidentally
Symptom: kubectl get pvc shows the PVC is gone; pods error with MountVolume.SetUp failed for volume "data" failed to get PVC .
Recovery:
- If the reclaim policy on the PV was Retain , the underlying volume still exists. Find the PV:
kubectl get pv | grep Released
It may show status Released with a claim reference to the deleted PVC.
- Create a new PVC with the same volumeName to rebind:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mysql-data # same name as before if possible
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
volumeName: pv-name # specify the existing PV
- If reclaim policy was Delete , the volume may have been deprovisioned. Restore from backup or snapshot.
### Failure Mode 5: PVC on a Node That Becomes NotReady
Symptom: Pods using the PVC are stuck terminating or cannot start because the node is NotReady.
Recovery:
- Check node status: kubectl get nodes
- If the node is down, cordon it:
kubectl cordon
- Delete the pod that was on that node, allowing it to be rescheduled elsewhere. For stateful workloads, ensure the PVC is not attached to the dead node before deleting.
- If the volume was attached to the dead node, force detach by deleting the VolumeAttachment object as described above.
- Once the node returns or is replaced, verify the pod starts on a new node and the PVC remains Bound.
## Operations Checklist
Use this checklist before and after any PVC change to ensure safety and verification.
### Pre-Change Checklist
- [ ] Record current PVC, PV, and pod status with kubectl get pvc,pv,pod -n -o wide .
- [ ] Capture describe output for failing objects: kubectl describe pvc > pvc-desc-before.txt .
- [ ] Export YAML snapshots for rollback: kubectl get pvc -o yaml > pvc-before.yaml .
- [ ] Identify the storage class and its provisioner: kubectl get storageclass -o yaml .
- [ ] Confirm the CSI driver or in-tree provisioner pods are healthy: kubectl get pods -n kube-system | grep csi .
- [ ] Check available capacity in the storage backend if known (e.g., AWS EBS volume limits in region/AZ).
- [ ] Determine impact: which pods use this PVC? kubectl get pods --all-namespaces -o json | jq '.items[] | select(.spec.volumes[].persistentVolumeClaim.claimName=="") | .metadata.name'
- [ ] Prepare a rollback plan: if the change fails, what object do you need to restore? Save copies of any object to be modified.
### Post-Change Verification
- [ ] Confirm PVC status is Bound and no new events are firing: kubectl get pvc -w for a few minutes.
- [ ] Check pod mount points: kubectl exec -- df -h
- [ ] Test application write access: kubectl exec -- touch /testfile && kubectl exec -- rm /testfile
- [ ] Validate data integrity if applicable (e.g., run a database query or check a file hash).
- [ ] Monitor logs for volume-related errors over the next 5-10 minutes: kubectl logs --since=10m | grep -i volume
- [ ] Document the change and verification results in the incident or change ticket.
### Example Verification Session
Suppose you increased PVC size from 20Gi to 30Gi. After applying the patch:
kubectl get pvc mysql-data -n production
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
mysql-data Bound pvc-1234abcd-5678-ef90-1234-567890abcdef 30Gi RWO fast 6d
Then inside the pod:
kubectl exec -it mysql-0 -n production -- df -h /var/lib/mysql
Filesystem Size Used Avail Use% Mounted on
/dev/nvme1n1 30G 14G 15G 47% /var/lib/mysql
That confirms the expansion was successful from the application's perspective.
## Conclusion
Troubleshooting Kubernetes Persistent Volume Claims is a structured process: identify the exact failure, observe the current state, change one variable at a time, and verify before moving on. This guide has walked through environment inventory, configuration validation, diagnostic commands, common failure modes with recovery steps, and an operational checklist.
The next time you face a Pending PVC or a mount error, start with kubectl get pvc --all-namespaces and kubectl describe pvc . Use the events as your primary signal. Remember that storage is a stateful dependency; changes can have lasting consequences. Always have a rollback plan and document your verification.
For a deeper dive into storage classes, PVC expansion, and CSI driver specifics, explore the official Kubernetes storage documentation. But the commands and patterns here should cover the majority of production incidents. Keep this guide handy, and may your volumes stay bound and healthy.