Kubernetes Persistent Volume Claim Troubleshooting: A Practical Field Guide

Intro

A Kubernetes Persistent Volume Claim (PVC) is the primary way developers and operators request storage without coupling a workload to a specific storage backend. When a PVC enters Pending, a pod cannot mount its volume, or data is inaccessible, every minute of uncertainty adds risk to stateful services.

This field guide targets developers, DevOps consultants, and technical startup teams who manage production Kubernetes clusters. It covers Kubernetes PVC troubleshooting steps, common errors, log inspection, command-line diagnostics, and recovery strategies with concrete examples. Each scenario includes commands, expected output, and failure signals so you can move from symptom to confirmed resolution without guesswork.

The operational principle is simple: observe before changing. Verify the current state, limit the blast radius of any modification, use placeholders instead of secrets, and document the rollback path. Each section builds a version-aware, reproducible procedure you can run against your own cluster.

Version and Environment Inventory

Before touching a PVC, establish what you are running. This inventory prevents mismatched assumptions about API versions, default storage classes, and provider-specific behavior.

Identify the Kubernetes Version and Storage Drivers

Run:

kubectl version --short

Expected output includes both client and server versions, for example:

Client Version: v1.27.1
Server Version: v1.26.3

If the server version is older than 1.20, the PersistentVolumeClaim API may still be v1beta1; for 1.20 and later it is stable v1. This matters because older clusters may lack features like volume expansion or CSI migration.

Next, list the configured storage classes:

kubectl get storageclass

Example output:

NAME                 PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
standard (default)   kubernetes.io/gce-pd   Delete          Immediate           false                  21d
fast                 ebs.csi.aws.com        Delete          WaitForFirstConsumer true                10d

Note the PROVISIONER column. In-tree provisioners like kubernetes.io/gce-pd are being replaced by CSI drivers such as pd.csi.storage.gke.io or ebs.csi.aws.com. If a PVC references a storage class with an in-tree provisioner that has been migrated or removed in your cluster version, provisioning will fail.

Check PVC Events and Status

Run a read-only observation:

kubectl get pvc --all-namespaces

Sample output:

NAMESPACE   NAME     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
default    data-x   Bound    pvc-9f3c2d1e-0a4b-4d8e-b1e2-123456789abc   10Gi       RWO            standard       5d
default    cache-y  Pending                                                                    fast           2m

A Pending PVC is the most common starting point for troubleshooting. Describe the PVC to surface provisioning errors:

kubectl describe pvc cache-y

Look for the Events section. Typical failure messages:

Events:
  Type     Reason                Age               From                         Message
  ----     ------                ----              ----                         -------
  Warning  ProvisioningFailed    10s               persistentvolume-controller  storageclass.storage.k8s.io "fast" not found

That specific error means the PVC references a storage class that does not exist in the cluster. Verify with kubectl get storageclass fast; if it returns NotFound, create the storage class or change the PVC to use an existing one.

Capture Environment Baseline

Record the following before making changes:

kubectl get pvc -o yaml > pvc-before.yaml
kubectl get pv -o yaml > pv-before.yaml
kubectl get events --sort-by=.lastTimestamp > events-before.log

This snapshot gives you a rollback reference and a timeline of recent events.

Safe Configuration Path

When a PVC is misconfigured, fix the smallest possible object first. Applying a broad set of changes without verification often obscures the root cause and expands the blast radius.

Validate a PVC Manifest Before Applying

Use --dry-run=server to test against the API server without persisting the object:

kubectl apply -f pvc.yaml --dry-run=server

If the manifest contains a deprecated API version, the server rejects it with a message like:

error: unable to recognize "pvc.yaml": no matches for kind "PersistentVolumeClaim" in version "v1beta1"

Fix the apiVersion to v1 and retry.

Example PVC Manifest

Here is a minimal PVC for a CSI-backed storage class (AWS EBS in this example):

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-data
  namespace: production
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: fast
  resources:
    requests:
      storage: 20Gi

Apply it:

kubectl apply -f pvc.yaml

Then watch the status:

kubectl get pvc mysql-data -w

Expected progression:

NAME         STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
mysql-data   Pending                                                                fast           0s
mysql-data   Bound     pvc-1234abcd-5678-ef90-1234-567890abcdef   20Gi       RWO            fast           5s

If it stays Pending beyond the normal provisioning time (usually 30-60 seconds for cloud volumes), move to the next step.

Isolate Storage Class Mismatches

A PVC may be Pending because the storage class requires WaitForFirstConsumer binding mode. In that case, the PVC remains Pending until a pod using it is scheduled. Check the storage class:

kubectl get storageclass fast -o yaml | grep volumeBindingMode

volumeBindingMode: WaitForFirstConsumer

If you expected immediate binding, either change the storage class or create a pod that references the PVC. For example:

apiVersion: v1
kind: Pod
metadata:
  name: mysql-test
spec:
  containers:
  - name: mysql
    image: mysql:8.0
    volumeMounts:
    - name: data
      mountPath: /var/lib/mysql
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: mysql-data

After applying the pod, re-check the PVC status. It should bind within a few seconds if the storage backend is healthy.

Verification and Diagnostics

Once the PVC is Bound, verify that applications can actually use it. Mount failures, permission errors, and capacity issues often surface only at runtime.

Confirm Volume Mount in Pod

List pods using the PVC:

kubectl get pods -n production -l app=mysql -o wide

Describe the pod and look for volume errors:

kubectl describe pod mysql-0 -n production

Common volume-related events:

Events:
  Type     Reason       Age   From               Message
  ----     ------       ----  ----               -------
  Warning  FailedMount  2m    kubelet            Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[data default-token-xxxxx]: timed out waiting for the condition

This often indicates the CSI driver cannot attach the volume to the node, perhaps due to IAM permissions, zone mismatch, or node capacity. Check the driver pods:

kubectl get pods -n kube-system | grep csi

If the CSI controller or node plugin pods are crash-looping, inspect their logs:

kubectl logs -n kube-system ebs-csi-controller-<pod-id> --previous

Look for permission denied errors like:

AccessDeniedException: User: arn:aws:sts::123456789012:assumed-role/eks-node-role/i-0abc123 is not authorized to perform: ec2:AttachVolume

Fix the IAM role or instance profile used by the node.

Validate File Permissions and Ownership

A volume mounts, but the application cannot write to it. Exec into the pod:

kubectl exec -it mysql-0 -n production -- ls -ld /var/lib/mysql

Output may show:

drwxr-xr-x 3 root root 4096 Jul 10 12:34 /var/lib/mysql

If the container runs as non-root (e.g., user 999), it lacks write permission. Fix by either using a securityContext with fsGroup in the pod spec, or by using an init container to adjust ownership. For example, add to the pod spec:

securityContext:
  fsGroup: 999

Apply the pod change and confirm ownership:

kubectl exec -it mysql-0 -- ls -ld /var/lib/mysql

Should now show group write and correct group ID.

Monitor Capacity and Expansion

Check current PVC capacity and usage:

kubectl get pvc mysql-data -n production -o wide

To see actual filesystem usage, exec into the pod:

kubectl exec -it mysql-0 -n production -- df -h /var/lib/mysql

Example:

Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme1n1    20G   14G  5.2G  73% /var/lib/mysql

If the storage class supports expansion (allowVolumeExpansion: true), you can increase the PVC size:

kubectl patch pvc mysql-data -n production -p '{"spec":{"resources":{"requests":{"storage":"30Gi"}}}}'

Watch the PVC condition:

kubectl get pvc mysql-data -n production -w

Expected event:

Normal  Resizing  10s  external-resizer  External resizer is resizing volume pvc-...

If expansion is not enabled, you will see an error like:

Warning  ExternalExpanding  5s  volume_expand  Ignoring the PVC: didn't find a plugin capable of expanding the volume

In that case, you must provision a new larger PVC and migrate data.

Failure Modes and Recovery

Several distinct failure modes affect PVCs. Knowing the symptoms and recovery path for each reduces downtime.

Failure Mode 1: PVC Stuck in Pending - No Storage Class Match

Symptom: kubectl get pvc shows Pending indefinitely. Describe shows no events or a ProvisioningFailed with "storageclass not found".

Recovery:

1. Verify storage class exists: kubectl get storageclass
2. If missing, create it or edit PVC's storageClassName field.

• To edit: kubectl patch pvc <name> -p '{"spec":{"storageClassName":"standard"}}'
• Note: You can only change storage class if the PVC is not yet bound and the new class exists.

1. If the storage class exists but no provisioner is running, check the CSI driver pods in kube-system.

Failure Mode 2: PVC Bound but Pod Fails to Mount

Symptom: Pod describe shows FailedMount with Unable to attach or mount volumes. The PVC status is Bound.

Recovery:

• Check if the pod is scheduled on a node in the same availability zone as the volume (for zonal storage like EBS). If not, the pod may need to be rescheduled or volume recreated in the correct zone.
• Verify the CSI driver node plugin is running on the node: kubectl get pods -n kube-system -o wide | grep csi-node
• Inspect CSI controller logs for attach errors.
• Check node's volume attachment limits.

Example check:

kubectl get volumeattachments

If the volume attachment is stuck in attached: false or not present, force delete the attachment after confirming the pod is no longer using it:

kubectl patch volumeattachment <name> -p '{"metadata":{"finalizers":[]}}' --type=merge
kubectl delete volumeattachment <name>

This should only be done when the attached pod is terminated or the cluster is in a known state.

Failure Mode 3: Data Corruption or Lost Filesystem

Symptom: Application logs show input/output errors, or files are missing.

Recovery:

• Take a filesystem snapshot if the storage provider supports it. For AWS EBS, you can create a snapshot from the PV's underlying volume. Identify the PV:

  kubectl get pv pvc-<id> -o yaml

Note the volumeID under spec.csi.volumeHandle, e.g., vol-0abcd1234ef567890.

• Create a snapshot via cloud CLI or use Kubernetes VolumeSnapshot if CSI driver supports it.
• Restore from snapshot into a new PVC and point the pod to it.
• For minor corruption, attempt fsck on a detached volume (not possible on attached volumes).

Failure Mode 4: PVC Deleted Accidentally

Symptom: kubectl get pvc shows the PVC is gone; pods error with MountVolume.SetUp failed for volume "data" failed to get PVC.

Recovery:

• If the reclaim policy on the PV was Retain, the underlying volume still exists. Find the PV:

  kubectl get pv | grep Released

It may show status Released with a claim reference to the deleted PVC.

• Create a new PVC with the same volumeName to rebind:

  apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    name: mysql-data   # same name as before if possible
  spec:
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 20Gi
    volumeName: pv-name   # specify the existing PV

• If reclaim policy was Delete, the volume may have been deprovisioned. Restore from backup or snapshot.

Failure Mode 5: PVC on a Node That Becomes NotReady

Symptom: Pods using the PVC are stuck terminating or cannot start because the node is NotReady.

Recovery:

• Check node status: kubectl get nodes
• If the node is down, cordon it:

  kubectl cordon <node>

• Delete the pod that was on that node, allowing it to be rescheduled elsewhere. For stateful workloads, ensure the PVC is not attached to the dead node before deleting.
• If the volume was attached to the dead node, force detach by deleting the VolumeAttachment object as described above.
• Once the node returns or is replaced, verify the pod starts on a new node and the PVC remains Bound.

Operations Checklist

Use this checklist before and after any PVC change to ensure safety and verification.

Pre-Change Checklist

• [ ] Record current PVC, PV, and pod status with kubectl get pvc,pv,pod -n <namespace> -o wide.
• [ ] Capture describe output for failing objects: kubectl describe pvc <name> > pvc-desc-before.txt.
• [ ] Export YAML snapshots for rollback: kubectl get pvc <name> -o yaml > pvc-before.yaml.
• [ ] Identify the storage class and its provisioner: kubectl get storageclass <name> -o yaml.
• [ ] Confirm the CSI driver or in-tree provisioner pods are healthy: kubectl get pods -n kube-system | grep csi.
• [ ] Check available capacity in the storage backend if known (e.g., AWS EBS volume limits in region/AZ).
• [ ] Determine impact: which pods use this PVC? kubectl get pods --all-namespaces -o json | jq '.items[] | select(.spec.volumes[].persistentVolumeClaim.claimName=="<pvc-name>") | .metadata.name'
• [ ] Prepare a rollback plan: if the change fails, what object do you need to restore? Save copies of any object to be modified.

Post-Change Verification

• [ ] Confirm PVC status is Bound and no new events are firing: kubectl get pvc <name> -w for a few minutes.
• [ ] Check pod mount points: kubectl exec <pod> -- df -h <mount-path>
• [ ] Test application write access: kubectl exec <pod> -- touch <mount-path>/testfile && kubectl exec <pod> -- rm <mount-path>/testfile
• [ ] Validate data integrity if applicable (e.g., run a database query or check a file hash).
• [ ] Monitor logs for volume-related errors over the next 5-10 minutes: kubectl logs <pod> --since=10m | grep -i volume
• [ ] Document the change and verification results in the incident or change ticket.

Example Verification Session

Suppose you increased PVC size from 20Gi to 30Gi. After applying the patch:

kubectl get pvc mysql-data -n production

NAME         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
mysql-data   Bound    pvc-1234abcd-5678-ef90-1234-567890abcdef   30Gi       RWO            fast           6d

Then inside the pod:

kubectl exec -it mysql-0 -n production -- df -h /var/lib/mysql

Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme1n1    30G   14G   15G   47% /var/lib/mysql

That confirms the expansion was successful from the application's perspective.

Conclusion

Troubleshooting Kubernetes Persistent Volume Claims is a structured process: identify the exact failure, observe the current state, change one variable at a time, and verify before moving on. This guide has walked through environment inventory, configuration validation, diagnostic commands, common failure modes with recovery steps, and an operational checklist.

The next time you face a Pending PVC or a mount error, start with kubectl get pvc --all-namespaces and kubectl describe pvc <name>. Use the events as your primary signal. Remember that storage is a stateful dependency; changes can have lasting consequences. Always have a rollback plan and document your verification.

For a deeper dive into storage classes, PVC expansion, and CSI driver specifics, explore the official Kubernetes storage documentation. But the commands and patterns here should cover the majority of production incidents. Keep this guide handy, and may your volumes stay bound and healthy.