## Intro
Kubernetes volumes provide storage that outlives individual containers and pods. Understanding how volumes, PersistentVolumes (PVs), PersistentVolumeClaims (PVCs), and StorageClasses interact is essential for running stateful workloads reliably. This article walks through the architecture with practical examples, commands, and troubleshooting steps to help you move from a problem to a verified solution.
We focus on operational safety: observe before changing, limit the blast radius, avoid hard-coded secrets, verify each step, and document recovery paths. Whether you are a developer, DevOps consultant, or part of a startup team, this guide connects core concepts to real commands, expected outputs, failure signals, and recovery decisions.
## Core Components and Data Flow
Kubernetes volumes are not just disk mounts; they are a multi-component system. The main building blocks are:
- Volume : A directory accessible to containers in a pod, defined in the pod spec. It can be backed by many sources: emptyDir, hostPath, cloud disks, NFS, and more.
- PersistentVolume (PV) : A cluster-level storage resource provisioned by an admin or dynamically via a StorageClass. It has a lifecycle independent of any pod.
- PersistentVolumeClaim (PVC) : A user's request for storage. It specifies size, access modes, and optionally a StorageClass. The control plane binds a PVC to a matching PV.
- StorageClass : Defines the provisioner and parameters for dynamic volume provisioning. It lets users request storage without knowing the underlying infrastructure.
- Container Storage Interface (CSI) : A standard for exposing storage systems to container orchestrators. Most modern drivers are CSI-based.
The data flow for dynamic provisioning is:
- A user creates a PVC with a StorageClass name.
- The StorageClass's provisioner creates a PV that matches the request.
- Kubernetes binds the PVC to the newly created PV.
- The pod using the PVC mounts the PV into its containers via the node's kubelet.
For static provisioning, an admin pre-creates PVs, and PVCs bind to the first matching PV.
## Version and Environment Inventory
Before working with volumes, confirm your Kubernetes version and storage drivers. Use kubectl version --short or kubectl version to get client and server versions. For example:
$ kubectl version --short
Client Version: v1.28.2
Server Version: v1.28.4
Check which CSI drivers are installed in your cluster. Most managed clusters have cloud-specific drivers. You can list CSI drivers with:
kubectl get csidrivers
Example output:
NAME ATTACHREQUIRED PODINFOONMOUNT STORAGECAPACITY TOKENREQUESTS REQUIRESREPUBLISH
pd.csi.storage.gke.io true true true false
Verify the default StorageClass, which is used when a PVC does not specify one:
kubectl get storageclass
You might see:
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
standard (default) kubernetes.io/gce-pd Delete WaitForFirstConsumer true 12d
If no default is set, mark one with:
kubectl patch storageclass standard -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
Always capture current state before changes. For example, record existing PVCs:
kubectl get pvc -n myapp
## Safe Configuration Path
A safe configuration path means applying changes incrementally and verifying each step. Start with a simple pod that uses an emptyDir volume to understand basic mounts.
Create test-volume.yaml :
apiVersion: v1
kind: Pod
metadata:
name: test-volume
spec:
containers:
- name: busybox
image: busybox:1.36
command: ['sh', '-c', 'echo hello > /data/message.txt && sleep 3600']
volumeMounts:
- name: data
mountPath: /data
volumes:
- name: data
emptyDir: {}
Apply and verify:
kubectl apply -f test-volume.yaml
kubectl get pod test-volume
Once running, exec to confirm the write:
kubectl exec test-volume -- cat /data/message.txt
Expected output: hello .
Now move to persistent storage. Create a PVC and a pod that uses it. First, define pvc.yaml :
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: my-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
# storageClassName: standard # optional if default is set
Apply and check status:
kubectl apply -f pvc.yaml
kubectl get pvc my-pvc
If the PVC is pending, it may be due to no matching PV or StorageClass. Describe it for events:
kubectl describe pvc my-pvc
Once bound, create a pod using the PVC in pod-with-pvc.yaml :
apiVersion: v1
kind: Pod
metadata:
name: pod-with-pvc
spec:
containers:
- name: app
image: nginx:1.25
volumeMounts:
- name: data
mountPath: /usr/share/nginx/html
volumes:
- name: data
persistentVolumeClaim:
claimName: my-pvc
Apply and verify:
kubectl apply -f pod-with-pvc.yaml
kubectl get pod pod-with-pvc
After verification, delete the test resources:
kubectl delete pod pod-with-pvc test-volume
kubectl delete pvc my-pvc
Note: because the default reclaim policy for dynamically provisioned volumes is usually Delete , the PV is removed. If you need to retain data, set persistentVolumeReclaimPolicy: Retain on the PV.
## Verification and Diagnostics
When volumes fail, use a systematic approach. Start with read-only observations:
kubectl get pods -o wide
kubectl describe pod
kubectl logs --previous
For example, if a pod is stuck in ContainerCreating , describe it:
kubectl describe pod my-app-7c8b9d6f4-abcde
Look for events like:
Warning FailedAttachVolume 2m42s attachdetach-controller AttachVolume.Attach failed for volume "pvc-123..." : timed out waiting for the condition
This often indicates a problem with the storage backend or node. Check node status:
kubectl get nodes
If the node is NotReady , inspect it:
kubectl describe node
Verify the PVC is bound:
kubectl get pvc
If the PVC is Pending , describe it:
kubectl describe pvc my-pvc
Common reasons:
- No StorageClass provisioner available.
- Requested size exceeds quota.
- Access mode mismatch.
- StorageClass has WaitForFirstConsumer and no pod scheduled.
You can also check PV details:
kubectl get pv
kubectl describe pv
Use kubectl get events --sort-by=.metadata.creationTimestamp to see recent cluster events.
For mounted volumes, test write/read inside the pod:
kubectl exec -- sh -c 'echo test > /data/testfile && cat /data/testfile'
If the pod crashes, use kubectl logs --previous to see previous container logs.
## Failure Modes and Recovery
Understanding common failure modes helps recovery.
1. PVC stuck in Pending
Cause: No matching PV or provisioner not working.
Diagnosis:
kubectl describe pvc my-pvc
If events show no persistent volumes available for this claim and no storage class is set , you may need to create a PV or set a default StorageClass.
Recovery: Create a PV manually if static provisioning is intended, or define a StorageClass and reference it in the PVC.
2. Pod stuck in ContainerCreating with volume errors
Cause: Volume cannot be attached or mounted (e.g., cloud disk in wrong zone, node missing driver).
Diagnosis:
kubectl describe pod
Look for events like FailedMount or FailedAttachVolume .
Recovery:
- If using zonal disks, ensure the pod is scheduled in the same zone as the disk. Use node affinity or WaitForFirstConsumer binding mode.
- For CSI drivers, verify driver pods are running: kubectl get pods -n kube-system | grep csi .
- Check node has the required mount utilities.
3. Data not persisting after pod deletion
Cause: The volume was emptyDir or the PVC reclaim policy is Delete .
Diagnosis: Inspect PV reclaim policy:
kubectl get pv -o jsonpath='{.spec.persistentVolumeReclaimPolicy}'
Recovery: For data retention, set persistentVolumeReclaimPolicy: Retain on the PV. After PVC deletion, the PV remains and can be manually reclaimed.
4. Multi-node access issues
Cause: Using ReadWriteOnce on a volume that needs multiple nodes.
Diagnosis: Check accessModes on PVC and PV:
kubectl get pvc my-pvc -o jsonpath='{.spec.accessModes}'
Recovery: Use a volume that supports ReadWriteMany (e.g., NFS, Azure Files, Amazon EFS) and update accessModes accordingly. Note that you cannot change accessModes of a bound PVC; you need to recreate it.
5. Disk full or quota exceeded
Cause: Application writes too much data.
Diagnosis: Check disk usage inside pod: kubectl exec -- df -h /mountpoint .
Recovery: Expand the volume if allowVolumeExpansion is true in StorageClass:
kubectl patch pvc my-pvc -p '{"spec":{"resources":{"requests":{"storage":"5Gi"}}}}'
Then verify with kubectl get pvc .
Always document recovery steps and test them in a non-production environment before an incident.
## Operations Checklist
Use this checklist for safe volume operations:
- [ ] Check Kubernetes version and storage drivers: kubectl version and kubectl get csidrivers .
- [ ] Verify default StorageClass: kubectl get storageclass .
- [ ] Record existing PVC/PV state: kubectl get pvc,pv -A .
- [ ] Test volume mounting with a simple pod (emptyDir) before moving to persistent volumes.
- [ ] Create PVC with appropriate accessMode and size, and specify StorageClass explicitly to avoid ambiguity.
- [ ] For production, use WaitForFirstConsumer binding mode to ensure pod scheduling is considered.
- [ ] Apply pod config and verify status: kubectl get pod .
- [ ] Check events and logs on failure: kubectl describe pod , kubectl logs --previous .
- [ ] Test data persistence: write a file, delete pod, recreate, and read file.
- [ ] Set reclaim policy to Retain if data must survive PVC deletion.
- [ ] Monitor volume capacity and expand if needed.
- [ ] Before deleting PVC, ensure no pod is using it: kubectl get pvc -o wide .
- [ ] Document recovery runbooks for common failures.
Example of a production PVC definition:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: app-data
namespace: production
spec:
accessModes:
- ReadWriteOnce
storageClassName: fast-ssd
resources:
requests:
storage: 20Gi
And the corresponding StorageClass:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: fast-ssd
provisioner: kubernetes.io/gce-pd
parameters:
type: pd-ssd
reclaimPolicy: Retain
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
Apply both, then create a Deployment that uses the PVC. Always verify the rollout:
kubectl apply -f storageclass.yaml
kubectl apply -f pvc.yaml
kubectl apply -f deployment.yaml
kubectl rollout status deployment/my-app
## Conclusion
Kubernetes volumes architecture is powerful but requires careful configuration and verification. By understanding PVs, PVCs, and StorageClasses, and by following a systematic approach to testing and troubleshooting, you can avoid common pitfalls and ensure your stateful applications run reliably.
Start with a low-risk verification: create a test pod with an emptyDir volume, then progress to persistent storage. Record current state, run documented checks, and compare results with expected outcomes. Always limit changes to the intended resource and document recovery steps before an incident forces a decision.
With the commands and examples in this article, you should be able to diagnose volume issues, recover from failures, and operate Kubernetes storage with confidence.