## Intro Running containers in production is straightforward, but operating stateful workloads reliably requires a solid grasp of Kubernetes storage primitives. This guide walks through PersistentVolumes (PV), PersistentVolumeClaims (PVC), and StorageClasses with real manifests, verification commands, and the most common failure modes you will encounter when moving from a laptop cluster to a production‑like environment. The target audience includes developers, DevOps consultants, and technical startup teams who need to move from concept to local verification quickly. By the end you will be able to create a PV, request it via a PVC, let a StorageClass provision it dynamically, and confirm that a Pod can read and write data that survives Pod restarts. ## Workflow Overview ### 1. Identify the resources you will create
ResourcePurposeTypical manifest fields
PersistentVolumeCluster‑wide storage object (static provisioning)capacity.storage, accessModes, persistentVolumeReclaimPolicy, hostPath.path or nfs.server
PersistentVolumeClaimNamespaced request for storageresources.requests.storage, accessModes, storageClassName
StorageClassDynamic provisioning templateprovisioner, parameters, reclaimPolicy, volumeBindingMode
Pod / DeploymentConsumer that mounts the PVCvolumes.persistentVolumeClaim.claimName, volumeMounts.mountPath
### 2. Verify each step with kubectl # List all PVs and their status kubectl get pv -o wide # List PVCs in the current namespace kubectl get pvc -o wide # Show StorageClasses available in the cluster kubectl get storageclass # Describe a specific PVC to see binding events kubectl describe pvc my-data-pvc # Describe the bound PV kubectl describe pv # Check Pod events and mount status kubectl describe pod my-app-xyz ### 3. Common failure scenarios and how to spot them Symptom Likely cause Diagnostic command Fix PVC stays Pending No matching StorageClass or volumeBindingMode: WaitForFirstConsumer with no Pod scheduled kubectl describe pvc shows Events: FailedBinding Ensure a StorageClass exists, set storageClassName correctly, or create a Pod that references the PVC PV stays Available but PVC Pending AccessModes mismatch (e.g., PV ReadWriteOnce vs PVC ReadWriteMany ) kubectl get pv,pvc -o custom-columns=NAME:.metadata.name,ACCESS:.spec.accessModes Align accessModes on both objects Pod stuck in ContainerCreating Volume mount permission error or node lacks the provisioned volume kubectl describe pod shows MountVolume.SetUp failed Check node labels, CSI driver logs, and fsGroup in Pod securityContext Data lost after Pod restart persistentVolumeReclaimPolicy: Delete on a statically provisioned PV, or emptyDir used by mistake kubectl get pv -o jsonpath='{.spec.persistentVolumeReclaimPolicy}' Use Retain for static PVs, ensure PVC binds to a PV with Retain Dynamic provisioning fails with ProvisioningFailed CSI driver not installed, wrong provisioner name, or cloud quota exhausted kubectl describe pvc shows ProvisioningFailed event Install the correct CSI driver, verify provisioner matches driver name, check cloud provider quotas ## Local Pilot Plan ### 1. Set up a minimal cluster (kind / minikube / k3d) # Example with kind kind create cluster --name storage-demo kubectl cluster-info --context kind-storage-demo ### 2. Install a local CSI driver for dynamic provisioning (optional but realistic) # Hostpath CSI driver for kind kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/csi-driver-host-path/master/deploy/kubernetes-1.24/deploy.yaml ### 3. Create a StorageClass that uses the hostpath driver # storageclass-hostpath.yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: local-hostpath provisioner: hostpath.csi.k8s.io volumeBindingMode: WaitForFirstConsumer reclaimPolicy: Delete parameters: type: directory Apply and verify: kubectl apply -f storageclass-hostpath.yaml kubectl get storageclass local-hostpath ### 4. Request storage with a PVC # pvc-data.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: data-pvc namespace: default spec: accessModes: - ReadWriteOnce resources: requests: storage: 2Gi storageClassName: local-hostpath kubectl apply -f pvc-data.yaml kubectl get pvc data-pvc -w # watch until STATUS becomes Bound ### 5. Deploy a workload that mounts the PVC # deployment-app.yaml apiVersion: apps/v1 kind: Deployment metadata: name: storage-writer spec: replicas: 1 selector: matchLabels: app: storage-writer template: metadata: labels: app: storage-writer spec: containers: - name: writer image: busybox:1.36 command: [ "sh", "-c", "while true; do echo $(date) >> /data/out.txt; sleep 5; done" ] volumeMounts: - name: data-volume mountPath: /data volumes: - name: data-volume persistentVolumeClaim: claimName: data-pvc kubectl apply -f deployment-app.yaml kubectl rollout status deployment/storage-writer ### 6. Validate write and persistence # Get the pod name POD=$(kubectl get pods -l app=storage-writer -o jsonpath='{.items[0].metadata.name}') # Inspect the file inside the container kubectl exec $POD -- cat /data/out.txt # Delete the pod to force recreation kubectl delete pod $POD # Wait for new pod and verify data survived kubectl rollout status deployment/storage-writer NEW_POD=$(kubectl get pods -l app=storage-writer -o jsonpath='{.items[0].metadata.name}') kubectl exec $NEW_POD -- cat /data/out.txt If the timestamps continue without gaps, the PV survived the Pod restart. ### 7. Simulate a failure: change the PVC to request ReadWriteMany # pvc-data-rwm.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: data-pvc-rwm spec: accessModes: - ReadWriteMany resources: requests: storage: 2Gi storageClassName: local-hostpath kubectl apply -f pvc-data-rwm.yaml kubectl get pvc data-pvc-rwm The PVC will stay Pending because the hostpath driver only supports ReadWriteOnce . The describe output shows FailedBinding with a message about incompatible access modes — exactly the scenario you will hit when moving to a network filesystem that does support ReadWriteMany . ## Conclusion Treating storage configuration as testable code — not as a one‑time copy‑paste — is the safest way to avoid surprises in CI/CD pipelines and production clusters. The workflow above lets you: - Define a StorageClass that matches your infrastructure (local hostpath, cloud CSI, NFS, etc.). - Request storage with a PVC that expresses the exact capacity and access mode you need. - Verify binding with kubectl get pvc and kubectl describe pvc . - Mount the PVC in a real workload and confirm data survives Pod recreation. - Intentionally break the request (wrong access mode, missing StorageClass, exhausted quota) to see the exact error messages you will later troubleshoot under pressure. Next steps for a production‑grade rollout: - Replace the hostpath driver with the CSI driver of your cloud provider (EBS, PD, Azure Disk, etc.). - Set volumeBindingMode: WaitForFirstConsumer for topology‑aware provisioning. - Use reclaimPolicy: Retain for critical data and implement a backup strategy (Velero, CSI snapshots). - Add fsGroup and runAsUser in the Pod securityContext to avoid permission issues on shared volumes. By running the commands locally first, you turn abstract concepts into observable behavior, making the transition to staging and production predictable rather than hopeful.