## Intro Operating Kubernetes Storage Classes in production is about ensuring that your stateful workloads get the storage they need reliably, without manual intervention, and with clear recovery paths when something goes wrong. A Storage Class abstracts the underlying storage technology (like AWS EBS, GCE Persistent Disk, or an on-premises CSI driver) and lets developers request storage without knowing the details. But in production, this abstraction means that failures can be subtle: a PVC may be stuck in Pending, a volume may not mount, or an application may write to the wrong storage tier. This checklist is designed for developers, DevOps engineers, and technical startup teams who manage Kubernetes clusters. It covers the full lifecycle: before you deploy, when you provision, when you verify, and when you need to recover. Each section provides concrete commands, expected outputs, and practical examples that you can run against your own cluster. We focus on operational safety: observe first, make minimal changes, protect secrets, and always know how to roll back. ## Version and Environment Inventory Before you touch anything, you need to know exactly what you are working with. Start by identifying the Kubernetes version, the CSI drivers installed, and the storage classes defined in the cluster. This information is read-only and will help you avoid making changes that are incompatible with your environment. Run these commands to capture the current state: ```bash kubectl version --short kubectl get storageclass kubectl get csidrivers ``` Expected output for the storage class list might look like: ``` NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE standard (default) kubernetes.io/aws-ebs Delete Immediate false 30d fast ebs.csi.aws.com Delete WaitForFirstConsumer true 10d ``` In this example, `standard` is the default class using the in-tree AWS EBS provisioner, while `fast` uses the newer CSI driver and supports volume expansion. Knowing which is default matters because a PVC without a `storageClassName` will use the default. Check the CSI driver status: ```bash kubectl get csidriver ``` Example output: ``` NAME ATTACHREQUIRED PODINFOONMOUNT STORAGECAPACITY TOKENREQUESTS REQUIRESREPUBLISH ... ebs.csi.aws.com true false true false false ``` If a CSI driver is missing or not ready, your PVCs may hang in `Pending`. Verify that the CSI controller pods are running: ```bash kubectl get pods -n kube-system | grep ebs ``` Expected: pods like `ebs-csi-controller-...` and `ebs-csi-node-...` should be `Running` and `Ready`. Document the versions of the CSI driver and any storage-related operators (like the AWS EBS CSI driver version). For example, if you are using the AWS EBS CSI driver, check its image tag with: ```bash kubectl get deployment ebs-csi-controller -n kube-system -o jsonpath='{.spec.template.spec.containers[0].image}' ``` This might output `public.ecr.aws/ebs-csi-driver/aws-ebs-csi-driver:v1.26.0`. Keep a record of this because a driver update can change behavior. One common mistake is checking only the storage class and not the CSI driver. A storage class may exist but be unusable if the driver is not installed or has crashed. The same applies to node-level components: run `kubectl get pods -n kube-system -l app=ebs-csi-node` to ensure the node plugin is scheduled on all nodes that need storage. Before making any change, ensure you have read access to the relevant namespaces and that you are not using a service account with overly broad permissions. For read-only checks, a role with `get`, `list`, and `watch` on storage classes, PVCs, and pods is enough. Never run cluster-admin commands from a shared terminal without confirming the context: `kubectl config current-context`. For local testing, create a small PVC and a pod that uses it, then delete both and confirm cleanup. This validates the full path without impacting production workloads. Example test manifest: ```yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: test-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi storageClassName: fast --- apiVersion: v1 kind: Pod metadata: name: test-pod spec: containers: - name: app image: busybox command: ["sh", "-c", "echo 'Storage works' > /data/test.txt && sleep 3600"] volumeMounts: - name: data mountPath: /data volumes: - name: data persistentVolumeClaim: claimName: test-pvc ``` Apply it, check the pod logs, then clean up with `kubectl delete -f test-manifest.yaml`. ## Safe Configuration Path The configuration of a Storage Class determines how volumes are provisioned, what parameters are passed to the provisioner, and the reclaim policy. A misconfigured Storage Class can lead to data loss (e.g., `reclaimPolicy: Delete` on a class used for critical data) or unexpected charges. Here is a well-annotated Storage Class example for AWS EBS with the CSI driver: ```yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: fast-ssd provisioner: ebs.csi.aws.com parameters: type: gp3 iops: "3000" throughput: "125" encrypted: "true" kmsKeyId: "arn:aws:kms:us-east-1:123456789012:key/abcd1234-..." reclaimPolicy: Retain volumeBindingMode: WaitForFirstConsumer allowVolumeExpansion: true ``` Key points: - `provisioner` must match the CSI driver name exactly. A typo here will cause PVCs to fail with `provisioner not found`. - `parameters` are passed to the CSI driver. For AWS EBS, `type` can be `gp2`, `gp3`, `io1`, `io2`, etc. The `iops` and `throughput` are only valid for certain types; specifying them for `gp2` will cause an error. - `reclaimPolicy` should be `Retain` for any volume that holds data you might need after the PVC is deleted. With `Delete`, the underlying cloud volume is deleted immediately, often without a backup. - `volumeBindingMode: WaitForFirstConsumer` is recommended for multi-AZ clusters because it delays volume creation until a pod is scheduled, ensuring the volume is created in the same AZ as the node. Without this, you may get a volume in a different AZ and the pod will fail to attach. - `allowVolumeExpansion: true` allows increasing the size of a PVC. This is useful but not supported by all drivers. To change an existing Storage Class, you can edit it with `kubectl edit storageclass fast-ssd`. However, most parameters are immutable after creation. You can add new parameters or change `allowVolumeExpansion`, but changing the provisioner or volume binding mode requires creating a new class and migrating workloads. When creating a new Storage Class, follow a naming convention that reflects performance and data durability, like `fast-ssd-encrypted` or `standard-hdd`. Avoid using the default class for production unless you explicitly set it to a safe configuration. You can see which class is default with: ```bash kubectl get storageclass ``` The default class is marked with `(default)` next to its name in the output. If you need programmatic extraction, use: ```bash kubectl get storageclass -o jsonpath='{.items[?(@.metadata.annotations.storageclass\.kubernetes\.io/is-default-class=="true")].metadata.name}' ``` If you need to change the default class, unset the old default and set the new one: ```bash kubectl patch storageclass standard -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}' kubectl patch storageclass fast-ssd -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}' ``` Be careful: changing the default class affects all future PVCs that do not specify a class. Existing PVCs are unaffected. Always test a new Storage Class with a small PVC in a staging namespace before using it for production workloads. Use `kubectl get events` to see provisioning errors. For example, an invalid parameter may produce an event like: ``` Warning ProvisioningFailed 2m (x5 over 10m) ebs.csi.aws.com_ebs-csi-controller-... failed to provision volume with StorageClass "fast-ssd": rpc error: code = InvalidArgument desc = Invalid parameter: iops ``` ## Verification and Diagnostics After you have a Storage Class configured and a PVC bound, you need to verify that the volume actually works as expected. This means checking the PVC status, the underlying PersistentVolume, and the actual data path. Start by checking the PVC: ```bash kubectl get pvc ``` Expected output: ``` NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE test-pvc Bound pvc-1234abcd-5678-efgh-ijkl-1234567890ab 1Gi RWO fast-ssd 5m ``` A PVC in `Pending` state means the provisioner has not created a volume. Common reasons: no Storage Class with that name, the CSI driver is not running, or the parameters are invalid. Describe the PVC for events: ```bash kubectl describe pvc test-pvc ``` Look for events at the bottom. If you see `waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator`, then the provisioner may be down or misconfigured. Check the provisioner logs: ```bash kubectl logs -n kube-system deployment/ebs-csi-controller -c ebs-plugin ``` For a Bound PVC, check the PersistentVolume: ```bash kubectl get pv pvc-1234abcd-5678-efgh-ijkl-1234567890ab ``` Note the `RECLAIM POLICY` and `STATUS`. The PV should be `Bound` to your PVC. If the reclaim policy is `Delete`, deleting the PVC will also delete the PV and the underlying storage. To verify that a pod can actually use the volume, attach a pod and write a file. Here is an example pod that writes and reads back: ```yaml apiVersion: v1 kind: Pod metadata: name: storage-verify spec: containers: - name: verifier image: alpine command: ["/bin/sh", "-c"] args: - | echo "$(date): storage test" > /data/testfile.txt cat /data/testfile.txt sleep 3600 volumeMounts: - name: data mountPath: /data volumes: - name: data persistentVolumeClaim: claimName: test-pvc ``` Check the logs: ```bash kubectl logs storage-verify ``` It should print the date and the test string. If the volume is read-only or the mount is wrong, you will see permission errors. Diagnostics should also include performance checks if your application is I/O sensitive. You can use a tool like `fio` inside a pod to test bandwidth and IOPS. Example: ```yaml apiVersion: v1 kind: Pod metadata: name: fio-test spec: containers: - name: fio image: ljishen/fio command: ["fio"] args: - --name=test - --filename=/data/testfile - --size=100M - --rw=randwrite - --bs=4k - --ioengine=libaio - --iodepth=32 - --direct=1 - --numjobs=1 - --runtime=60 - --time_based volumeMounts: - name: data mountPath: /data volumes: - name: data persistentVolumeClaim: claimName: test-pvc ``` Run it and examine the output for IOPS and latency. Compare with the expected performance for your storage type (e.g., gp3 baseline is 3000 IOPS and 125 MB/s). If you see far lower numbers, check for throttling (e.g., burst credits exhausted on gp2) or network issues. Finally, verify that volume expansion works if you enabled `allowVolumeExpansion`. First, increase the PVC size: ```bash kubectl patch pvc test-pvc -p '{"spec":{"resources":{"requests":{"storage":"2Gi"}}}}' ``` Then check that the PVC eventually shows the new capacity and the pod can see the expanded filesystem. Note that some drivers require the pod to be restarted to resize the filesystem. For AWS EBS CSI, the filesystem is resized online if the pod is running. ## Failure Modes and Recovery Storage failures in Kubernetes can be broadly categorized into provisioning failures (PVC stuck in Pending), binding failures (PVC not binding to a suitable PV), attachment/mount failures (pod cannot start because volume cannot be attached), and data integrity issues (corruption or accidental deletion). Here we cover the most common scenarios and recovery steps. ### Provisioning Failure **Symptom:** PVC remains `Pending` and events show errors like `failed to provision volume` or `provisioner not found`. **Cause:** Incorrect provisioner name in Storage Class, CSI driver not running, or invalid parameters. **Recovery:** 1. Check Storage Class: `kubectl get storageclass -o yaml` to see the provisioner. 2. Check CSI driver pods: `kubectl get pods -n kube-system | grep `. 3. If driver pods are crash-looping, check logs: `kubectl logs -n kube-system --previous`. 4. Fix the issue (e.g., provide correct IAM permissions to the CSI controller, repair the driver deployment). 5. If the Storage Class itself is wrong and immutable, create a new one and update the PVC to use it (you may need to delete and recreate the PVC, but be careful with data). ### Binding Failure with WaitForFirstConsumer **Symptom:** PVC is `Pending` even though the provisioner works, but the pod that references it is not scheduled. **Cause:** `volumeBindingMode: WaitForFirstConsumer` means Kubernetes will not provision until a pod using the PVC is scheduled to a node. If the pod cannot be scheduled (insufficient resources, node selector mismatch), the PVC stays pending. **Recovery:** Check pod events: `kubectl describe pod `. Resolve the scheduling issue. If you need volume immediately regardless of pod scheduling, use `volumeBindingMode: Immediate` instead (but be aware of cross-AZ issues). ### Pod Stuck in ContainerCreating **Symptom:** Pod is in `ContainerCreating` state for a long time; `kubectl describe pod` shows `Multi-Attach error` or `FailedAttachVolume`. **Cause:** The volume is already attached to another node (perhaps a previous pod did not detach cleanly) or the node cannot reach the storage backend. **Recovery:** 1. Check node status: `kubectl get nodes`. 2. Check volume attachment objects: `kubectl get volumeattachment`. 3. If a volume attachment is stuck, you may need to force detach by deleting the VolumeAttachment object: `kubectl delete volumeattachment ` (only after ensuring the pod using it is gone). 4. Restart the CSI node plugin on the affected node: `kubectl delete pod -n kube-system -l app=ebs-csi-node --field-selector spec.nodeName=`. ### Data Loss from Reclaim Policy Delete **Symptom:** A PVC was deleted and the underlying data is gone. **Cause:** The Storage Class has `reclaimPolicy: Delete`, which automatically deletes the cloud volume when the PVC is deleted. **Recovery:** If you have a backup or snapshot, restore from it. If not, the data is likely unrecoverable. To prevent this, set `reclaimPolicy: Retain` for important data and implement a backup strategy. Note that changing the reclaim policy of an existing Storage Class will not affect existing PVs; you need to patch the PV: `kubectl patch pv -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'`. ### Volume Corruption or Read-Only **Symptom:** Application fails to write, pod logs show `read-only file system` or I/O errors. **Cause:** Filesystem corruption, disk full, or the volume was mounted read-only due to errors. **Recovery:** 1. Check filesystem usage: `kubectl exec -- df -h /data`. 2. If full, increase PVC size or clean up files. 3. If corrupted, you may need to run `fsck` on the volume. This typically requires detaching the volume from the node and running a repair pod with the volume attached. For cloud volumes, you can often take a snapshot first and then repair or restore. For each failure, always check the Kubernetes events first: `kubectl get events --all-namespaces --sort-by='.lastTimestamp'`. Events often contain the exact error message from the CSI driver. ## Operations Checklist This section is a concise checklist you can use during day-to-day operations. It is organized by frequency and includes the owner for each item. In a startup team, ownership may be shared, but each item should have a single responsible person to avoid ambiguity. Review frequencies assume a small team with a few stateful applications; adjust as needed. ### Daily Checks (Owner: DevOps Engineer on call – e.g., Alex Johnson, DevOps Lead) - [ ] Check for any PVCs in `Pending` or `Lost` state: `kubectl get pvc --all-namespaces | grep -v Bound` - [ ] Check for pods stuck in `ContainerCreating` or `CrashLoopBackOff` related to storage: `kubectl get pods --all-namespaces | grep -E 'ContainerCreating|CrashLoopBackOff'` - [ ] Review events for the last 24 hours for storage-related warnings: `kubectl get events --all-namespaces --field-selector type=Warning | grep -i -E 'storage|volume|disk'` ### Weekly Checks (Owner: Platform Engineer – e.g., Priya Shah, Senior Platform Engineer) - [ ] Review storage class usage and ensure default class is correct: `kubectl get storageclass` - [ ] Check CSI driver logs for errors: `kubectl logs -n kube-system deployment/ebs-csi-controller -c ebs-plugin --since=48h | grep -i error` - [ ] Verify that all nodes have the CSI node plugin running: `kubectl get pods -n kube-system -l app=ebs-csi-node -o wide` (ensure one per node) - [ ] Test a sample PVC provisioning and deletion in a test namespace to ensure the provisioner works end-to-end. ### Monthly Checks (Owner: Engineering Manager – e.g., Carlos Rodriguez, Engineering Manager) - [ ] Review storage costs in the cloud provider console. Look for orphaned volumes (volumes not attached to any node but still billed). - [ ] Check for volumes that are not managed by Kubernetes (e.g., manually created EBS volumes) and decide if they should be imported. - [ ] Review backup policies for PersistentVolumeClaims. Ensure that all critical PVCs have snapshots or backups. - [ ] Test restore from a snapshot or backup in a staging environment. ### Quarterly Checks (Owner: Tech Lead – e.g., Emma Wilson, Tech Lead) - [ ] Review the list of Storage Classes and remove unused ones. Keep only classes that are actively used. - [ ] Update CSI driver versions after testing in a staging cluster. - [ ] Review resource limits for CSI controller pods to avoid OOM kills under load. - [ ] Revisit the reclaim policies of all Storage Classes and PVs. Ensure `Retain` is used for anything important. Each checklist item should have a documented runbook. For example, the runbook for "PVC stuck in Pending" should include steps like: 1. Check if the Storage Class exists: `kubectl get sc ` 2. Describe the PVC: `kubectl describe pvc ` 3. Check provisioner logs: `kubectl logs -n kube-system deployment/` 4. Check if the CSI driver pods are running 5. Check if the cloud provider has sufficient quota for the storage type 6. Escalate to the storage provider support if the issue persists ## Common Pitfalls and How to Avoid Them Pitfalls in storage operations often arise from misunderstanding the dynamic provisioning lifecycle or overlooking cluster-specific settings. Here are the most frequent mistakes and their remedies. ### Using a Non-Existent Storage Class in a PVC **Why it happens:** Developers copy PVC manifests from examples without checking if the Storage Class exists in the cluster. **How to avoid:** Before applying a PVC, verify the Storage Class name: `kubectl get storageclass`. If you want to use a default class, omit `storageClassName` entirely or set it to `""` (empty string) if you want to disable dynamic provisioning. **Recovery:** Create the Storage Class or fix the PVC to use an existing one. In some cases, you can patch the PVC's `storageClassName` field if it is still unbound, but once bound, you cannot change it. ### Not Setting Volume Binding Mode to WaitForFirstConsumer in Multi-AZ Clusters **Why it happens:** The default `volumeBindingMode` is `Immediate`, which creates the volume as soon as the PVC is created, often in a random AZ. If the pod is scheduled in a different AZ, the volume cannot attach. **How to avoid:** Always use `WaitForFirstConsumer` for storage classes that are used in multi-AZ clusters, especially for block storage like EBS. Check the AZ of your nodes with `kubectl get nodes -L topology.kubernetes.io/zone`. **Recovery:** For existing PVCs, if the volume is in the wrong AZ, you need to delete the PVC (and possibly the PV) and recreate it with the correct class. To avoid data loss, snapshot the volume first, create a new volume from the snapshot in the right AZ, and then bind to it manually. ### Ignoring Reclaim Policy **Why it happens:** The default reclaim policy in many cloud provider Storage Classes is `Delete`, and teams do not realize that deleting a PVC will also delete the data. **How to avoid:** For any production data, use a Storage Class with `reclaimPolicy: Retain`. If you must use `Delete`, ensure you have backups. Also, set up alerts for PVC deletion events. **Recovery:** If you accidentally deleted a PVC with `Delete` policy, check if the cloud provider has a recycle bin or trash feature (some providers retain deleted volumes for a short period). If not, restore from backup. ### Expanding a Volume Without Checking Driver Support **Why it happens:** A user sets `allowVolumeExpansion: true` but the CSI driver does not support expansion, or the pod does not support online resizing. **How to avoid:** Verify the driver's capability by checking its documentation or running `kubectl get csidriver -o yaml` and looking for `supportsVolumeExpansion` (though not always present). Test expansion in a staging environment first. **Recovery:** If expansion fails, check the events and driver logs. For some drivers, you may need to restart the pod to trigger filesystem resize. If the volume is corrupted, restore from snapshot. ### Not Monitoring CSI Driver Health **Why it happens:** Teams assume the CSI driver is always running, but it can crash, be evicted, or fail to start after a node reboot. **How to avoid:** Set up alerts for CSI controller pod restarts and node plugin pod failures. Use `kubectl get pods -n kube-system -l app=` in your monitoring dashboards. **Recovery:** If the driver is down, PVCs will fail to provision and pods may not start. Restart the deployment or daemonset, and check logs for the root cause. ## Conclusion Kubernetes Storage Classes are a powerful abstraction, but they require careful operational discipline. By following this checklist, you can avoid most common issues, diagnose problems quickly, and recover from failures with minimal data loss. Remember the core principles: observe before changing, use read-only commands for diagnosis, make one scoped change at a time, and always have a rollback plan. As a next step, pick one low-risk verification from this article and run it in your cluster. For example, check if your default Storage Class has `reclaimPolicy: Retain` for important workloads. If not, create a new class with safer settings and gradually migrate your stateful applications. A reliable storage operations workflow ensures that failures are visible early, sensitive data is protected, and recovery steps are documented before you need them. This checklist gives you the foundation; adapt it to your team's specific storage backend and operational maturity.