## Intro Kubernetes Persistent Volume (PV) capacity planning is often treated as a one-time guess: pick a number, create a PersistentVolumeClaim (PVC), and hope the application never fills the disk. That approach leads to outages, stuck pods, and emergency expansion attempts. This guide moves you from that observed problem to a verified result with practical, repeatable steps. The article is written for developers, DevOps consultants, and technical startup teams who run stateful workloads on Kubernetes. It connects PV sizing, PVC requests, StorageClass expansion policies, and resource monitoring to commands, expected output, failure signals, and recovery decisions. You will learn how to inspect current usage, choose a sizing strategy, expand volumes safely, and handle common failure modes. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached. ## Version and Environment Inventory Before planning capacity, you must know exactly what you are working with. Run these commands in your target cluster and record the output. Replace placeholders like `` and `` with your actual values. ### 1. Check the Kubernetes version and storage capabilities ```bash kubectl version --short kubectl get storageclass ``` Expected output for `kubectl version --short` includes a server version such as `Server Version: v1.28.5`. The storage class list shows names like `standard`, `gp2`, or `managed-premium`, along with provisioner and reclaim policy. Pay attention to the `ALLOWVOLUMEEXPANSION` column (visible with `kubectl get storageclass -o wide`). If it is `true`, the StorageClass supports online expansion. If `false`, you must handle expansion manually or use a different class. ```bash kubectl get storageclass -o wide NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE standard (default) kubernetes.io/gce-pd Delete Immediate true 12d ``` ### 2. List existing PVs and PVCs ```bash kubectl get pv kubectl get pvc -n ``` Look for PVs with status `Released` or `Failed`. A `Released` PV means the PVC was deleted but the PV was not reclaimed (reclaim policy `Retain`). Such a PV cannot be bound to a new PVC without manual cleanup. This is a common source of wasted capacity and unexpected failures. Example: ```bash NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-abc123 10Gi RWO Delete Bound default/my-app-data standard 7d pvc-def456 5Gi RWO Retain Released default/old-claim standard 14d ``` The `pvc-def456` PV occupies 5Gi but is not usable. You may need to delete it after ensuring it is not needed. ### 3. Inspect a PVC's current request and capacity ```bash kubectl get pvc -n -o yaml ``` This shows `spec.resources.requests.storage` and `status.capacity.storage`. In many cases they are equal, but they differ after expansion (capacity catches up) or with certain drivers. ### 4. Check the pod using the PVC and its filesystem usage ```bash kubectl get pod -n -l app= kubectl exec -it -n -- df -h /data ``` Example output: ``` Filesystem Size Used Avail Use% Mounted on /dev/sdb 9.8G 7.2G 2.4G 76% /data ``` This tells you actual consumption, not just the requested size. A pod may be using far less than the PVC request, indicating overallocation. Conversely, if usage is above 80%, you should plan expansion or cleanup. ### Prerequisites for capacity planning - Cluster access with `kubectl` configured. - Permission to get and describe storage resources (`kubectl auth can-i get pvc --namespace `). - Knowledge of the StorageClass provisioner (cloud or on-prem) and its expansion support. - For StatefulSets, understanding of how volumeClaimTemplates create per-pod PVCs. Always separate observation from intervention. Capture the current state and timestamps before making any change. For example, save a snapshot: ```bash date -u +%Y%m%dT%H%M%SZ > /tmp/ts kubectl get pvc -n -o yaml > /tmp/pvc-before-$(cat /tmp/ts).yaml ``` ## Safe Configuration Path This section outlines a least-risk method for changing PV capacity or related settings. The central principle: make one scoped change, verify, and know the rollback. ### Step 1: Identify the smallest justified change Capacity planning typically involves: - Increasing a PVC's requested storage. - Adjusting an application's resource limits to prevent disk overflow (e.g., log rotation). - Changing the StorageClass default or adding a new one for future workloads. Do not change the StorageClass of an existing bound PVC; it is immutable. Instead, create a new PVC with the desired class and migrate data if necessary. ### Step 2: Validate the expansion capability Before attempting expansion, check the StorageClass: ```bash kubectl get storageclass -o yaml | grep allowVolumeExpansion ``` If empty or `false`, expansion via Kubernetes is not supported. You may need to provision a new larger volume and copy data. ### Step 3: Test with a non-critical PVC If possible, perform a dry run on a test PVC in the same namespace or a development environment. Expand a PVC from 1Gi to 2Gi and observe: ```bash kubectl patch pvc test-pvc -n dev -p '{"spec":{"resources":{"requests":{"storage":"2Gi"}}}}' kubectl get pvc test-pvc -n dev -w ``` Watch for the status `Bound` and capacity updated. If the expansion hangs in `Resizing`, investigate events: ```bash kubectl describe pvc test-pvc -n dev ``` ### Step 4: Expand the production PVC Once validated, apply the same patch to the production PVC: ```bash kubectl patch pvc my-app-data -n prod -p '{"spec":{"resources":{"requests":{"storage":"20Gi"}}}}' ``` Monitor the expansion: ```bash kubectl get pvc my-app-data -n prod -w ``` Most cloud providers complete expansion within a few minutes. The pod using the volume should automatically see the new size without restart (if the driver supports online expansion). For file system resize, the kubelet resizes the file system automatically after the volume is expanded. You can verify inside the pod: ```bash kubectl exec -it -n prod -- df -h /data ``` Expected output shows the new size and increased available space. ### Step 5: Document the change and rollback Record the before/after state: ```bash kubectl get pvc my-app-data -n prod -o yaml > /tmp/pvc-after-$(date -u +%Y%m%dT%H%M%SZ).yaml ``` Expansion is generally one-way; you cannot shrink a PVC. If you need to revert, you must create a smaller PVC and restore data from backup. Therefore, always verify the need and available underlying storage before expanding. ### Configuration snippet example: PVC with explicit size and StorageClass ```yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: app-data namespace: prod spec: accessModes: - ReadWriteOnce storageClassName: standard resources: requests: storage: 10Gi ``` Apply it: ```bash kubectl apply -f pvc.yaml ``` ### Keep the local test small Apply one manifest, inspect the generated resources, and verify access with a temporary pod before moving to a production workload. For example: ```bash kubectl run -it --rm test-pod --image=busybox --restart=Never -- sh # Inside pod: df -h /mnt ``` ## Verification and Diagnostics After any change or on a regular schedule, verify the actual state of PVs and PVCs. This section provides concrete commands and expected outputs. ### Check PVC status and conditions ```bash kubectl get pvc -n kubectl describe pvc -n ``` Look for: - Status `Bound` (expected). - Events showing `ProvisioningSucceeded`, `Resizing`, or warnings. - Conditions `FileSystemResizePending` (true means the PVC was expanded at the volume level but the file system resize has not completed, often due to the pod not being restarted or the driver not supporting online resize). - Condition `Resizing` (true while expansion is in progress). ### Monitor actual usage over time For a pod with PVC mounted at `/data`, run: ```bash kubectl exec -it -n -- sh -c 'df -h /data; du -sh /data/* | sort -rh | head -n 10' ``` This gives usage percentage and top directories. It helps identify whether data logs or application files are growing unexpectedly. For a more systematic approach, use a Prometheus query if node exporter or kubelet metrics are available: ```promql kubelet_volume_stats_used_bytes{namespace="",persistentvolumeclaim=""} / kubelet_volume_stats_capacity_bytes{namespace="",persistentvolumeclaim=""} ``` Set an alert when usage exceeds 80%. ### Check events for storage issues ```bash kubectl get events -n --sort-by=.lastTimestamp | grep -i volume ``` Expected output includes `ProvisioningSucceeded`, `VolumeResizeSuccessful`, or failure messages like `Failed to resize volume` or `Insufficient capacity`. ### Verify StatefulSet PVCs For StatefulSets, each pod has its own PVC named `--`. List them: ```bash kubectl get pvc -n -l app= ``` Use the same monitoring techniques per pod. ### Diagnostic example: PVC stuck in Pending If a PVC remains `Pending`, run: ```bash kubectl describe pvc -n ``` Common reasons: - No matching StorageClass or provisioner. - Insufficient capacity in the backing storage system. - Access mode not supported (e.g., requesting ReadWriteMany on a block storage class). - Resource quota exceeded. Fix the root cause, not the symptom. For example, if the StorageClass name has a typo, create a new PVC with the correct name, or update the deployment to use the correct claim. ## Failure Modes and Recovery This section details common failures with PV capacity and how to recover. ### Failure 1: PVC expansion hangs **Symptom**: `kubectl get pvc` shows `Resizing` for a long time, or `kubectl describe pvc` shows an event like `Waiting for user to (re-)start a pod to finish file system resize of volume on node`. **Why it happens**: For some volume types (e.g., certain CSI drivers or older Kubernetes versions), the file system resize only occurs when the pod using the volume is restarted. Online expansion without restart is not universally supported. **Recovery**: Restart the pod that uses the PVC. For a deployment: ```bash kubectl rollout restart deployment -n ``` After restart, check the PVC condition `FileSystemResizePending` becomes `false` and the new size is reflected inside the pod. ### Failure 2: Disk full and pod evicted or crashing **Symptom**: Pod logs show `no space left on device`, pod enters `CrashLoopBackOff`, or node reports pressure. The PVC usage is at 100%. **Why it happens**: Application logs or data grew beyond the requested size, and no cleanup or expansion was done in time. **Recovery**: 1. Immediately increase the PVC size if the StorageClass allows expansion (see Safe Configuration Path). 2. If expansion is impossible, free space inside the pod: ```bash kubectl exec -it -n -- sh -c 'du -sh /data/* | sort -rh | head' ``` Delete unneeded logs or tmp files, then restart the pod. This is a stopgap; implement log rotation or data archival. 3. For long-term, configure application-level retention (e.g., logrotate) or move non-critical data to object storage. ### Failure 3: PVC deleted accidentally, data lost **Symptom**: PVC is missing, pod cannot mount volume. The PV may still exist with status `Released` if reclaim policy was `Retain`. **Why it happens**: Human error or a script deleting the wrong resource. **Recovery**: - If PV still exists with `Retain`, you can potentially rebind it manually. However, it is safer to restore from backup. - If reclaim policy was `Delete`, the PV and underlying storage are gone. Restore from backup. This underscores the need for regular backups of stateful data. Use tools like Velero or cloud snapshots. Check the reclaim policy before deletion: ```bash kubectl get pv -o yaml | grep reclaimPolicy ``` ### Failure 4: Cannot expand because StorageClass does not allow expansion **Symptom**: `kubectl patch pvc` returns an error like `only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize`. **Why it happens**: The StorageClass has `allowVolumeExpansion: false` or is absent, or the volume is statically provisioned. **Recovery**: - Create a new larger PVC with a StorageClass that supports expansion, then copy data using a temporary pod or `kubectl cp`. - Alternatively, provision a new PV manually and migrate. Steps to migrate: 1. Create new PVC with desired size. 2. Spin up a helper pod with both PVCs mounted. 3. Copy data: `kubectl cp -n :/data /tmp/data` then `kubectl cp -n /tmp/data :/data` (adjust paths). 4. Update the deployment to use the new PVC. 5. Delete old PVC. Always test the migration in a non-production environment first. ### Failure 5: Capacity mismatch between request and actual usage **Symptom**: PVC requests 100Gi but pod uses only 10Gi. Or storage costs escalate. **Why it happens**: Initial sizing based on guesswork rather than metrics. **Recovery**: - PVC size cannot be reduced in-place. To right-size, you must create a new smaller PVC and migrate data (as above). - Plan for future growth by setting a reasonable initial size and monitoring trends. Use a policy to start with 1.5x the expected peak usage, with a minimum of 5Gi for small workloads. ## Operations Checklist Use this checklist for recurring PV capacity management. Assign a single accountable owner for each major decision, and review at a defined cadence (e.g., weekly for high-growth workloads, monthly for stable). | Task | Command or method | Expected result | Owner | Review frequency | |------|-------------------|-----------------|-------|------------------| | Check all PVC statuses | `kubectl get pvc --all-namespaces` | All `Bound`, no `Pending` or `Resizing` | Storage admin (e.g., Jordan Lee, Platform Engineering) | Weekly | | Monitor usage percentage top 10 PVCs | `kubectl get --raw /api/v1/nodes//proxy/stats/summary` or Prometheus query | Usage below 80% | DevOps engineer (e.g., Priya Shah, DevOps Lead) | Weekly | | Review StorageClass expansion settings | `kubectl get storageclass -o wide` | Desired classes have `true` under `ALLOWVOLUMEEXPANSION` | Platform engineer (e.g., Alex Kim, Platform Lead) | Monthly | | Check for Released PVs needing cleanup | `kubectl get pv --field-selector=status.phase=Released` | List empty or cleanup scheduled | Storage admin | Monthly | | Verify backup restoration test | Run restore drill on a non-critical PVC | Backup restores successfully within RTO | Site Reliability Engineer (SRE) | Quarterly | | Review capacity forecasts vs. actual growth | Compare `df -h` trends over 3 months, adjust quotas | Forecast error < 10% | Engineering Manager (e.g., Sam Rivera, VP Engineering) | Quarterly | For each checklist item, the owner records the result in a tracking system (e.g., a GitHub issue or Jira ticket) and notes any deviations. The engineering lead reviews the tracked items during the monthly infrastructure review. ## Common Pitfalls and How to Avoid Them ### Pitfall 1: Using the default StorageClass without checking expansion support Many clusters set a default StorageClass (e.g., `standard`) that may have `allowVolumeExpansion: false`. Developers assume they can expand PVCs later, only to discover it is impossible. **Why it happens**: The cluster was set up with minimal configuration, and no one verified expansion capabilities before deploying stateful apps. **How to avoid**: Before deploying a workload, run: ```bash kubectl get storageclass -o yaml | grep allowVolumeExpansion ``` If `false` or missing, either use a different class that supports expansion or adjust expectations and plan for migration. ### Pitfall 2: Ignoring file system overhead and inode exhaustion Although a PVC is 10Gi, the file system (ext4, XFS) reserves some space for metadata. Additionally, if the application creates millions of small files, you can run out of inodes (e.g., ext4 default is one inode per 16KiB of space). The disk may show low usage by `df` but new files cannot be created. **Why it happens**: The initial capacity planning only considered bytes, not inode requirements. **How to avoid**: Check inode usage: ```bash kubectl exec -it -n -- df -i /data ``` If inode usage is high (>80%), consider a file system with more inodes or adjust the application to use fewer, larger files. ### Pitfall 3: Not setting resource quotas on namespaces Without quotas, developers can create many large PVCs, exhausting cluster storage and costs. For example, a namespace with a 100Gi quota might request 10 PVCs of 20Gi each. **Why it happens**: Quotas are not configured by default, and the organization lacks a storage governance policy. **How to avoid**: Define a ResourceQuota for storage: ```yaml apiVersion: v1 kind: ResourceQuota metadata: name: storage-quota namespace: team-a spec: hard: requests.storage: 200Gi persistentvolumeclaims: "10" ``` Apply it and educate teams on the limits. ### Pitfall 4: Assuming deletion of a PVC deletes the data Depending on the reclaim policy (`Delete` vs `Retain`), deleting a PVC may or may not delete the underlying storage. With `Retain`, the PV remains and must be manually cleaned up; otherwise it leaks capacity. **Why it happens**: Developers expect the default `Delete` policy but the StorageClass uses `Retain`. **How to avoid**: Check the StorageClass reclaim policy before deletion: ```bash kubectl get storageclass -o yaml | grep reclaimPolicy ``` If `Retain`, explicitly delete the PV after verifying data is no longer needed or backed up. ### Pitfall 5: Not planning for StatefulSet scale-down When scaling down a StatefulSet, the PVCs of the terminated pods are not deleted automatically (unless you use `volumeClaimTemplates` with `delete` policy, but that is risky). Those orphaned PVCs consume storage. **Why it happens**: Kubernetes preserves data for safety, and operators forget to clean up. **How to avoid**: Define a policy for orphaned PVCs. After a scale-down, review and delete unused PVCs if data is backed up or not needed: ```bash kubectl get pvc -n -l app= | grep -v Bound ``` Alternatively, set the StatefulSet `persistentVolumeClaimRetentionPolicy.whenDeleted: Delete` if appropriate. ## Conclusion Kubernetes Persistent Volume capacity planning succeeds only when each recommendation is version-scoped, observable, and reversible where the technology permits. Copying a command without checking prerequisites and expected output is not an operations procedure. Use the commands and checks in this guide to build a baseline for your cluster. As a next step, choose one low-risk verification for a PVC: record the current state, run a monitoring command like `kubectl exec ... -- df -h`, compare the result with the requested size, and schedule a review with your team. Pay attention to dependent components such as PVCs, StatefulSets, and StorageClasses; their configuration affects your ability to expand and recover. A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. Revisit your capacity plan monthly and adjust based on actual usage trends. The payoff is a stable stateful workload and fewer 3 a.m. pages.