Kubernetes Persistent Volume Capacity Planning: A Practical Guide

Intro

Kubernetes Persistent Volume (PV) capacity planning is often treated as a one-time guess: pick a number, create a PersistentVolumeClaim (PVC), and hope the application never fills the disk. That approach leads to outages, stuck pods, and emergency expansion attempts. This guide moves you from that observed problem to a verified result with practical, repeatable steps.

The article is written for developers, DevOps consultants, and technical startup teams who run stateful workloads on Kubernetes. It connects PV sizing, PVC requests, StorageClass expansion policies, and resource monitoring to commands, expected output, failure signals, and recovery decisions. You will learn how to inspect current usage, choose a sizing strategy, expand volumes safely, and handle common failure modes.

The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached.

Version and Environment Inventory

Before planning capacity, you must know exactly what you are working with. Run these commands in your target cluster and record the output. Replace placeholders like <namespace> and <pvc-name> with your actual values.

1. Check the Kubernetes version and storage capabilities

kubectl version --short
kubectl get storageclass

Expected output for kubectl version --short includes a server version such as Server Version: v1.28.5. The storage class list shows names like standard, gp2, or managed-premium, along with provisioner and reclaim policy. Pay attention to the ALLOWVOLUMEEXPANSION column (visible with kubectl get storageclass -o wide). If it is true, the StorageClass supports online expansion. If false, you must handle expansion manually or use a different class.

kubectl get storageclass -o wide
NAME                 PROVISIONER            RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
standard (default)   kubernetes.io/gce-pd   Delete          Immediate           true                   12d

2. List existing PVs and PVCs

kubectl get pv
kubectl get pvc -n <namespace>

Look for PVs with status Released or Failed. A Released PV means the PVC was deleted but the PV was not reclaimed (reclaim policy Retain). Such a PV cannot be bound to a new PVC without manual cleanup. This is a common source of wasted capacity and unexpected failures.

Example:

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM                     STORAGECLASS   REASON   AGE
pvc-abc123                                10Gi       RWO            Delete           Bound      default/my-app-data       standard                7d
pvc-def456                                5Gi        RWO            Retain           Released   default/old-claim         standard                14d

The pvc-def456 PV occupies 5Gi but is not usable. You may need to delete it after ensuring it is not needed.

3. Inspect a PVC's current request and capacity

kubectl get pvc <pvc-name> -n <namespace> -o yaml

This shows spec.resources.requests.storage and status.capacity.storage. In many cases they are equal, but they differ after expansion (capacity catches up) or with certain drivers.

4. Check the pod using the PVC and its filesystem usage

kubectl get pod -n <namespace> -l app=<app-label>
kubectl exec -it <pod-name> -n <namespace> -- df -h /data

Example output:

Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb        9.8G  7.2G  2.4G  76% /data

This tells you actual consumption, not just the requested size. A pod may be using far less than the PVC request, indicating overallocation. Conversely, if usage is above 80%, you should plan expansion or cleanup.

Prerequisites for capacity planning

• Cluster access with kubectl configured.
• Permission to get and describe storage resources (kubectl auth can-i get pvc --namespace <ns>).
• Knowledge of the StorageClass provisioner (cloud or on-prem) and its expansion support.
• For StatefulSets, understanding of how volumeClaimTemplates create per-pod PVCs.

Always separate observation from intervention. Capture the current state and timestamps before making any change. For example, save a snapshot:

date -u +%Y%m%dT%H%M%SZ > /tmp/ts
kubectl get pvc -n <namespace> -o yaml > /tmp/pvc-before-$(cat /tmp/ts).yaml

Safe Configuration Path

This section outlines a least-risk method for changing PV capacity or related settings. The central principle: make one scoped change, verify, and know the rollback.

Step 1: Identify the smallest justified change

Capacity planning typically involves:

• Increasing a PVC's requested storage.
• Adjusting an application's resource limits to prevent disk overflow (e.g., log rotation).
• Changing the StorageClass default or adding a new one for future workloads.

Do not change the StorageClass of an existing bound PVC; it is immutable. Instead, create a new PVC with the desired class and migrate data if necessary.

Step 2: Validate the expansion capability

Before attempting expansion, check the StorageClass:

kubectl get storageclass <class-name> -o yaml | grep allowVolumeExpansion

If empty or false, expansion via Kubernetes is not supported. You may need to provision a new larger volume and copy data.

Step 3: Test with a non-critical PVC

If possible, perform a dry run on a test PVC in the same namespace or a development environment. Expand a PVC from 1Gi to 2Gi and observe:

kubectl patch pvc test-pvc -n dev -p '{"spec":{"resources":{"requests":{"storage":"2Gi"}}}}'
kubectl get pvc test-pvc -n dev -w

Watch for the status Bound and capacity updated. If the expansion hangs in Resizing, investigate events:

kubectl describe pvc test-pvc -n dev

Step 4: Expand the production PVC

Once validated, apply the same patch to the production PVC:

kubectl patch pvc my-app-data -n prod -p '{"spec":{"resources":{"requests":{"storage":"20Gi"}}}}'

Monitor the expansion:

kubectl get pvc my-app-data -n prod -w

Most cloud providers complete expansion within a few minutes. The pod using the volume should automatically see the new size without restart (if the driver supports online expansion). For file system resize, the kubelet resizes the file system automatically after the volume is expanded. You can verify inside the pod:

kubectl exec -it <pod> -n prod -- df -h /data

Expected output shows the new size and increased available space.

Step 5: Document the change and rollback

Record the before/after state:

kubectl get pvc my-app-data -n prod -o yaml > /tmp/pvc-after-$(date -u +%Y%m%dT%H%M%SZ).yaml

Expansion is generally one-way; you cannot shrink a PVC. If you need to revert, you must create a smaller PVC and restore data from backup. Therefore, always verify the need and available underlying storage before expanding.

Configuration snippet example: PVC with explicit size and StorageClass

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: app-data
  namespace: prod
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: standard
  resources:
    requests:
      storage: 10Gi

Apply it:

kubectl apply -f pvc.yaml

Keep the local test small

Apply one manifest, inspect the generated resources, and verify access with a temporary pod before moving to a production workload. For example:

kubectl run -it --rm test-pod --image=busybox --restart=Never -- sh
# Inside pod: df -h /mnt

Verification and Diagnostics

After any change or on a regular schedule, verify the actual state of PVs and PVCs. This section provides concrete commands and expected outputs.

Check PVC status and conditions

kubectl get pvc -n <namespace>
kubectl describe pvc <pvc-name> -n <namespace>

Look for:

• Status Bound (expected).
• Events showing ProvisioningSucceeded, Resizing, or warnings.
• Conditions FileSystemResizePending (true means the PVC was expanded at the volume level but the file system resize has not completed, often due to the pod not being restarted or the driver not supporting online resize).
• Condition Resizing (true while expansion is in progress).

Monitor actual usage over time

For a pod with PVC mounted at /data, run:

kubectl exec -it <pod> -n <namespace> -- sh -c 'df -h /data; du -sh /data/* | sort -rh | head -n 10'

This gives usage percentage and top directories. It helps identify whether data logs or application files are growing unexpectedly.

For a more systematic approach, use a Prometheus query if node exporter or kubelet metrics are available:

kubelet_volume_stats_used_bytes{namespace="<ns>",persistentvolumeclaim="<pvc>"} / kubelet_volume_stats_capacity_bytes{namespace="<ns>",persistentvolumeclaim="<pvc>"}

Set an alert when usage exceeds 80%.

Check events for storage issues

kubectl get events -n <namespace> --sort-by=.lastTimestamp | grep -i volume

Expected output includes ProvisioningSucceeded, VolumeResizeSuccessful, or failure messages like Failed to resize volume or Insufficient capacity.

Verify StatefulSet PVCs

For StatefulSets, each pod has its own PVC named <volumeClaimTemplate-name>-<statefulset-name>-<ordinal>. List them:

kubectl get pvc -n <namespace> -l app=<statefulset-label>

Use the same monitoring techniques per pod.

Diagnostic example: PVC stuck in Pending

If a PVC remains Pending, run:

kubectl describe pvc <pvc-name> -n <namespace>

Common reasons:

• No matching StorageClass or provisioner.
• Insufficient capacity in the backing storage system.
• Access mode not supported (e.g., requesting ReadWriteMany on a block storage class).
• Resource quota exceeded.

Fix the root cause, not the symptom. For example, if the StorageClass name has a typo, create a new PVC with the correct name, or update the deployment to use the correct claim.

Failure Modes and Recovery

This section details common failures with PV capacity and how to recover.

Failure 1: PVC expansion hangs

Symptom: kubectl get pvc shows Resizing for a long time, or kubectl describe pvc shows an event like Waiting for user to (re-)start a pod to finish file system resize of volume on node.

Why it happens: For some volume types (e.g., certain CSI drivers or older Kubernetes versions), the file system resize only occurs when the pod using the volume is restarted. Online expansion without restart is not universally supported.

Recovery: Restart the pod that uses the PVC. For a deployment:

kubectl rollout restart deployment <deployment-name> -n <namespace>

After restart, check the PVC condition FileSystemResizePending becomes false and the new size is reflected inside the pod.

Failure 2: Disk full and pod evicted or crashing

Symptom: Pod logs show no space left on device, pod enters CrashLoopBackOff, or node reports pressure. The PVC usage is at 100%.

Why it happens: Application logs or data grew beyond the requested size, and no cleanup or expansion was done in time.

Recovery:

1. Immediately increase the PVC size if the StorageClass allows expansion (see Safe Configuration Path).
2. If expansion is impossible, free space inside the pod:

kubectl exec -it <pod> -n <namespace> -- sh -c 'du -sh /data/* | sort -rh | head'

Delete unneeded logs or tmp files, then restart the pod. This is a stopgap; implement log rotation or data archival.

1. For long-term, configure application-level retention (e.g., logrotate) or move non-critical data to object storage.

Failure 3: PVC deleted accidentally, data lost

Symptom: PVC is missing, pod cannot mount volume. The PV may still exist with status Released if reclaim policy was Retain.

Why it happens: Human error or a script deleting the wrong resource.

Recovery:

• If PV still exists with Retain, you can potentially rebind it manually. However, it is safer to restore from backup.
• If reclaim policy was Delete, the PV and underlying storage are gone. Restore from backup. This underscores the need for regular backups of stateful data. Use tools like Velero or cloud snapshots.

Check the reclaim policy before deletion:

kubectl get pv <pv-name> -o yaml | grep reclaimPolicy

Failure 4: Cannot expand because StorageClass does not allow expansion

Symptom: kubectl patch pvc returns an error like only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize.

Why it happens: The StorageClass has allowVolumeExpansion: false or is absent, or the volume is statically provisioned.

Recovery:

• Create a new larger PVC with a StorageClass that supports expansion, then copy data using a temporary pod or kubectl cp.
• Alternatively, provision a new PV manually and migrate.

Steps to migrate:

1. Create new PVC with desired size.
2. Spin up a helper pod with both PVCs mounted.
3. Copy data: kubectl cp -n <namespace> <old-pvc>:/data /tmp/data then kubectl cp -n <namespace> /tmp/data <new-pvc>:/data (adjust paths).
4. Update the deployment to use the new PVC.
5. Delete old PVC.

Always test the migration in a non-production environment first.

Failure 5: Capacity mismatch between request and actual usage

Symptom: PVC requests 100Gi but pod uses only 10Gi. Or storage costs escalate.

Why it happens: Initial sizing based on guesswork rather than metrics.

Recovery:

• PVC size cannot be reduced in-place. To right-size, you must create a new smaller PVC and migrate data (as above).
• Plan for future growth by setting a reasonable initial size and monitoring trends. Use a policy to start with 1.5x the expected peak usage, with a minimum of 5Gi for small workloads.

Operations Checklist

Use this checklist for recurring PV capacity management. Assign a single accountable owner for each major decision, and review at a defined cadence (e.g., weekly for high-growth workloads, monthly for stable).

For each checklist item, the owner records the result in a tracking system (e.g., a GitHub issue or Jira ticket) and notes any deviations. The engineering lead reviews the tracked items during the monthly infrastructure review.

Common Pitfalls and How to Avoid Them

Pitfall 1: Using the default StorageClass without checking expansion support

Many clusters set a default StorageClass (e.g., standard) that may have allowVolumeExpansion: false. Developers assume they can expand PVCs later, only to discover it is impossible.

Why it happens: The cluster was set up with minimal configuration, and no one verified expansion capabilities before deploying stateful apps.

How to avoid: Before deploying a workload, run:

kubectl get storageclass <class> -o yaml | grep allowVolumeExpansion

If false or missing, either use a different class that supports expansion or adjust expectations and plan for migration.

Pitfall 2: Ignoring file system overhead and inode exhaustion

Although a PVC is 10Gi, the file system (ext4, XFS) reserves some space for metadata. Additionally, if the application creates millions of small files, you can run out of inodes (e.g., ext4 default is one inode per 16KiB of space). The disk may show low usage by df but new files cannot be created.

Why it happens: The initial capacity planning only considered bytes, not inode requirements.

How to avoid: Check inode usage:

kubectl exec -it <pod> -n <namespace> -- df -i /data

If inode usage is high (>80%), consider a file system with more inodes or adjust the application to use fewer, larger files.

Pitfall 3: Not setting resource quotas on namespaces

Without quotas, developers can create many large PVCs, exhausting cluster storage and costs. For example, a namespace with a 100Gi quota might request 10 PVCs of 20Gi each.

Why it happens: Quotas are not configured by default, and the organization lacks a storage governance policy.

How to avoid: Define a ResourceQuota for storage:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: storage-quota
  namespace: team-a
spec:
  hard:
    requests.storage: 200Gi
    persistentvolumeclaims: "10"

Apply it and educate teams on the limits.

Pitfall 4: Assuming deletion of a PVC deletes the data

Depending on the reclaim policy (Delete vs Retain), deleting a PVC may or may not delete the underlying storage. With Retain, the PV remains and must be manually cleaned up; otherwise it leaks capacity.

Why it happens: Developers expect the default Delete policy but the StorageClass uses Retain.

How to avoid: Check the StorageClass reclaim policy before deletion:

kubectl get storageclass <class> -o yaml | grep reclaimPolicy

If Retain, explicitly delete the PV after verifying data is no longer needed or backed up.

Pitfall 5: Not planning for StatefulSet scale-down

When scaling down a StatefulSet, the PVCs of the terminated pods are not deleted automatically (unless you use volumeClaimTemplates with delete policy, but that is risky). Those orphaned PVCs consume storage.

Why it happens: Kubernetes preserves data for safety, and operators forget to clean up.

How to avoid: Define a policy for orphaned PVCs. After a scale-down, review and delete unused PVCs if data is backed up or not needed:

kubectl get pvc -n <namespace> -l app=<statefulset> | grep -v Bound

Alternatively, set the StatefulSet persistentVolumeClaimRetentionPolicy.whenDeleted: Delete if appropriate.

Conclusion

Kubernetes Persistent Volume capacity planning succeeds only when each recommendation is version-scoped, observable, and reversible where the technology permits. Copying a command without checking prerequisites and expected output is not an operations procedure. Use the commands and checks in this guide to build a baseline for your cluster.

As a next step, choose one low-risk verification for a PVC: record the current state, run a monitoring command like kubectl exec ... -- df -h, compare the result with the requested size, and schedule a review with your team. Pay attention to dependent components such as PVCs, StatefulSets, and StorageClasses; their configuration affects your ability to expand and recover.

A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. Revisit your capacity plan monthly and adjust based on actual usage trends. The payoff is a stable stateful workload and fewer 3 a.m. pages.