## Intro Kubernetes volume performance problems often hide in plain sight: a database pod that slows under load, a CI job that times out on writes, or an application that stutters only during peak hours. Without a structured approach, operators waste time guessing at StorageClass parameters, resizing PVCs, or changing node types without evidence. This guide gives you a practical, evidence-driven path from symptom to resolution. It covers the core concepts of persistent volumes (PVs), persistent volume claims (PVCs), and storage classes, then walks through observation, tuning, and verification with real commands and expected outputs. You will learn how to measure volume latency and throughput, interpret filesystem and block device signals, and safely apply changes with rollback in mind. The target audience is developers, DevOps engineers, and technical startup teams managing production Kubernetes clusters. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document recovery paths. ## Version and Environment Inventory Before tuning any volume, you must know exactly what you are running. This section shows how to capture cluster, storage provisioner, CSI driver, and kernel versions, along with the relevant storage topology, so that recommendations are reproducible. Start with a read-only inventory: ```bash kubectl version --client kubectl get nodes -o wide kubectl get storageclass kubectl get csinodes -o wide # if CSI drivers are used kubectl get pv -o wide kubectl get pvc --all-namespaces ``` Expected output for storage classes might look like: ``` NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE gp2 (default) kubernetes.io/aws-ebs Delete Immediate false 365d gp3 ebs.csi.aws.com Delete WaitForFirstConsumer true 120d fast-local kubernetes.io/no-provisioner Delete Immediate false 90d ``` Capture the provisioner and CSI driver version. For example, with AWS EBS CSI driver: ```bash kubectl -n kube-system get pods -l app=ebs-csi-controller -o jsonpath='{.items[0].spec.containers[*].image}' ``` Output: ``` public.ecr.aws/ebs-csi-driver/aws-ebs-csi-driver:v1.23.0 ``` For on-premises storage like Ceph RBD or local volumes, check the node's kernel and filesystem: ```bash uname -a df -hT /var/lib/kubelet/pods ``` Output example: ``` Linux node-3 5.15.0-102-generic #112-Ubuntu SMP ... x86_64 GNU/Linux Filesystem Type Size Used Avail Use% Mounted on /dev/nvme1n1 ext4 220G 140G 70G 67% /var/lib/kubelet/pods ``` Note the filesystem type (ext4 vs xfs) and mount options, as these affect volume performance and tuning parameters. ### Practical Check for Version and Environment - Run `kubectl get events --all-namespaces --sort-by=.lastTimestamp | tail -20` to see recent storage-related events. - For a specific PVC, check its status and events: ```bash kubectl describe pvc my-app-pvc ``` Expect `Status: Bound` and recent `ProvisioningSucceeded` event. If you see `ProvisioningFailed`, inspect the storage class and provisioner logs. - Verify the CSI driver node plugin is running on all nodes: ```bash kubectl -n kube-system get pods -l app=ebs-csi-node -o wide ``` Every node should have a running pod. Keep the environment inventory doc in the same repo as your manifests, so anyone troubleshooting later sees the exact versions. ## Safe Configuration Path Volumes are critical infrastructure; a wrong change can cause data loss or downtime. Follow a safe path: start with read-only observation, create a test PV/PVC in an isolated namespace, apply one change at a time, and always have a rollback plan. ### Step 1: Observe Current Volume Performance Use `kubectl top` for pod CPU and memory, but for storage I/O you need node-level tools. Run an ephemeral debug pod with `fio` (flexible I/O tester) against an existing volume. Example: First, create a PVC for testing, or use an existing one. If you need a new PVC: ```yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: perf-test-pvc namespace: perf-test spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi storageClassName: gp3 ``` Apply it: ```bash kubectl apply -f perf-test-pvc.yaml kubectl -n perf-test get pvc perf-test-pvc ``` Expected output: ``` NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE perf-test-pvc Bound pvc-abcdefgh-1234-5678-ijkl-mnopqrstuvwx 10Gi RWO gp3 5s ``` Now run a fio job inside a pod using that PVC: ```yaml apiVersion: v1 kind: Pod metadata: name: fio-tester namespace: perf-test spec: containers: - name: fio image: ljishen/fio:latest command: ["sleep", "3600"] volumeMounts: - mountPath: /test-volume name: test-volume volumes: - name: test-volume persistentVolumeClaim: claimName: perf-test-pvc ``` Apply and exec into the pod: ```bash kubectl apply -f fio-pod.yaml kubectl -n perf-test exec -it fio-tester -- bash ``` Inside the pod, run a basic sequential write test: ```bash fio --name=write-test --filename=/test-volume/fio-test-file --size=1G --bs=4k --iodepth=32 --rw=write --direct=1 --ioengine=libaio --runtime=60 --time_based --group_reporting ``` Sample output: ``` write: IOPS=12.5k, BW=48.8MiB/s (51.2MB/s)(2930MiB/60001msec) slat (usec): min=2, max=1000, avg= 5.00, stdev= 3.00 clat (usec): min=100, max=5000, avg=250, stdev=100 lat (usec): 99.99th=[ 500] ``` Record these baseline numbers: IOPS, bandwidth, and latency percentiles. ### Step 2: Identify Bottleneck If latency is high (e.g., 99.99th percentile > 10ms for SSD-backed volumes), check the storage class parameters. For gp3, you can independently set IOPS and throughput. For gp2, IOPS scales with volume size. Compare your observed IOPS with the theoretical limit. For example, a gp2 volume of 100GiB has baseline 300 IOPS with burst up to 3000. If your workload consistently requires 5000 IOPS, gp2 is insufficient. Switching to gp3 with `iops=5000` might solve it, but that is a change. ### Step 3: Apply Smallest Change with Rollback Example: change storage class from gp2 to gp3 for a new PVC. You cannot change the storage class of an existing PVC directly. You must create a new PVC, copy data, and then switch the deployment. Create a new StorageClass: ```yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: gp3-high-iops provisioner: ebs.csi.aws.com parameters: type: gp3 iops: "5000" throughput: "500" volumeBindingMode: WaitForFirstConsumer allowVolumeExpansion: true ``` Apply it and create a new PVC using this class. Then migrate data using `rsync` or a tool like `velero`. To rollback, delete the new PVC and keep the old one untouched. Always take a snapshot before migration: ```bash kubectl -n myapp create -f volume-snapshot.yaml ``` Snapshot example: ```yaml apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshot metadata: name: myapp-snapshot-before-migration spec: volumeSnapshotClassName: ebs-snapclass source: persistentVolumeClaimName: myapp-pvc ``` ### Safe Configuration Checklist - [ ] Read-only observation completed and baseline recorded. - [ ] Test PVC in isolated namespace works as expected. - [ ] Snapshot or backup exists before any change. - [ ] Change is limited to a new resource, not modifying existing one. - [ ] Rollback procedure documented and tested. ## Verification and Diagnostics After applying a configuration change, you must verify that performance improved and no regression occurred. Use the same fio test and compare metrics. ### Baseline and After-Change Comparison Run the fio test with identical parameters on the new volume. Compare IOPS, bandwidth, and latency. Example before and after: | Metric | Baseline gp2 100GiB | After gp3 high-iops | |--------|---------------------|---------------------| | Write IOPS (4k random) | 2500 | 5200 | | Write bandwidth (sequential) | 150 MB/s | 480 MB/s | | 99.99th percentile latency | 12 ms | 4 ms | If the new values meet your application requirements, proceed. If not, investigate further. ### Real Workload Monitoring Synthetic benchmarks do not tell the whole story. Monitor actual pod behavior. Check pod CPU, memory, and filesystem usage: ```bash kubectl top pod my-database-pod ``` Output: ``` NAME CPU(cores) MEMORY(bytes) my-database-pod 1200m 1.5Gi ``` For storage I/O, if your node has `iotop` or `pidstat`, you can get per-process I/O inside the pod. Alternatively, use `kubectl exec` to run `iostat` if the pod image has it. Many base images do not, so consider using a sidecar or node-level tools. Node-level I/O usage: ```bash ssh node-1 'iostat -x 1 10' ``` Output excerpt: ``` Device r/s w/s rkB/s wkB/s await %util nvme1n1 1200.00 500.00 12000.00 8000.00 2.50 55.00 ``` High `%util` near 100% indicates saturation. High `await` (>10ms for SSD) indicates latency issues. ### Diagnostics Commands for Specific Provisioners - **AWS EBS CSI driver**: Check controller logs for provisioning errors: ```bash kubectl -n kube-system logs deployment/ebs-csi-controller -c ebs-plugin --tail=50 ``` - **Ceph RBD**: Check if RBD client is connected and performance using `rbd perf image` on the node. - **Local volumes**: Compare performance between nodes; if one node is slower, check disk health with `smartctl -a /dev/sdX`. ### Verification Checklist - [ ] Baseline recorded before change. - [ ] Same test run after change under same conditions. - [ ] Improvement matches expectation; no regression in other metrics. - [ ] Real workload monitoring shows improved performance sustained over time. - [ ] No storage-related events or errors in cluster events. ## Failure Modes and Recovery Even careful changes can fail. This section covers common failure modes for volume tuning and how to recover. ### Failure: PVC stuck in Pending If a PVC does not bind, check events: ```bash kubectl describe pvc my-pvc ``` Common reasons: - No matching PV if storage class is not dynamic. - Provisioner not running. - Node selector or zone constraints not satisfied for WaitForFirstConsumer. Recovery: For dynamic provisioning, ensure provisioner pod is running. For WaitForFirstConsumer, ensure a pod is scheduled on the consumer node. If zone mismatch, adjust node affinity. ### Failure: Performance lower than expected If you switched to a higher IOPS volume but performance did not improve, consider: - Filesystem not using direct I/O for fio; set `--direct=1`. - Volume not fully attached or mounted; check `mount` inside pod. - Kubernetes pod's CPU limits throttle I/O; increase CPU limit. - Network-attached storage (e.g., EBS) uses EC2 instance bandwidth; instance type may limit throughput. Check instance EBS bandwidth limits. Recovery: Increase CPU limit, use a larger instance with more EBS bandwidth, or use instance store for ephemeral high-perf needs. ### Failure: Data integrity issue after migration If you copied data and after switch application reports errors, immediately rollback: 1. Take snapshot of new PVC before copy. 2. Switch deployment back to old PVC (keep old PVC until verified). 3. Check filesystem consistency with `fsck` (for block volumes) before remounting. Example rollback command: ```bash kubectl -n myapp set volume deployment/my-app --add --name=old-volume --mount-path=/data --claim-name=old-pvc ``` Then remove the new volume. ### Recovery Drill At least once, simulate a failure in a staging cluster: create a PVC, snapshot it, delete the PVC, restore from snapshot, and verify data. Document the exact steps. ## Operations Checklist Use this checklist for any volume performance tuning task. Print it, share it, and adapt to your environment. ### Preparation - [ ] Identify the application, its storage requirements (IOPS, throughput, latency). - [ ] Record current storage class, PV, PVC names and parameters. - [ ] Take a backup or snapshot of the volume if possible. - [ ] Create a test namespace and PVC for benchmarking (not production data). - [ ] Notify stakeholders about potential brief performance impact. ### Execution - [ ] Run baseline benchmark with fio or application-specific load test. - [ ] Analyze bottleneck: storage class, node type, filesystem, network, CPU. - [ ] Determine smallest change: new storage class, increased IOPS, different provisioner, etc. - [ ] Apply change to a new resource, not existing. - [ ] Migrate data if needed, verifying checksums. - [ ] Switch workload to new volume gradually (e.g., one replica at a time). ### Verification - [ ] Run benchmark again under same conditions. - [ ] Compare metrics; confirm improvement. - [ ] Monitor real workload for at least 24 hours (or one full business cycle). - [ ] Check cluster events for storage errors. - [ ] Document new configuration and performance numbers. ### Rollback Plan - [ ] Keep old PVC and PV until new volume proven stable. - [ ] Know how to switch deployment back to old volume (kubectl set volume command). - [ ] Have snapshot/restore tested. - [ ] Communicate rollback criteria to team. ### Continuous Monitoring - [ ] Set up alerts for storage metrics: PV usage above 80%, high latency, high I/O wait. - [ ] Use Prometheus metrics from kubelet (kubelet_volume_stats_*) to track volume usage and performance. - [ ] Schedule regular performance audits, especially after cluster upgrades. ## Conclusion Kubernetes volume performance tuning is not a one-time fix but an ongoing process of measurement, analysis, and controlled change. By following the structured approach in this guide, you avoid cargo-cult tuning and make decisions based on evidence. Start with one low-risk verification: run a baseline fio test on an existing volume using the provided pod, record the metrics, and compare with expected performance for your storage class. Then, if needed, make a small, reversible change, and verify the outcome with real workload monitoring. Reliable performance tuning makes failure visible, protects data, limits changes to intended resources, and defines recovery before an incident forces the decision. Your future self, and your on-call rotation, will thank you.