Kubernetes Volumes Architecture Explained with Practical Examples

Intro

Kubernetes volumes provide storage that outlives individual containers and pods. Understanding how volumes, PersistentVolumes (PVs), PersistentVolumeClaims (PVCs), and StorageClasses interact is essential for running stateful workloads reliably. This article walks through the architecture with practical examples, commands, and troubleshooting steps to help you move from a problem to a verified solution.

We focus on operational safety: observe before changing, limit the blast radius, avoid hard-coded secrets, verify each step, and document recovery paths. Whether you are a developer, DevOps consultant, or part of a startup team, this guide connects core concepts to real commands, expected outputs, failure signals, and recovery decisions.

Core Components and Data Flow

Kubernetes volumes are not just disk mounts; they are a multi-component system. The main building blocks are:

• Volume: A directory accessible to containers in a pod, defined in the pod spec. It can be backed by many sources: emptyDir, hostPath, cloud disks, NFS, and more.
• PersistentVolume (PV): A cluster-level storage resource provisioned by an admin or dynamically via a StorageClass. It has a lifecycle independent of any pod.
• PersistentVolumeClaim (PVC): A user's request for storage. It specifies size, access modes, and optionally a StorageClass. The control plane binds a PVC to a matching PV.
• StorageClass: Defines the provisioner and parameters for dynamic volume provisioning. It lets users request storage without knowing the underlying infrastructure.
• Container Storage Interface (CSI): A standard for exposing storage systems to container orchestrators. Most modern drivers are CSI-based.

The data flow for dynamic provisioning is:

1. A user creates a PVC with a StorageClass name.
2. The StorageClass's provisioner creates a PV that matches the request.
3. Kubernetes binds the PVC to the newly created PV.
4. The pod using the PVC mounts the PV into its containers via the node's kubelet.

For static provisioning, an admin pre-creates PVs, and PVCs bind to the first matching PV.

Version and Environment Inventory

Before working with volumes, confirm your Kubernetes version and storage drivers. Use kubectl version --short or kubectl version to get client and server versions. For example:

$ kubectl version --short
Client Version: v1.28.2
Server Version: v1.28.4

Check which CSI drivers are installed in your cluster. Most managed clusters have cloud-specific drivers. You can list CSI drivers with:

kubectl get csidrivers

Example output:

NAME                       ATTACHREQUIRED   PODINFOONMOUNT   STORAGECAPACITY   TOKENREQUESTS   REQUIRESREPUBLISH
pd.csi.storage.gke.io     true             true             true              <unset>         false

Verify the default StorageClass, which is used when a PVC does not specify one:

kubectl get storageclass

You might see:

NAME                 PROVISIONER                RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
standard (default)   kubernetes.io/gce-pd      Delete          WaitForFirstConsumer   true                   12d

If no default is set, mark one with:

kubectl patch storageclass standard -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

Always capture current state before changes. For example, record existing PVCs:

kubectl get pvc -n myapp

Safe Configuration Path

A safe configuration path means applying changes incrementally and verifying each step. Start with a simple pod that uses an emptyDir volume to understand basic mounts.

Create test-volume.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: test-volume
spec:
  containers:
  - name: busybox
    image: busybox:1.36
    command: ['sh', '-c', 'echo hello > /data/message.txt && sleep 3600']
    volumeMounts:
    - name: data
      mountPath: /data
  volumes:
  - name: data
    emptyDir: {}

Apply and verify:

kubectl apply -f test-volume.yaml
kubectl get pod test-volume

Once running, exec to confirm the write:

kubectl exec test-volume -- cat /data/message.txt

Expected output: hello.

Now move to persistent storage. Create a PVC and a pod that uses it. First, define pvc.yaml:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  # storageClassName: standard  # optional if default is set

Apply and check status:

kubectl apply -f pvc.yaml
kubectl get pvc my-pvc

If the PVC is pending, it may be due to no matching PV or StorageClass. Describe it for events:

kubectl describe pvc my-pvc

Once bound, create a pod using the PVC in pod-with-pvc.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: pod-with-pvc
spec:
  containers:
  - name: app
    image: nginx:1.25
    volumeMounts:
    - name: data
      mountPath: /usr/share/nginx/html
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: my-pvc

Apply and verify:

kubectl apply -f pod-with-pvc.yaml
kubectl get pod pod-with-pvc

After verification, delete the test resources:

kubectl delete pod pod-with-pvc test-volume
kubectl delete pvc my-pvc

Note: because the default reclaim policy for dynamically provisioned volumes is usually Delete, the PV is removed. If you need to retain data, set persistentVolumeReclaimPolicy: Retain on the PV.

Verification and Diagnostics

When volumes fail, use a systematic approach. Start with read-only observations:

kubectl get pods -o wide
kubectl describe pod <pod-name>
kubectl logs <pod-name> --previous

For example, if a pod is stuck in ContainerCreating, describe it:

kubectl describe pod my-app-7c8b9d6f4-abcde

Look for events like:

Warning  FailedAttachVolume  2m42s  attachdetach-controller  AttachVolume.Attach failed for volume "pvc-123..." : timed out waiting for the condition

This often indicates a problem with the storage backend or node. Check node status:

kubectl get nodes

If the node is NotReady, inspect it:

kubectl describe node <node-name>

Verify the PVC is bound:

kubectl get pvc

If the PVC is Pending, describe it:

kubectl describe pvc my-pvc

Common reasons:

• No StorageClass provisioner available.
• Requested size exceeds quota.
• Access mode mismatch.
• StorageClass has WaitForFirstConsumer and no pod scheduled.

You can also check PV details:

kubectl get pv
kubectl describe pv <pv-name>

Use kubectl get events --sort-by=.metadata.creationTimestamp to see recent cluster events.

For mounted volumes, test write/read inside the pod:

kubectl exec <pod-name> -- sh -c 'echo test > /data/testfile && cat /data/testfile'

If the pod crashes, use kubectl logs <pod-name> --previous to see previous container logs.

Failure Modes and Recovery

Understanding common failure modes helps recovery.

1. PVC stuck in Pending

Cause: No matching PV or provisioner not working.

Diagnosis:

kubectl describe pvc my-pvc

If events show no persistent volumes available for this claim and no storage class is set, you may need to create a PV or set a default StorageClass.

Recovery: Create a PV manually if static provisioning is intended, or define a StorageClass and reference it in the PVC.

2. Pod stuck in ContainerCreating with volume errors

Cause: Volume cannot be attached or mounted (e.g., cloud disk in wrong zone, node missing driver).

Diagnosis:

kubectl describe pod <pod-name>

Look for events like FailedMount or FailedAttachVolume.

Recovery:

• If using zonal disks, ensure the pod is scheduled in the same zone as the disk. Use node affinity or WaitForFirstConsumer binding mode.
• For CSI drivers, verify driver pods are running: kubectl get pods -n kube-system | grep csi.
• Check node has the required mount utilities.

3. Data not persisting after pod deletion

Cause: The volume was emptyDir or the PVC reclaim policy is Delete.

Diagnosis: Inspect PV reclaim policy:

kubectl get pv <pv-name> -o jsonpath='{.spec.persistentVolumeReclaimPolicy}'

Recovery: For data retention, set persistentVolumeReclaimPolicy: Retain on the PV. After PVC deletion, the PV remains and can be manually reclaimed.

4. Multi-node access issues

Cause: Using ReadWriteOnce on a volume that needs multiple nodes.

Diagnosis: Check accessModes on PVC and PV:

kubectl get pvc my-pvc -o jsonpath='{.spec.accessModes}'

Recovery: Use a volume that supports ReadWriteMany (e.g., NFS, Azure Files, Amazon EFS) and update accessModes accordingly. Note that you cannot change accessModes of a bound PVC; you need to recreate it.

5. Disk full or quota exceeded

Cause: Application writes too much data.

Diagnosis: Check disk usage inside pod: kubectl exec <pod> -- df -h /mountpoint.

Recovery: Expand the volume if allowVolumeExpansion is true in StorageClass:

kubectl patch pvc my-pvc -p '{"spec":{"resources":{"requests":{"storage":"5Gi"}}}}'

Then verify with kubectl get pvc.

Always document recovery steps and test them in a non-production environment before an incident.

Operations Checklist

Use this checklist for safe volume operations:

• [ ] Check Kubernetes version and storage drivers: kubectl version and kubectl get csidrivers.
• [ ] Verify default StorageClass: kubectl get storageclass.
• [ ] Record existing PVC/PV state: kubectl get pvc,pv -A.
• [ ] Test volume mounting with a simple pod (emptyDir) before moving to persistent volumes.
• [ ] Create PVC with appropriate accessMode and size, and specify StorageClass explicitly to avoid ambiguity.
• [ ] For production, use WaitForFirstConsumer binding mode to ensure pod scheduling is considered.
• [ ] Apply pod config and verify status: kubectl get pod <name>.
• [ ] Check events and logs on failure: kubectl describe pod <name>, kubectl logs <name> --previous.
• [ ] Test data persistence: write a file, delete pod, recreate, and read file.
• [ ] Set reclaim policy to Retain if data must survive PVC deletion.
• [ ] Monitor volume capacity and expand if needed.
• [ ] Before deleting PVC, ensure no pod is using it: kubectl get pvc -o wide.
• [ ] Document recovery runbooks for common failures.

Example of a production PVC definition:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: app-data
  namespace: production
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: fast-ssd
  resources:
    requests:
      storage: 20Gi

And the corresponding StorageClass:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-ssd
reclaimPolicy: Retain
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer

Apply both, then create a Deployment that uses the PVC. Always verify the rollout:

kubectl apply -f storageclass.yaml
kubectl apply -f pvc.yaml
kubectl apply -f deployment.yaml
kubectl rollout status deployment/my-app

Conclusion

Kubernetes volumes architecture is powerful but requires careful configuration and verification. By understanding PVs, PVCs, and StorageClasses, and by following a systematic approach to testing and troubleshooting, you can avoid common pitfalls and ensure your stateful applications run reliably.

Start with a low-risk verification: create a test pod with an emptyDir volume, then progress to persistent storage. Record current state, run documented checks, and compare results with expected outcomes. Always limit changes to the intended resource and document recovery steps before an incident forces a decision.

With the commands and examples in this article, you should be able to diagnose volume issues, recover from failures, and operate Kubernetes storage with confidence.