Kubernetes Volumes Troubleshooting: A Practical Field Guide

Intro

Kubernetes volumes are a common source of frustration. A pod may stay in ContainerCreating, an application may crash with permission denied, or data may disappear after a restart. Troubleshooting requires a systematic approach: identify the version and environment, understand the volume lifecycle, inspect the right Kubernetes objects, and verify fixes with precise commands and expected outputs.

This guide provides practical, command-driven troubleshooting for Kubernetes volumes. It is written for developers, DevOps engineers, and technical startup teams who manage Kubernetes clusters and need to resolve volume issues quickly. We cover PersistentVolumes (PVs), PersistentVolumeClaims (PVCs), StorageClasses, and common volume types like hostPath, emptyDir, and cloud provider disks. Throughout, we emphasize operational safety: observe before changing, limit the blast radius, and verify results.

Version and Environment Inventory

Before troubleshooting, gather information about the Kubernetes cluster and the storage subsystem.

Check Kubernetes version:

kubectl version --short

Example output:

Client Version: v1.26.5
Server Version: v1.26.7

Check cluster nodes and any volume-related labels:

kubectl get nodes -o wide

Note any nodes with specific zone or region labels that may affect volume attachment.

Identify the storage provider. For cloud-based clusters, determine the CSI driver in use:

kubectl get csidrivers

Example:

NAME                     ATTACH    PODINFOONMOUNT   STORAGECAPACITY
pd.csi.storage.gke.io    true      true             true

For on-premises, check if a local provisioner or NFS is configured by listing StorageClasses (see below).

Review existing StorageClasses:

kubectl get storageclass

Output may show classes like standard (default), premium-rwo, etc. Check the default:

kubectl get storageclass -o jsonpath='{.items[?(@.metadata.annotations.storageclass\.kubernetes\.io/is-default-class=="true")].metadata.name}'

Check existing PVs and PVCs:

kubectl get pv,pvc -A

This shows all volumes and claims, their status, capacity, and bound status.

Prerequisites for effective troubleshooting:

• Cluster access with kubectl configured.
• Permissions to view and describe storage resources (typically cluster-admin or a role with get/list/watch on persistentvolumes, persistentvolumeclaims, storageclasses, pods, events).
• Understanding of how storage is provisioned in your environment (static vs dynamic).

For initial observation, avoid making changes. Start with read-only commands:

kubectl get events -n <namespace> --sort-by='.lastTimestamp'

Events often contain clues about volume failures. For a specific pod:

kubectl describe pod <pod-name> -n <namespace>

Understanding the Volume Lifecycle

Kubernetes storage has three main objects:

1. PersistentVolume (PV): A cluster resource representing physical storage (e.g., AWS EBS volume, GCE PD, hostPath).
2. PersistentVolumeClaim (PVC): A request for storage by a user, specifying size and access modes.
3. StorageClass: Defines provisioner parameters and reclaim policy; enables dynamic provisioning.

The lifecycle in simple terms:

• A PVC is created; the control plane finds or dynamically provisions a matching PV.
• The PVC binds to a PV.
• A pod mounts the PVC.
• When the pod is deleted, the PVC remains (unless the reclaim policy deletes it).

Troubleshooting often involves determining where in this lifecycle the failure occurs.

Common Volume Types and Failure Patterns

hostPath

Used for development and single-node testing. Files are stored on the node. If the node dies, data may be lost. Pods using hostPath may not be rescheduled to other nodes (unless using nodeSelector or affinity). Not suitable for production.

Typical issues:

• Path does not exist on node.
• Permission denied because pod runs as non-root but hostPath directory owned by root.
• Data persistence across pod restarts on same node, but not across node reboot.

emptyDir

Temporary storage scoped to a pod's lifetime. Created when pod assigned to node, deleted when pod removed. Used for scratch space, caching.

Issues:

• Data lost if pod evicted or node drained.
• Exceeding node's ephemeral storage limit triggers eviction.

Network File System (NFS)

Allows multiple pods to read/write simultaneously across nodes. Requires NFS server and client support. Common in legacy environments not using dynamic provisioning.

Issues:

• Mount fails due to missing NFS packages or firewall blocking ports 111 and 2049.
• Performance issues.

Cloud Provider Volumes (EBS, GCE PD, Azure Disk)

Persistent block storage in cloud, typically dynamically provisioned via CSI drivers. Support ReadWriteOnce only (can be mounted on one node at a time), except for some new capabilities.

Issues:

• Volume stuck in Attaching state due to zone mismatch between node and volume.
• Pod remains ContainerCreating for a long time, then times out.
• Node failure leading to volume not detaching.
• Cost: orphaned volumes after PVC deletion if reclaim policy is not set correctly.

Step-by-Step Troubleshooting Procedures

We'll follow a pod stuck in ContainerCreating with volume issues.

Step 1: Identify the Pod and Its Volumes

kubectl get pods -n <namespace>

Find the problematic pod (e.g., my-app-6948c7d9d-xxxxx).

Describe the pod:

kubectl describe pod my-app-6948c7d9d-xxxxx -n <namespace>

Look under Volumes: and Conditions: sections. The events at the bottom often reveal errors like:

Warning  FailedMount  2m   kubelet  Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[data default-token-xxxxx]: timed out waiting for the condition

This indicates the volume data could not be mounted. Note the volume name and type.

Step 2: Inspect the PVC and PV

From pod description, identify the claim name. Then:

kubectl describe pvc <claim-name> -n <namespace>

Check Status, Volume, Capacity, Access Modes, and events.

If PVC is Pending, no PV is bound. See troubleshooting PVC pending below.

If bound, get the PV name from Volume: and describe it:

kubectl describe pv <pv-name>

Look for status, reclaim policy, and any messages about availability.

Step 3: Check StorageClass and Provisioner

If dynamic provisioning, identify the StorageClass used by the PVC (if any):

kubectl get pvc <claim-name> -o jsonpath='{.spec.storageClassName}'

Then describe the StorageClass:

kubectl describe storageclass <sc-name>

Check if provisioner is valid and there are any parameters that may cause failure.

Step 4: Examine Events

kubectl get events -n <namespace> --field-selector involvedObject.name=<pod-name> --sort-by='.lastTimestamp'

Events often contain precise error messages from the volume plugin or CSI driver.

Step 5: Check Kubelet Logs (if node access)

On the node where pod is scheduled (from kubectl get pod -o wide), examine kubelet logs for volume mount failures.

For systemd-based nodes:

journalctl -u kubelet -f | grep -i volume

Common Failure Modes and Recovery

PVC Stuck in Pending

Possible causes:

• No StorageClass specified and no default StorageClass set.
• No available PV matching request.
• Provisioner not running or misconfigured.
• Insufficient capacity.

Diagnostics:

kubectl describe pvc <claim-name>

Look for events like:

Warning  ProvisioningFailed  3m  persistentvolume-controller  storageclass.storage.k8s.io "standard" not found

Recovery:

• If using dynamic provisioning, ensure StorageClass exists and is default if not specified.
• If static provisioning, create a PV with matching capacity and access modes.
• Check provisioner pod status (for CSI): kubectl get pods -n kube-system | grep csi.

Example fix: Create a default StorageClass:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: standard
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

Note: no-provisioner requires manual PV creation, but it demonstrates setting default. For production, use a proper provisioner.

Pod Stuck in ContainerCreating with FailedMount

Symptoms: Pod not starting, describe shows FailedMount with reason Unable to attach or mount volumes.

Possible causes:

• Volume not attached to node (e.g., zone mismatch).
• Volume already mounted by another node (ReadWriteOnce conflict).
• Mount options invalid.
• Permission issues on hostPath.

Diagnostics:

• Check node where pod assigned and volume's availability zone (for cloud volumes).
• For cloud volumes, use cloud CLI to inspect volume status (e.g., aws ec2 describe-volumes, gcloud compute disks describe).
• Look for events indicating Multi-Attach error.

Recovery:

• Ensure node and volume in same zone. This may require moving pod to correct node or creating volume in correct zone.
• If volume stuck attached to a dead node, force detach: Use cloud provider console or CLI to detach volume.
• For hostPath, ensure path exists and permissions allow container user.

Example: Force detach EBS volume from AWS:

aws ec2 detach-volume --volume-id vol-1234567890abcdef0 --force

Then wait for state available.

Permission Denied Errors

Often seen with hostPath or NFS where server enforces root_squash or user IDs mismatch.

Diagnostics:

• Pod logs may show Permission denied.
• Running ls -l inside container with kubectl exec to check ownership.

Recovery:

• For hostPath, either change permissions on host directory or run container as root (if acceptable), or use fsGroup in pod security context.
• For NFS, configure exports with no_root_squash or adjust user IDs.

Example: Set fsGroup in pod spec:

spec:
  securityContext:
    fsGroup: 2000

This changes group ownership of volume files to GID 2000.

Data Loss or Corruption

Symptoms: Application reports missing files, database corruption.

Possible causes:

• Pod running with multiple replicas sharing ReadWriteOnce volume (only one node can mount; the other fails or data corruption if filesystem not designed for concurrent access).
• Reclaim policy set to Delete and PVC deleted accidentally.
• Node failure causing filesystem inconsistency.

Recovery:

• Always use appropriate access mode: ReadWriteMany for shared storage (NFS, GlusterFS).
• Set reclaim policy to Retain for important volumes to prevent automatic deletion.
• Take regular snapshots (e.g., CSI snapshots).
• If using block storage for databases, ensure pod is rescheduled on same node (use node affinity) to avoid multi-node corruption.

Example snapshot (using VolumeSnapshot API):

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: my-snapshot
spec:
  volumeSnapshotClassName: csi-hostpath-snapclass
  source:
    persistentVolumeClaimName: my-pvc

Volume Expansion Issues

Expanding a PVC can fail if StorageClass does not allow expansion or CSI driver does not support it.

Check if StorageClass allows expansion:

kubectl get storageclass <sc-name> -o jsonpath='{.allowVolumeExpansion}'

If not true, cannot expand.

Recovery: Either create new PVC with larger size and migrate data, or enable expansion by editing StorageClass (if driver supports).

Common Pitfalls and How to Avoid Them

1. Ignoring Access Modes: A PVC with ReadWriteOnce cannot be mounted by multiple nodes. Deployments with multiple replicas need ReadWriteMany or use topologySpreadConstraints/affinity to schedule on same node. Avoid: Review access modes for your workload.

1. Not Setting a Default StorageClass: In many clusters, if no default StorageClass, PVCs without storageClassName stay pending. Avoid: Always specify storageClassName explicitly or set a default.

1. Using hostPath in Production: hostPath ties pod to a node and can cause permission issues and security risks. Avoid: Use dynamic provisioning or local persistent volumes for node-affine storage.

1. Forgetting Volume Snapshots and Backups: Kubernetes does not automatically back up volumes. Data loss is common after deletion. Avoid: Implement snapshotting and backup strategies.

1. Reclaim Policy Misconfiguration: Default reclaim policy for dynamically provisioned volumes is usually Delete. If you delete a PVC, the underlying volume is deleted. Avoid: Set Retain for critical data, and manually clean up volumes.

1. Zone Mismatch in Cloud: Creating a StorageClass with volumeBindingMode: Immediate may provision a volume in a random zone, then pod scheduled in another zone can't attach. Avoid: Use WaitForFirstConsumer to delay provisioning until pod is scheduled, ensuring zone alignment.

Operations Checklist

For each troubleshooting session, follow this checklist:

• [ ] Verify cluster version and storage drivers: kubectl version --short, kubectl get csidrivers. Record outputs.
• [ ] List all storage resources: kubectl get pv,pvc,sc -A. Check statuses.
• [ ] Describe the failing pod: kubectl describe pod <pod-name> -n <namespace>. Note volume mounts and events.
• [ ] Describe the PVC: kubectl describe pvc <claim-name> -n <namespace>. Note phase and events.
• [ ] Describe the PV (if bound): kubectl describe pv <pv-name>. Check reclaim policy and capacity.
• [ ] Check events sorted by time: kubectl get events -n <namespace> --sort-by='.lastTimestamp'. Look for warnings.
• [ ] Validate node affinity and zones: kubectl get nodes -o wide and cloud volume zone info. Ensure pod node and volume zone match.
• [ ] Check kubelet logs on the node: journalctl -u kubelet -f while reproducing the mount attempt.
• [ ] Test with a minimal pod: Create a simple pod using the same PVC to isolate app-specific issues.
• [ ] After applying fix, verify: kubectl get pod -n <namespace> shows Running, and kubectl exec into pod to verify file access (ls -l /mount/path).
• [ ] Document the incident: Record commands run, findings, and fix applied for future reference.

Conclusion

Kubernetes volume troubleshooting requires a methodical approach: understand the environment, identify the component at fault, use precise diagnostics, and apply minimal fixes with verification. The commands and examples in this guide provide a foundation. Adapt them to your specific storage provider and cluster configuration.

Remember to prioritize safety: never delete volumes or change reclaim policies without understanding the consequences. Use snapshots and backups to prevent data loss. With practice, you'll resolve volume issues efficiently and keep your applications running smoothly.

Additional Resources

While this guide covers common scenarios, refer to the official Kubernetes documentation for in-depth information:

• Persistent Volumes
• Storage Classes
• Volume Snapshots

Also, consult your cloud provider's documentation for CSI driver specifics and troubleshooting tools.