## Intro Kubernetes volumes are a common source of frustration. A pod may stay in ContainerCreating, an application may crash with permission denied, or data may disappear after a restart. Troubleshooting requires a systematic approach: identify the version and environment, understand the volume lifecycle, inspect the right Kubernetes objects, and verify fixes with precise commands and expected outputs. This guide provides practical, command-driven troubleshooting for Kubernetes volumes. It is written for developers, DevOps engineers, and technical startup teams who manage Kubernetes clusters and need to resolve volume issues quickly. We cover PersistentVolumes (PVs), PersistentVolumeClaims (PVCs), StorageClasses, and common volume types like hostPath, emptyDir, and cloud provider disks. Throughout, we emphasize operational safety: observe before changing, limit the blast radius, and verify results. ## Version and Environment Inventory Before troubleshooting, gather information about the Kubernetes cluster and the storage subsystem. **Check Kubernetes version:** ```bash kubectl version --short ``` Example output: ``` Client Version: v1.26.5 Server Version: v1.26.7 ``` **Check cluster nodes and any volume-related labels:** ```bash kubectl get nodes -o wide ``` Note any nodes with specific zone or region labels that may affect volume attachment. **Identify the storage provider.** For cloud-based clusters, determine the CSI driver in use: ```bash kubectl get csidrivers ``` Example: ``` NAME ATTACH PODINFOONMOUNT STORAGECAPACITY pd.csi.storage.gke.io true true true ``` For on-premises, check if a local provisioner or NFS is configured by listing StorageClasses (see below). **Review existing StorageClasses:** ```bash kubectl get storageclass ``` Output may show classes like `standard (default)`, `premium-rwo`, etc. Check the default: ```bash kubectl get storageclass -o jsonpath='{.items[?(@.metadata.annotations.storageclass\.kubernetes\.io/is-default-class=="true")].metadata.name}' ``` **Check existing PVs and PVCs:** ```bash kubectl get pv,pvc -A ``` This shows all volumes and claims, their status, capacity, and bound status. **Prerequisites for effective troubleshooting:** - Cluster access with `kubectl` configured. - Permissions to view and describe storage resources (typically cluster-admin or a role with get/list/watch on persistentvolumes, persistentvolumeclaims, storageclasses, pods, events). - Understanding of how storage is provisioned in your environment (static vs dynamic). For initial observation, avoid making changes. Start with read-only commands: ```bash kubectl get events -n --sort-by='.lastTimestamp' ``` Events often contain clues about volume failures. For a specific pod: ```bash kubectl describe pod -n ``` ## Understanding the Volume Lifecycle Kubernetes storage has three main objects: 1. **PersistentVolume (PV):** A cluster resource representing physical storage (e.g., AWS EBS volume, GCE PD, hostPath). 2. **PersistentVolumeClaim (PVC):** A request for storage by a user, specifying size and access modes. 3. **StorageClass:** Defines provisioner parameters and reclaim policy; enables dynamic provisioning. The lifecycle in simple terms: - A PVC is created; the control plane finds or dynamically provisions a matching PV. - The PVC binds to a PV. - A pod mounts the PVC. - When the pod is deleted, the PVC remains (unless the reclaim policy deletes it). Troubleshooting often involves determining where in this lifecycle the failure occurs. ## Common Volume Types and Failure Patterns ### hostPath Used for development and single-node testing. Files are stored on the node. If the node dies, data may be lost. Pods using hostPath may not be rescheduled to other nodes (unless using `nodeSelector` or affinity). Not suitable for production. **Typical issues:** - Path does not exist on node. - Permission denied because pod runs as non-root but hostPath directory owned by root. - Data persistence across pod restarts on same node, but not across node reboot. ### emptyDir Temporary storage scoped to a pod's lifetime. Created when pod assigned to node, deleted when pod removed. Used for scratch space, caching. **Issues:** - Data lost if pod evicted or node drained. - Exceeding node's ephemeral storage limit triggers eviction. ### Network File System (NFS) Allows multiple pods to read/write simultaneously across nodes. Requires NFS server and client support. Common in legacy environments not using dynamic provisioning. **Issues:** - Mount fails due to missing NFS packages or firewall blocking ports 111 and 2049. - Performance issues. ### Cloud Provider Volumes (EBS, GCE PD, Azure Disk) Persistent block storage in cloud, typically dynamically provisioned via CSI drivers. Support ReadWriteOnce only (can be mounted on one node at a time), except for some new capabilities. **Issues:** - Volume stuck in `Attaching` state due to zone mismatch between node and volume. - Pod remains `ContainerCreating` for a long time, then times out. - Node failure leading to volume not detaching. - Cost: orphaned volumes after PVC deletion if reclaim policy is not set correctly. ## Step-by-Step Troubleshooting Procedures We'll follow a pod stuck in `ContainerCreating` with volume issues. ### Step 1: Identify the Pod and Its Volumes ```bash kubectl get pods -n ``` Find the problematic pod (e.g., `my-app-6948c7d9d-xxxxx`). Describe the pod: ```bash kubectl describe pod my-app-6948c7d9d-xxxxx -n ``` Look under `Volumes:` and `Conditions:` sections. The events at the bottom often reveal errors like: ``` Warning FailedMount 2m kubelet Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[data default-token-xxxxx]: timed out waiting for the condition ``` This indicates the volume `data` could not be mounted. Note the volume name and type. ### Step 2: Inspect the PVC and PV From pod description, identify the claim name. Then: ```bash kubectl describe pvc -n ``` Check `Status`, `Volume`, `Capacity`, `Access Modes`, and events. If PVC is `Pending`, no PV is bound. See troubleshooting PVC pending below. If bound, get the PV name from `Volume:` and describe it: ```bash kubectl describe pv ``` Look for status, reclaim policy, and any messages about availability. ### Step 3: Check StorageClass and Provisioner If dynamic provisioning, identify the StorageClass used by the PVC (if any): ```bash kubectl get pvc -o jsonpath='{.spec.storageClassName}' ``` Then describe the StorageClass: ```bash kubectl describe storageclass ``` Check if provisioner is valid and there are any parameters that may cause failure. ### Step 4: Examine Events ```bash kubectl get events -n --field-selector involvedObject.name= --sort-by='.lastTimestamp' ``` Events often contain precise error messages from the volume plugin or CSI driver. ### Step 5: Check Kubelet Logs (if node access) On the node where pod is scheduled (from `kubectl get pod -o wide`), examine kubelet logs for volume mount failures. For systemd-based nodes: ```bash journalctl -u kubelet -f | grep -i volume ``` ## Common Failure Modes and Recovery ### PVC Stuck in Pending **Possible causes:** - No StorageClass specified and no default StorageClass set. - No available PV matching request. - Provisioner not running or misconfigured. - Insufficient capacity. **Diagnostics:** ```bash kubectl describe pvc ``` Look for events like: ``` Warning ProvisioningFailed 3m persistentvolume-controller storageclass.storage.k8s.io "standard" not found ``` **Recovery:** - If using dynamic provisioning, ensure StorageClass exists and is default if not specified. - If static provisioning, create a PV with matching capacity and access modes. - Check provisioner pod status (for CSI): `kubectl get pods -n kube-system | grep csi`. **Example fix:** Create a default StorageClass: ```yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: standard annotations: storageclass.kubernetes.io/is-default-class: "true" provisioner: kubernetes.io/no-provisioner volumeBindingMode: WaitForFirstConsumer ``` Note: `no-provisioner` requires manual PV creation, but it demonstrates setting default. For production, use a proper provisioner. ### Pod Stuck in ContainerCreating with FailedMount **Symptoms:** Pod not starting, describe shows `FailedMount` with reason `Unable to attach or mount volumes`. **Possible causes:** - Volume not attached to node (e.g., zone mismatch). - Volume already mounted by another node (ReadWriteOnce conflict). - Mount options invalid. - Permission issues on hostPath. **Diagnostics:** - Check node where pod assigned and volume's availability zone (for cloud volumes). - For cloud volumes, use cloud CLI to inspect volume status (e.g., `aws ec2 describe-volumes`, `gcloud compute disks describe`). - Look for events indicating `Multi-Attach error`. **Recovery:** - Ensure node and volume in same zone. This may require moving pod to correct node or creating volume in correct zone. - If volume stuck attached to a dead node, force detach: Use cloud provider console or CLI to detach volume. - For hostPath, ensure path exists and permissions allow container user. **Example: Force detach EBS volume from AWS:** ```bash aws ec2 detach-volume --volume-id vol-1234567890abcdef0 --force ``` Then wait for state `available`. ### Permission Denied Errors Often seen with hostPath or NFS where server enforces root_squash or user IDs mismatch. **Diagnostics:** - Pod logs may show `Permission denied`. - Running `ls -l` inside container with `kubectl exec` to check ownership. **Recovery:** - For hostPath, either change permissions on host directory or run container as root (if acceptable), or use `fsGroup` in pod security context. - For NFS, configure exports with `no_root_squash` or adjust user IDs. **Example: Set fsGroup in pod spec:** ```yaml spec: securityContext: fsGroup: 2000 ``` This changes group ownership of volume files to GID 2000. ### Data Loss or Corruption **Symptoms:** Application reports missing files, database corruption. **Possible causes:** - Pod running with multiple replicas sharing ReadWriteOnce volume (only one node can mount; the other fails or data corruption if filesystem not designed for concurrent access). - Reclaim policy set to `Delete` and PVC deleted accidentally. - Node failure causing filesystem inconsistency. **Recovery:** - Always use appropriate access mode: `ReadWriteMany` for shared storage (NFS, GlusterFS). - Set reclaim policy to `Retain` for important volumes to prevent automatic deletion. - Take regular snapshots (e.g., CSI snapshots). - If using block storage for databases, ensure pod is rescheduled on same node (use node affinity) to avoid multi-node corruption. **Example snapshot (using VolumeSnapshot API):** ```yaml apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshot metadata: name: my-snapshot spec: volumeSnapshotClassName: csi-hostpath-snapclass source: persistentVolumeClaimName: my-pvc ``` ### Volume Expansion Issues Expanding a PVC can fail if StorageClass does not allow expansion or CSI driver does not support it. **Check if StorageClass allows expansion:** ```bash kubectl get storageclass -o jsonpath='{.allowVolumeExpansion}' ``` If not `true`, cannot expand. **Recovery:** Either create new PVC with larger size and migrate data, or enable expansion by editing StorageClass (if driver supports). ## Common Pitfalls and How to Avoid Them 1. **Ignoring Access Modes:** A PVC with `ReadWriteOnce` cannot be mounted by multiple nodes. Deployments with multiple replicas need `ReadWriteMany` or use topologySpreadConstraints/affinity to schedule on same node. **Avoid:** Review access modes for your workload. 2. **Not Setting a Default StorageClass:** In many clusters, if no default StorageClass, PVCs without `storageClassName` stay pending. **Avoid:** Always specify `storageClassName` explicitly or set a default. 3. **Using hostPath in Production:** hostPath ties pod to a node and can cause permission issues and security risks. **Avoid:** Use dynamic provisioning or local persistent volumes for node-affine storage. 4. **Forgetting Volume Snapshots and Backups:** Kubernetes does not automatically back up volumes. Data loss is common after deletion. **Avoid:** Implement snapshotting and backup strategies. 5. **Reclaim Policy Misconfiguration:** Default reclaim policy for dynamically provisioned volumes is usually `Delete`. If you delete a PVC, the underlying volume is deleted. **Avoid:** Set `Retain` for critical data, and manually clean up volumes. 6. **Zone Mismatch in Cloud:** Creating a StorageClass with `volumeBindingMode: Immediate` may provision a volume in a random zone, then pod scheduled in another zone can't attach. **Avoid:** Use `WaitForFirstConsumer` to delay provisioning until pod is scheduled, ensuring zone alignment. ## Operations Checklist For each troubleshooting session, follow this checklist: - [ ] **Verify cluster version and storage drivers:** `kubectl version --short`, `kubectl get csidrivers`. Record outputs. - [ ] **List all storage resources:** `kubectl get pv,pvc,sc -A`. Check statuses. - [ ] **Describe the failing pod:** `kubectl describe pod -n `. Note volume mounts and events. - [ ] **Describe the PVC:** `kubectl describe pvc -n `. Note phase and events. - [ ] **Describe the PV (if bound):** `kubectl describe pv `. Check reclaim policy and capacity. - [ ] **Check events sorted by time:** `kubectl get events -n --sort-by='.lastTimestamp'`. Look for warnings. - [ ] **Validate node affinity and zones:** `kubectl get nodes -o wide` and cloud volume zone info. Ensure pod node and volume zone match. - [ ] **Check kubelet logs on the node:** `journalctl -u kubelet -f` while reproducing the mount attempt. - [ ] **Test with a minimal pod:** Create a simple pod using the same PVC to isolate app-specific issues. - [ ] **After applying fix, verify:** `kubectl get pod -n ` shows Running, and `kubectl exec` into pod to verify file access (`ls -l /mount/path`). - [ ] **Document the incident:** Record commands run, findings, and fix applied for future reference. ## Conclusion Kubernetes volume troubleshooting requires a methodical approach: understand the environment, identify the component at fault, use precise diagnostics, and apply minimal fixes with verification. The commands and examples in this guide provide a foundation. Adapt them to your specific storage provider and cluster configuration. Remember to prioritize safety: never delete volumes or change reclaim policies without understanding the consequences. Use snapshots and backups to prevent data loss. With practice, you'll resolve volume issues efficiently and keep your applications running smoothly. ## Additional Resources While this guide covers common scenarios, refer to the official Kubernetes documentation for in-depth information: - [Persistent Volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) - [Storage Classes](https://kubernetes.io/docs/concepts/storage/storage-classes/) - [Volume Snapshots](https://kubernetes.io/docs/concepts/storage/volume-snapshots/) Also, consult your cloud provider's documentation for CSI driver specifics and troubleshooting tools.