## Intro Kubernetes Persistent Volume (PV) networking issues can surface as slow application responses, failed mounts, or inaccessible services. Troubleshooting requires methodical observation, a clear understanding of the component interactions, and safe, reversible changes. This guide provides practical, example-driven steps to diagnose and resolve networking problems related to Persistent Volumes in Kubernetes, focusing on DNS resolution, port connectivity, and end-to-end network troubleshooting. We will walk through a realistic scenario: an application pod that cannot access a Persistent Volume served over the network, leading to intermittent write failures. You will learn how to gather version and environment details, verify configuration, perform diagnostics, identify failure modes, and follow a recovery checklist. Every step includes concrete commands, expected outputs, and decision points. The goal is operational confidence: observe before changing, limit the blast radius, use placeholders for sensitive data, and verify outcomes. This guide targets developers, DevOps consultants, and technical startup teams who manage stateful workloads in Kubernetes. While the focus is on PV networking, we touch on Persistent Volume Claims (PVC), StatefulSets, and Storage Classes only when they affect connectivity or recovery. ## Version and Environment Inventory Before diving into troubleshooting, establish the exact version and topology of your Kubernetes cluster and the storage backend. This prevents mismatched assumptions and helps you compare against known issues. Run these commands and record the output alongside a timestamp. ### Cluster and Node Information kubectl version --short kubectl get nodes -o wide Expected output is similar to: Client Version: v1.28.2 Server Version: v1.28.2 NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME node-1 Ready control-plane 30d v1.28.2 10.0.0.11 Ubuntu 22.04.3 LTS 5.15.0-86-generic containerd://1.7.11 node-2 Ready 30d v1.28.2 10.0.0.12 Ubuntu 22.04.3 LTS 5.15.0-86-generic containerd://1.7.11 Note the Kubernetes version (e.g., 1.28) and container runtime (containerd). Network plugin details matter; for example, with Calico: kubectl get pods -n kube-system | grep -E 'calico|kube-proxy' Expected output: calico-node-abcde 1/1 Running 0 30d calico-kube-controllers-12345 1/1 Running 0 30d kube-proxy-xyz1 1/1 Running 0 30d ### Storage Backend Information Identify the StorageClass and provisioner used by your PV. Suppose you use NFS for a shared storage network. List storage classes: kubectl get storageclass Expected output: NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE nfs-client (default) nfs.csi.k8s.io Delete Immediate true 60d Then inspect the PV and PVC details: kubectl get pv kubectl get pvc -n application-namespace Example output: NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-38b7f2a3-4c5d-4e6f-8a1b-9c0d1e2f3a4b 20Gi RWO Delete Bound app/data-pvc nfs-client 15d NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE data-pvc Bound pvc-38b7f2a3-4c5d-4e6f-8a1b-9c0d1e2f3a4b 20Gi RWO nfs-client 15d Record the NFS server endpoint if using NFS: kubectl describe pv pvc-38b7f2a3-4c5d-4e6f-8a1b-9c0d1e2f3a4b | grep -A5 'Source:' Expected output snippet: Source: Type: NFS (an NFS mount that lasts the lifetime of a pod) Server: 192.168.1.100 Path: /exports/data ReadOnly: false This tells you the network endpoint your pods must reach. If the NFS server IP is wrong or unreachable, all PV networking fails. ### Pod and Workload Overview Check the pods using the PVC: kubectl get pods -n application-namespace -o wide Example: NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES app-db-0 1/1 Running 4 15d 10.244.2.55 node-2 app-web-7b9c8d5f6-abcde 1/1 Running 0 15d 10.244.1.89 node-1 Notice the restarts on app-db-0 ; this may indicate intermittent storage connectivity. ### Observing Without Intervening At this stage, only run read-only commands. Avoid edits or deletions until you understand the current state. Capture the output of the following for later comparison: kubectl describe pod app-db-0 -n application-namespace kubectl logs app-db-0 -n application-namespace --tail=50 If the pod is crash-looping, use --previous : kubectl logs app-db-0 -n application-namespace --previous This inventory gives you a baseline. Now, move to safe configuration checks.

Quick check 1 of 2 What command output shows the NFS server endpoint for a PersistentVolume? (Assume the PV is using NFS type) - kubectl get pv - kubectl describe pv | grep -A5 'Source:' - kubectl get storageclass - kubectl get pvc Submit The 'kubectl describe pv' command with a grep for 'Source:' shows the NFS server and path details for the PersistentVolume, as indicated in the expected output snippet in the reference. ## Safe Configuration Path The safe configuration path ensures you make minimal, well-understood changes. For PV networking, focus on the StorageClass, PVC, and pod mount configurations. ### Inspect the StorageClass For NFS, you might have a custom StorageClass with mount options that affect network behavior. View the YAML: kubectl get storageclass nfs-client -o yaml Key fields: apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: nfs-client provisioner: nfs.csi.k8s.io parameters: server: 192.168.1.100 share: /exports mountOptions: - nfsvers=4.1 - hard - timeo=600 - retrans=2 reclaimPolicy: Delete volumeBindingMode: Immediate Check that nfsvers matches the NFS server capability. Mismatched version (e.g., server supports NFSv3 only, but client requests 4.1) causes mount failures or poor performance. Also, hard mount option means the pod will block indefinitely on network partitions; soft may be preferable for faster failure detection, but it can cause data corruption. Decide based on workload tolerance. ### PVC and Pod Mount Verification Verify the PVC has correct access modes and is bound: kubectl get pvc data-pvc -n application-namespace -o yaml Look for accessModes : accessModes: - ReadWriteOnce If your application needs shared storage (e.g., multiple pods reading/writing), you need ReadWriteMany . Using ReadWriteOnce with multiple replicas can cause only one node to mount successfully, leading to networking errors for others. Inspect the pod's volume mount and read-only flags: kubectl get pod app-db-0 -n application-namespace -o yaml | grep -A10 'volumeMounts:' Expected snippet: volumeMounts: - mountPath: /var/lib/data name: data readOnly: false Ensure mountPath matches the application's expected path and that it is not read-only when writes are required. ### Smallest Justified Change If you need to alter a configuration (e.g., add soft mount option or change NFS version), update the StorageClass with a patch, but first test in a non-production environment. Example patch to change nfsvers from 4.1 to 4.0: kubectl patch storageclass nfs-client --type='json' -p='[{"op": "replace", "path": "/mountOptions/0", "value": "nfsvers=4.0"}]' Verify the change: kubectl get storageclass nfs-client -o yaml | grep -A5 'mountOptions' New pods will use the updated option. Existing pods must be restarted to pick up changes; consider a rolling restart of the StatefulSet. ### Local Testing with port-forward Before involving external network services, test connectivity locally. For a pod that serves a network file protocol or API, use kubectl port-forward : kubectl port-forward pod/app-db-0 -n application-namespace 2049:2049 Then from another terminal, test the NFS mount using local tools: sudo mount -t nfs -o nfsvers=4.1 localhost:/exports/data /mnt/test If this fails, the issue is inside the cluster, not with external networking. ### Verifying Without Load Balancers Avoid creating a LoadBalancer service just to test. Instead, use a temporary NodePort service or port-forward. This limits exposure and cost. ## Verification and Diagnostics Now that the configuration is understood, perform active diagnostics to isolate the network issue. ### DNS Resolution Checks Many PV networking issues stem from DNS failures, especially when the NFS server is referenced by hostname. First, check the CoreDNS pods: kubectl get pods -n kube-system -l k8s-app=kube-dns Expected: NAME READY STATUS RESTARTS AGE coredns-787d4945fb-abcde 1/1 Running 0 30d coredns-787d4945fb-fghij 1/1 Running 0 30d Test DNS resolution from inside a pod. Exec into the app pod (or a debug pod) and run nslookup : kubectl exec -it app-db-0 -n application-namespace -- nslookup nfs-server.example.com Expected output: Server: 10.96.0.10 Address: 10.96.0.10#53 Name: nfs-server.example.com Address: 192.168.1.100 If you get server can't find nfs-server.example.com: NXDOMAIN , the hostname is not in DNS. Check if you need to add an entry to /etc/hosts via hostAliases in the pod spec or fix external DNS. ### Port Connectivity Tests NFS typically uses TCP/UDP port 2049. Test connectivity from the pod to the NFS server: kubectl exec -it app-db-0 -n application-namespace -- nc -zvw3 192.168.1.100 2049 If nc is not installed, use bash with /dev/tcp : kubectl exec -it app-db-0 -n application-namespace -- bash -c 'echo > /dev/tcp/192.168.1.100/2049 && echo open || echo closed' Expected success output: open . If closed, check NetworkPolicies, firewall rules, or the NFS server service status. Check for NetworkPolicies that might block egress to the NFS server: kubectl get networkpolicies -n application-namespace If a policy exists, inspect it: kubectl describe networkpolicy allow-nfs -n application-namespace Ensure the policy allows egress to 192.168.1.100/32 on port 2049. ### Endpoint and Service Checks If the storage is exposed via a Kubernetes Service (e.g., an NFS server running as a pod), verify endpoints: kubectl get endpoints nfs-service -n storage-namespace Expected: NAME ENDPOINTS AGE nfs-service 10.244.2.10:2049 20d If endpoints are empty, check the service selector matches the pod labels. ### Logs and Events Check pod events for mount errors: kubectl describe pod app-db-0 -n application-namespace | tail -30 Look for messages like: Warning FailedMount 2m (x12 over 30m) kubelet MountVolume.SetUp failed for volume "pvc-38b7f2a3..." : mount failed: exit status 32 Mounting command: mount Mounting arguments: -t nfs 192.168.1.100:/exports/data /var/lib/kubelet/pods/.../volumes/kubernetes.io~nfs/pvc-38b7f2a3... Output: mount.nfs: access denied by server while mounting 192.168.1.100:/exports/data This indicates an NFS export permission issue or incorrect client IP in the export configuration. Check kubelet logs on the node where the pod runs (node-2): journalctl -u kubelet -n 50 --no-pager | grep -i 'nfs\|mount' This may reveal network timeouts or RPC errors. ### Network Plugin Debugging If using Calico, check calico-node logs: kubectl logs -n kube-system calico-node-abcde --tail=50 | grep -i 'nfs\|drop\|deny' Look for dropped packets due to policy. You may also use calicoctl to check network policy evaluation. ### Temporary Debug Pod Run a temporary pod with network tools to isolate whether the issue is pod-specific or cluster-wide: apiVersion: v1 kind: Pod metadata: name: debug-network namespace: application-namespace spec: containers: - name: debug image: nicolaka/netshoot command: ["sleep", "3600"] restartPolicy: Never Create it: kubectl apply -f debug-pod.yaml Then test NFS mount manually inside debug pod: kubectl exec -it debug-network -n application-namespace -- mount -t nfs 192.168.1.100:/exports/data /mnt If this works but the app pod fails, the issue may be with the app pod's security context or volume mount configuration.

Quick check 2 of 2 According to the article, what is a recommended practice before making changes during troubleshooting? - Immediately restart all pods - Make multiple changes at once to speed up the process - Observe and capture the current state before intervening - Delete and recreate the PersistentVolumeClaim

The article emphasizes 'observe before changing' and capturing the current state with read-only commands before making any changes, as stated in the 'Observing Without Intervening' section.

## Failure Modes and Recovery Understanding common failure modes accelerates recovery. Here are specific scenarios and how to address them. ### Failure Mode 1: NFS Server Unreachable Symptoms: Pod logs show I/O error , mount hangs, or FailedMount events with connection timed out . Diagnosis: kubectl exec -it app-db-0 -n application-namespace -- ping -c 3 192.168.1.100 If ping fails, check network routing: are the cluster nodes able to reach the NFS server subnet? If the NFS server is on a different network, ensure routing tables and firewall rules allow traffic. Recovery: Fix network connectivity (e.g., add route or firewall rule), then restart pod: kubectl delete pod app-db-0 -n application-namespace Wait for StatefulSet to recreate it. ### Failure Mode 2: DNS Resolution Failure for NFS Server Hostname Symptoms: Mount errors with Unable to resolve host or server not found . If the NFS server IP changed but DNS not updated. Diagnosis: Confirm with nslookup (see earlier). Recovery: Update DNS record or, as an immediate workaround, use IP address in the PV spec. To change NFS server in PV: kubectl patch pv pvc-38b7f2a3-4c5d-4e6f-8a1b-9c0d1e2f3a4b --type='json' -p='[{"op": "replace", "path": "/spec/nfs/server", "value": "192.168.1.101"}]' Then delete the pod to remount: kubectl delete pod app-db-0 -n application-namespace ### Failure Mode 3: Export Permissions Denied Symptoms: access denied by server in mount output (see earlier). Diagnosis: Compare the NFS export configuration on the server. The export should allow the pod node IPs or the cluster network CIDR (e.g., 10.244.0.0/16 for Calico). Example export on NFS server /etc/exports : /exports/data 192.168.1.0/24(rw,sync,no_subtree_check,no_root_squash) If your cluster pods use a different CIDR (e.g., 10.244.0.0/16), the NFS server might reject because client IP is from that range. Update export to include that CIDR: /exports/data 10.244.0.0/16(rw,sync,no_subtree_check,no_root_squash) Then run exportfs -ra on the NFS server. No pod restart needed if mount succeeds on next attempt. ### Failure Mode 4: Network Policy Blocking Symptoms: Pod can reach NFS server from other pods but not from its own namespace, or connectivity fails after applying a NetworkPolicy. Diagnosis: Review NetworkPolicies as described. Use kubectl describe networkpolicy to see rules. If a deny-all policy is present, ensure allow rules for NFS. Recovery: Apply a new NetworkPolicy allowing egress to NFS server: apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-nfs-egress namespace: application-namespace spec: podSelector: matchLabels: app: db policyTypes: - Egress egress: - to: - ipBlock: cidr: 192.168.1.100/32 ports: - protocol: TCP port: 2049 - protocol: UDP port: 2049 Apply and test connectivity. ### Failure Mode 5: Mount Option Mismatch Symptoms: Mount succeeds but performance is poor, or frequent timeouts. nfsvers mismatch causes fallback to older version or failure. Diagnosis: Check StorageClass mount options and NFS server supported versions. Use rpcinfo -p 192.168.1.100 | grep nfs (from a node with access) to list versions. Recovery: Update StorageClass with correct nfsvers and restart pods. ## Operations Checklist Use this checklist to ensure you cover all bases during PV networking troubleshooting.

Step	Action	Command or Check	Expected Result
1	Record cluster version and nodes	`kubectl version --short`, `kubectl get nodes -o wide`	All nodes Ready, version consistent
2	Identify storage provisioner and backend	`kubectl get storageclass`, `kubectl get pv`	Provisioner matches intended (e.g., nfs.csi.k8s.io), PV Bound
3	Check NFS server reachability from node	`ping -c 3 192.168.1.100` from node	0% packet loss
4	Check NFS port open	`telnet 192.168.1.100 2049` or `nc -zv`	Connection successful
5	Verify DNS resolution inside pod	`kubectl exec <pod> -- nslookup nfs-server.example.com`	Returns correct IP
6	Test NFS mount manually in debug pod	`mount -t nfs 192.168.1.100:/exports/data /mnt` inside debug pod	Mount succeeds without errors
7	Review pod events	`kubectl describe pod <pod-name>`	No `FailedMount` warnings
8	Check kubelet logs	`journalctl -u kubelet -n 50 --no-pager` on node	No mount errors
9	Inspect NetworkPolicies	`kubectl get networkpolicies -n <ns>`	Rules allow NFS traffic if needed
10	Confirm PV/PVC status	`kubectl get pv,pvc -n <ns>`	Bound and available
11	Validate mount options	`kubectl get storageclass <name> -o yaml`	Options match server capabilities
12	Perform a controlled test change	Patch StorageClass or PV, restart pod, verify mount	No adverse effects
13	Document findings and recovery steps	Update runbook	Clear notes

### Runbook Example Entry After resolving an issue (e.g., NFS export permission denied), add an entry to your runbook: ## Incident: PVC mount failed with "access denied by server" Date: 2025-04-02 Cluster: prod-shared Namespace: application-namespace PV: pvc-38b7f2a3-4c5d-4e6f-8a1b-9c0d1e2f3a4b Symptoms: app-db-0 pod restarts, event shows mount.nfs access denied. Root cause: NFS server export only allowed 192.168.1.0/24, but pod network CIDR is 10.244.0.0/16. Resolution: Updated /etc/exports to include 10.244.0.0/16, ran exportfs -ra, restarted pod. Verification: pod restarted successfully, application writes resumed. ## Conclusion Troubleshooting Kubernetes Persistent Volume networking requires a structured approach: inventory the environment, verify configuration, run targeted diagnostics, understand failure modes, and follow a recovery checklist. By using the concrete commands and examples in this guide, you can quickly identify whether the issue lies in DNS, port connectivity, mount options, network policies, or backend permissions. Always observe before intervening. Capture current state, make one small change at a time, and verify the result. Keep sensitive data out of logs and commands by using placeholders and redacting when sharing. With these practices, you will reduce downtime and improve the reliability of your stateful applications. As a next step, pick one low-risk verification from the Operations Checklist, run it in your environment, and document the outcome. Over time, you will build a robust troubleshooting playbook for Kubernetes Persistent Volume networking.