Kubernetes Persistent Volume Networking Troubleshooting: A Practical Guide

Intro

Kubernetes Persistent Volume (PV) networking issues can surface as slow application responses, failed mounts, or inaccessible services. Troubleshooting requires methodical observation, a clear understanding of the component interactions, and safe, reversible changes. This guide provides practical, example-driven steps to diagnose and resolve networking problems related to Persistent Volumes in Kubernetes, focusing on DNS resolution, port connectivity, and end-to-end network troubleshooting.

We will walk through a realistic scenario: an application pod that cannot access a Persistent Volume served over the network, leading to intermittent write failures. You will learn how to gather version and environment details, verify configuration, perform diagnostics, identify failure modes, and follow a recovery checklist. Every step includes concrete commands, expected outputs, and decision points. The goal is operational confidence: observe before changing, limit the blast radius, use placeholders for sensitive data, and verify outcomes.

This guide targets developers, DevOps consultants, and technical startup teams who manage stateful workloads in Kubernetes. While the focus is on PV networking, we touch on Persistent Volume Claims (PVC), StatefulSets, and Storage Classes only when they affect connectivity or recovery.

Version and Environment Inventory

Before diving into troubleshooting, establish the exact version and topology of your Kubernetes cluster and the storage backend. This prevents mismatched assumptions and helps you compare against known issues. Run these commands and record the output alongside a timestamp.

Cluster and Node Information

kubectl version --short
kubectl get nodes -o wide

Expected output is similar to:

Client Version: v1.28.2
Server Version: v1.28.2

NAME       STATUS   ROLES           AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
node-1     Ready    control-plane   30d   v1.28.2   10.0.0.11     <none>        Ubuntu 22.04.3 LTS   5.15.0-86-generic   containerd://1.7.11
node-2     Ready    <none>          30d   v1.28.2   10.0.0.12     <none>        Ubuntu 22.04.3 LTS   5.15.0-86-generic   containerd://1.7.11

Note the Kubernetes version (e.g., 1.28) and container runtime (containerd). Network plugin details matter; for example, with Calico:

kubectl get pods -n kube-system | grep -E 'calico|kube-proxy'

Expected output:

calico-node-abcde                    1/1     Running   0          30d
calico-kube-controllers-12345        1/1     Running   0          30d
kube-proxy-xyz1                      1/1     Running   0          30d

Storage Backend Information

Identify the StorageClass and provisioner used by your PV. Suppose you use NFS for a shared storage network. List storage classes:

kubectl get storageclass

Expected output:

NAME                 PROVISIONER       RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
nfs-client (default) nfs.csi.k8s.io   Delete          Immediate           true                   60d

Then inspect the PV and PVC details:

kubectl get pv
kubectl get pvc -n application-namespace

Example output:

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                      STORAGECLASS   REASON   AGE
pvc-38b7f2a3-4c5d-4e6f-8a1b-9c0d1e2f3a4b   20Gi       RWO            Delete           Bound    app/data-pvc               nfs-client              15d

NAME        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-pvc    Bound    pvc-38b7f2a3-4c5d-4e6f-8a1b-9c0d1e2f3a4b   20Gi       RWO            nfs-client     15d

Record the NFS server endpoint if using NFS:

kubectl describe pv pvc-38b7f2a3-4c5d-4e6f-8a1b-9c0d1e2f3a4b | grep -A5 'Source:'

Expected output snippet:

Source:
    Type:      NFS (an NFS mount that lasts the lifetime of a pod)
    Server:    192.168.1.100
    Path:      /exports/data
    ReadOnly:  false

This tells you the network endpoint your pods must reach. If the NFS server IP is wrong or unreachable, all PV networking fails.

Pod and Workload Overview

Check the pods using the PVC:

kubectl get pods -n application-namespace -o wide

Example:

NAME                         READY   STATUS    RESTARTS   AGE   IP               NODE     NOMINATED NODE   READINESS GATES
app-db-0                     1/1     Running   4          15d   10.244.2.55      node-2   <none>           <none>
app-web-7b9c8d5f6-abcde      1/1     Running   0          15d   10.244.1.89      node-1   <none>           <none>

Notice the restarts on app-db-0; this may indicate intermittent storage connectivity.

Observing Without Intervening

At this stage, only run read-only commands. Avoid edits or deletions until you understand the current state. Capture the output of the following for later comparison:

kubectl describe pod app-db-0 -n application-namespace
kubectl logs app-db-0 -n application-namespace --tail=50

If the pod is crash-looping, use --previous:

kubectl logs app-db-0 -n application-namespace --previous

This inventory gives you a baseline. Now, move to safe configuration checks.

Safe Configuration Path

The safe configuration path ensures you make minimal, well-understood changes. For PV networking, focus on the StorageClass, PVC, and pod mount configurations.

Inspect the StorageClass

For NFS, you might have a custom StorageClass with mount options that affect network behavior. View the YAML:

kubectl get storageclass nfs-client -o yaml

Key fields:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-client
provisioner: nfs.csi.k8s.io
parameters:
  server: 192.168.1.100
  share: /exports
mountOptions:
  - nfsvers=4.1
  - hard
  - timeo=600
  - retrans=2
reclaimPolicy: Delete
volumeBindingMode: Immediate

Check that nfsvers matches the NFS server capability. Mismatched version (e.g., server supports NFSv3 only, but client requests 4.1) causes mount failures or poor performance. Also, hard mount option means the pod will block indefinitely on network partitions; soft may be preferable for faster failure detection, but it can cause data corruption. Decide based on workload tolerance.

PVC and Pod Mount Verification

Verify the PVC has correct access modes and is bound:

kubectl get pvc data-pvc -n application-namespace -o yaml

Look for accessModes:

accessModes:
- ReadWriteOnce

If your application needs shared storage (e.g., multiple pods reading/writing), you need ReadWriteMany. Using ReadWriteOnce with multiple replicas can cause only one node to mount successfully, leading to networking errors for others.

Inspect the pod's volume mount and read-only flags:

kubectl get pod app-db-0 -n application-namespace -o yaml | grep -A10 'volumeMounts:'

Expected snippet:

volumeMounts:
- mountPath: /var/lib/data
  name: data
  readOnly: false

Ensure mountPath matches the application's expected path and that it is not read-only when writes are required.

Smallest Justified Change

If you need to alter a configuration (e.g., add soft mount option or change NFS version), update the StorageClass with a patch, but first test in a non-production environment.

Example patch to change nfsvers from 4.1 to 4.0:

kubectl patch storageclass nfs-client --type='json' -p='[{"op": "replace", "path": "/mountOptions/0", "value": "nfsvers=4.0"}]'

Verify the change:

kubectl get storageclass nfs-client -o yaml | grep -A5 'mountOptions'

New pods will use the updated option. Existing pods must be restarted to pick up changes; consider a rolling restart of the StatefulSet.

Local Testing with port-forward

Before involving external network services, test connectivity locally. For a pod that serves a network file protocol or API, use kubectl port-forward:

kubectl port-forward pod/app-db-0 -n application-namespace 2049:2049

Then from another terminal, test the NFS mount using local tools:

sudo mount -t nfs -o nfsvers=4.1 localhost:/exports/data /mnt/test

If this fails, the issue is inside the cluster, not with external networking.

Verifying Without Load Balancers

Avoid creating a LoadBalancer service just to test. Instead, use a temporary NodePort service or port-forward. This limits exposure and cost.

Verification and Diagnostics

Now that the configuration is understood, perform active diagnostics to isolate the network issue.

DNS Resolution Checks

Many PV networking issues stem from DNS failures, especially when the NFS server is referenced by hostname. First, check the CoreDNS pods:

kubectl get pods -n kube-system -l k8s-app=kube-dns

Expected:

NAME                       READY   STATUS    RESTARTS   AGE
coredns-787d4945fb-abcde   1/1     Running   0          30d
coredns-787d4945fb-fghij   1/1     Running   0          30d

Test DNS resolution from inside a pod. Exec into the app pod (or a debug pod) and run nslookup:

kubectl exec -it app-db-0 -n application-namespace -- nslookup nfs-server.example.com

Expected output:

Server:         10.96.0.10
Address:        10.96.0.10#53

Name:   nfs-server.example.com
Address: 192.168.1.100

If you get server can't find nfs-server.example.com: NXDOMAIN, the hostname is not in DNS. Check if you need to add an entry to /etc/hosts via hostAliases in the pod spec or fix external DNS.

Port Connectivity Tests

NFS typically uses TCP/UDP port 2049. Test connectivity from the pod to the NFS server:

kubectl exec -it app-db-0 -n application-namespace -- nc -zvw3 192.168.1.100 2049

If nc is not installed, use bash with /dev/tcp:

kubectl exec -it app-db-0 -n application-namespace -- bash -c 'echo > /dev/tcp/192.168.1.100/2049 && echo open || echo closed'

Expected success output: open. If closed, check NetworkPolicies, firewall rules, or the NFS server service status.

Check for NetworkPolicies that might block egress to the NFS server:

kubectl get networkpolicies -n application-namespace

If a policy exists, inspect it:

kubectl describe networkpolicy allow-nfs -n application-namespace

Ensure the policy allows egress to 192.168.1.100/32 on port 2049.

Endpoint and Service Checks

If the storage is exposed via a Kubernetes Service (e.g., an NFS server running as a pod), verify endpoints:

kubectl get endpoints nfs-service -n storage-namespace

Expected:

NAME          ENDPOINTS           AGE
nfs-service   10.244.2.10:2049    20d

If endpoints are empty, check the service selector matches the pod labels.

Logs and Events

Check pod events for mount errors:

kubectl describe pod app-db-0 -n application-namespace | tail -30

Look for messages like:

Warning  FailedMount  2m (x12 over 30m)  kubelet  MountVolume.SetUp failed for volume "pvc-38b7f2a3..." : mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t nfs 192.168.1.100:/exports/data /var/lib/kubelet/pods/.../volumes/kubernetes.io~nfs/pvc-38b7f2a3...
Output: mount.nfs: access denied by server while mounting 192.168.1.100:/exports/data

This indicates an NFS export permission issue or incorrect client IP in the export configuration.

Check kubelet logs on the node where the pod runs (node-2):

journalctl -u kubelet -n 50 --no-pager | grep -i 'nfs\|mount'

This may reveal network timeouts or RPC errors.

Network Plugin Debugging

If using Calico, check calico-node logs:

kubectl logs -n kube-system calico-node-abcde --tail=50 | grep -i 'nfs\|drop\|deny'

Look for dropped packets due to policy. You may also use calicoctl to check network policy evaluation.

Temporary Debug Pod

Run a temporary pod with network tools to isolate whether the issue is pod-specific or cluster-wide:

apiVersion: v1
kind: Pod
metadata:
  name: debug-network
  namespace: application-namespace
spec:
  containers:
  - name: debug
    image: nicolaka/netshoot
    command: ["sleep", "3600"]
  restartPolicy: Never

Create it:

kubectl apply -f debug-pod.yaml

Then test NFS mount manually inside debug pod:

kubectl exec -it debug-network -n application-namespace -- mount -t nfs 192.168.1.100:/exports/data /mnt

If this works but the app pod fails, the issue may be with the app pod's security context or volume mount configuration.

Failure Modes and Recovery

Understanding common failure modes accelerates recovery. Here are specific scenarios and how to address them.

Failure Mode 1: NFS Server Unreachable

Symptoms: Pod logs show I/O error, mount hangs, or FailedMount events with connection timed out.

Diagnosis:

kubectl exec -it app-db-0 -n application-namespace -- ping -c 3 192.168.1.100

If ping fails, check network routing: are the cluster nodes able to reach the NFS server subnet? If the NFS server is on a different network, ensure routing tables and firewall rules allow traffic.

Recovery: Fix network connectivity (e.g., add route or firewall rule), then restart pod:

kubectl delete pod app-db-0 -n application-namespace

Wait for StatefulSet to recreate it.

Failure Mode 2: DNS Resolution Failure for NFS Server Hostname

Symptoms: Mount errors with Unable to resolve host or server not found. If the NFS server IP changed but DNS not updated.

Diagnosis: Confirm with nslookup (see earlier).

Recovery: Update DNS record or, as an immediate workaround, use IP address in the PV spec. To change NFS server in PV:

kubectl patch pv pvc-38b7f2a3-4c5d-4e6f-8a1b-9c0d1e2f3a4b --type='json' -p='[{"op": "replace", "path": "/spec/nfs/server", "value": "192.168.1.101"}]'

Then delete the pod to remount:

kubectl delete pod app-db-0 -n application-namespace

Failure Mode 3: Export Permissions Denied

Symptoms: access denied by server in mount output (see earlier).

Diagnosis: Compare the NFS export configuration on the server. The export should allow the pod node IPs or the cluster network CIDR (e.g., 10.244.0.0/16 for Calico).

Example export on NFS server /etc/exports:

/exports/data 192.168.1.0/24(rw,sync,no_subtree_check,no_root_squash)

If your cluster pods use a different CIDR (e.g., 10.244.0.0/16), the NFS server might reject because client IP is from that range. Update export to include that CIDR:

/exports/data 10.244.0.0/16(rw,sync,no_subtree_check,no_root_squash)

Then run exportfs -ra on the NFS server. No pod restart needed if mount succeeds on next attempt.

Failure Mode 4: Network Policy Blocking

Symptoms: Pod can reach NFS server from other pods but not from its own namespace, or connectivity fails after applying a NetworkPolicy.

Diagnosis: Review NetworkPolicies as described. Use kubectl describe networkpolicy to see rules. If a deny-all policy is present, ensure allow rules for NFS.

Recovery: Apply a new NetworkPolicy allowing egress to NFS server:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-nfs-egress
  namespace: application-namespace
spec:
  podSelector:
    matchLabels:
      app: db
  policyTypes:
  - Egress
  egress:
  - to:
    - ipBlock:
        cidr: 192.168.1.100/32
    ports:
    - protocol: TCP
      port: 2049
    - protocol: UDP
      port: 2049

Apply and test connectivity.

Failure Mode 5: Mount Option Mismatch

Symptoms: Mount succeeds but performance is poor, or frequent timeouts. nfsvers mismatch causes fallback to older version or failure.

Diagnosis: Check StorageClass mount options and NFS server supported versions. Use rpcinfo -p 192.168.1.100 | grep nfs (from a node with access) to list versions.

Recovery: Update StorageClass with correct nfsvers and restart pods.

Operations Checklist

Use this checklist to ensure you cover all bases during PV networking troubleshooting.

Runbook Example Entry

After resolving an issue (e.g., NFS export permission denied), add an entry to your runbook:

## Incident: PVC mount failed with "access denied by server"
Date: 2025-04-02
Cluster: prod-shared
Namespace: application-namespace
PV: pvc-38b7f2a3-4c5d-4e6f-8a1b-9c0d1e2f3a4b
Symptoms: app-db-0 pod restarts, event shows mount.nfs access denied.
Root cause: NFS server export only allowed 192.168.1.0/24, but pod network CIDR is 10.244.0.0/16.
Resolution: Updated /etc/exports to include 10.244.0.0/16, ran exportfs -ra, restarted pod.
Verification: pod restarted successfully, application writes resumed.

Conclusion

Troubleshooting Kubernetes Persistent Volume networking requires a structured approach: inventory the environment, verify configuration, run targeted diagnostics, understand failure modes, and follow a recovery checklist. By using the concrete commands and examples in this guide, you can quickly identify whether the issue lies in DNS, port connectivity, mount options, network policies, or backend permissions.

Always observe before intervening. Capture current state, make one small change at a time, and verify the result. Keep sensitive data out of logs and commands by using placeholders and redacting when sharing. With these practices, you will reduce downtime and improve the reliability of your stateful applications.

As a next step, pick one low-risk verification from the Operations Checklist, run it in your environment, and document the outcome. Over time, you will build a robust troubleshooting playbook for Kubernetes Persistent Volume networking.