Intro
Stateful applications such as databases, message queues, and distributed storage systems are critical components of many Kubernetes deployments. Unlike stateless applications, stateful applications require stable network identities, persistent storage, and often direct peer-to-peer communication between replicas. When networking issues arise, they can lead to data inconsistencies, split-brain scenarios, or complete service outages. Troubleshooting these issues requires a systematic approach that understands the unique networking model of Kubernetes StatefulSets and their associated Services.
This guide provides a practical, hands-on approach to diagnosing and resolving networking problems in replicated stateful applications running on Kubernetes. We will cover essential concepts, common failure modes, diagnostic commands, and recovery strategies. Whether you are a developer deploying your first stateful workload or a DevOps engineer managing production clusters, this guide will equip you with the tools and knowledge to keep your stateful applications connected and healthy.
Version and Environment Inventory
Before diving into troubleshooting, it is crucial to establish a clear picture of your environment. This includes Kubernetes versions, container runtime, network plugin, application configuration, and the topology of your StatefulSet and Services. Inconsistent versions or misconfigurations can themselves be the root cause of networking issues.
Kubernetes and Network Plugin Versions
First, verify the version of your Kubernetes cluster and the networking solution in use. Different network plugins (CNI) may have specific behaviors, especially regarding NetworkPolicies, service load balancing, and DNS integration.
kubectl version --short
Expected output (example):
Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.2
To check the CNI plugin, you can inspect the pods in the kube-system namespace:
kubectl get pods -n kube-system | grep -E 'calico|flannel|weave|cilium'
Example output:
calico-node-4x7vz 1/1 Running 0 5d2h
calico-node-8j2kq 1/1 Running 0 5d2h
calico-node-mp9wd 1/1 Running 0 5d2h
StatefulSet and Service Topology
Identify the StatefulSet and its associated governing Service. The governing Service is essential for providing stable network identities to the Pods. Use the following commands to list them:
kubectl get statefulsets -n <namespace>
kubectl get services -n <namespace>
Example output:
NAME READY AGE
my-stateful-app 3/3 2d
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-stateful-app ClusterIP None <none> 5432/TCP 2d
Note that the Service for a StatefulSet is often headless (ClusterIP: None). This is intentional: it allows Pods to be addressed individually via their stable DNS names.
Prerequisites for Troubleshooting
- Ensure you have kubectl configured with appropriate permissions to exec into pods and inspect resources.
- Have access to application logs (via
kubectl logs) and cluster events (kubectl get events). - For advanced network debugging, tools like
tcpdump,netcat,curl, anddigshould be available inside your application containers or in a separate debug container.
Safe Configuration Path
When configuring networking for replicated stateful applications, certain choices have a significant impact on reliability and ease of troubleshooting. In this section, we discuss scoped implementation choices that minimize risk and improve observability.
Use a Headless Service for Stable Identities
A headless Service (with clusterIP: None) is the standard way to govern a StatefulSet. It does not load-balance traffic but instead returns the individual Pod IPs when queried via DNS. This allows direct pod-to-pod communication, which is often required by stateful applications for replication and consensus protocols.
Example Service definition:
apiVersion: v1
kind: Service
metadata:
name: my-stateful-app
namespace: production
spec:
clusterIP: None
selector:
app: my-stateful-app
ports:
- name: replication
port: 5432
targetPort: 5432
- name: metrics
port: 9104
targetPort: 9104
With this Service, each Pod gets a DNS entry of the form <pod-name>.<service-name>.<namespace>.svc.cluster.local. For example, my-stateful-app-0.my-stateful-app.production.svc.cluster.local.
To ensure the headless service is functioning correctly, you can verify its configuration:
kubectl get service my-stateful-app -n production -o jsonpath='{.spec.clusterIP}'
Expected output:
None
Also, confirm that the service has a selector that matches the StatefulSet pods:
kubectl get service my-stateful-app -n production -o jsonpath='{.spec.selector}'
Expected output (example):
{"app":"my-stateful-app"}
Expose Individual Pods for External Access (if needed)
If external clients need to reach specific pods (e.g., for direct database connections), consider creating a per-pod Service or using a headless Service with an external load balancer in front. However, be cautious: exposing individual pods can bypass the StatefulSet's orderly management and may lead to split-brain if not carefully coordinated.
One approach is to create a separate Service for each pod using a unique label. For example, you can add a label statefulset.kubernetes.io/pod-name: my-stateful-app-0 to each pod and create a Service with that selector. This allows you to expose each pod individually while maintaining stable DNS names.
Implement Network Policies with Care
NetworkPolicies provide segmentation but can inadvertently block required traffic between replicas. When defining policies, be explicit about allowed ingress and egress for the application's ports. Test changes in a staging environment first.
Example NetworkPolicy allowing intra-StatefulSet traffic:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-stateful-app-replication
namespace: production
spec:
podSelector:
matchLabels:
app: my-stateful-app
policyTypes:
- Ingress
- Egress
ingress:
- from:
- podSelector:
matchLabels:
app: my-stateful-app
ports:
- protocol: TCP
port: 5432
egress:
- to:
- podSelector:
matchLabels:
app: my-stateful-app
ports:
- protocol: TCP
port: 5432
This policy ensures that all pods with label app: my-stateful-app can communicate with each other on TCP port 5432, but only in the specified direction. Note that the policy above allows ingress from pods with the same label and egress to pods with the same label, but only on port 5432. If your application uses additional ports (e.g., for metrics or admin), you must add rules for those as well.
After applying a NetworkPolicy, always verify that it is correctly selected and check the actual rules using:
kubectl describe networkpolicy allow-stateful-app-replication -n production
This will show you the policy's selectors and rules. Ensure that the podSelector matches your StatefulSet pods and that the allowed ports are correct.
Enable Proper Logging and Monitoring
Ensure your stateful application logs include network-related information (e.g., connection attempts, errors). Integrate with cluster monitoring to track network metrics such as connection latency, packet loss, and dropped packets. This facilitates faster diagnosis when issues arise.
For example, you can configure your application to log at DEBUG level for network components. Additionally, use a monitoring stack like Prometheus and Grafana to scrape metrics from your application and the CNI plugin. Key metrics to watch include:
node_network_receive_drop_totalandnode_network_transmit_drop_totalfor packet drops.container_network_receive_bytes_totalandcontainer_network_transmit_bytes_totalfor pod traffic.- Application-specific metrics like connection counts or replication lag.
Verification and Diagnostics
Once your StatefulSet and Services are deployed, you should verify that networking is functioning correctly. This section provides a series of checks and diagnostic steps, each with expected results.
Verify Pod Readiness and Service Endpoints
Start by ensuring all pods are running and ready:
kubectl get pods -n <namespace> -l app=my-stateful-app
Expected output:
NAME READY STATUS RESTARTS AGE
my-stateful-app-0 1/1 Running 0 5m
my-stateful-app-1 1/1 Running 0 4m
my-stateful-app-2 1/1 Running 0 3m
Check that the Service has endpoints corresponding to each pod's IP:
kubectl get endpoints my-stateful-app -n <namespace>
Expected output:
NAME ENDPOINTS AGE
my-stateful-app 10.244.1.5:5432,10.244.2.3:5432,10.244.3.7:5432 5m
If the endpoints list is empty or missing some pods, investigate the Service's selector and pod labels. Confirm that the labels on the pods match the service selector:
kubectl get pods -n <namespace> --show-labels
Look for the app: my-stateful-app label on each pod. If labels are missing, the service will not select them, and endpoints will be empty.
Test DNS Resolution
DNS is a critical component for pod-to-pod communication. Use nslookup or dig from within a pod to verify that each pod's hostname resolves to the correct IP address. Execute commands in one of the pods:
kubectl exec -it my-stateful-app-0 -n <namespace> -- nslookup my-stateful-app-1.my-stateful-app.<namespace>.svc.cluster.local
Expected output (example):
Server: 10.96.0.10
Address: 10.96.0.10#53
Name: my-stateful-app-1.my-stateful-app.<namespace>.svc.cluster.local
Address: 10.244.2.3
The IP address should match the pod IP from kubectl get pods -o wide.
For more detailed DNS debugging, you can use dig:
kubectl exec -it my-stateful-app-0 -n <namespace> -- dig +short my-stateful-app-1.my-stateful-app.<namespace>.svc.cluster.local
This should return just the IP address.
Also test the service name without the pod prefix to see all IPs:
kubectl exec -it my-stateful-app-0 -n <namespace> -- nslookup my-stateful-app.<namespace>.svc.cluster.local
For a headless service, this should return multiple A records for each pod.
Check Connectivity Between Pods
Use telnet, nc, or curl to test TCP connectivity on the application port. For example, from pod-0 connect to pod-1 on port 5432:
kubectl exec -it my-stateful-app-0 -n <namespace> -- nc -zv my-stateful-app-1.my-stateful-app.<namespace>.svc.cluster.local 5432
Expected output:
Connection to my-stateful-app-1.my-stateful-app.<namespace>.svc.cluster.local 5432 port [tcp/*] succeeded!
If the connection fails, proceed to the failure modes section.
For a more thorough test, you can use curl if the application speaks HTTP, or use a custom script to send application-specific probes.
Inspect Network Policies
If NetworkPolicies are in place, verify that they allow the required traffic. List all policies in the namespace:
kubectl get networkpolicies -n <namespace>
Describe a specific policy to review its rules:
kubectl describe networkpolicy allow-stateful-app-replication -n <namespace>
Look for any PodSelector or NamespaceSelector that might exclude the pods. Ensure the policy's policyTypes includes both Ingress and Egress if bidirectional communication is needed.
Also, check if any network policy is applied to the pods by examining the pod's labels and the policy's podSelector. You can use:
kubectl get pods -n <namespace> --show-labels
and compare with the policy's selector.
Validate Service Type and ClusterIP
Ensure the Service is headless if you rely on individual pod DNS. Check the Service details:
kubectl get service my-stateful-app -n <namespace> -o yaml
In the output, verify clusterIP: None. If it is not headless, the DNS query for <pod-name>.<service-name> will not resolve to the pod; instead, it may resolve to the service's cluster IP and load balance, which may break stateful communication.
You can also check via command:
kubectl get service my-stateful-app -n <namespace> -o jsonpath='{.spec.clusterIP}'
If the output is an IP address (e.g., 10.96.0.100), then the service is not headless.
Failure Modes and Recovery
Even with careful configuration, networking issues can occur. Understanding common failure modes helps you quickly recover. This section outlines typical problems, their symptoms, diagnostic steps, and recovery actions.
DNS Resolution Failures
Symptom: Pods cannot resolve each other's hostnames; application logs show "unknown host" or connection timeouts.
Possible Causes:
- CoreDNS pods not running.
- DNS service (kube-dns) misconfigured.
- Pod's
/etc/resolv.confhas incorrect nameserver. - Network policy blocking DNS traffic (UDP/TCP port 53).
Diagnostics:
- Check CoreDNS pods:
kubectl get pods -n kube-system -l k8s-app=kube-dns
Expected: all pods Running and Ready.
- Test DNS from a pod:
kubectl exec -it <pod> -n <namespace> -- nslookup kubernetes.default
If this fails, the issue is cluster-wide DNS.
- Inspect
/etc/resolv.confinside the pod:
kubectl exec -it <pod> -n <namespace> -- cat /etc/resolv.conf
It should contain nameserver 10.96.0.10 (or your cluster's DNS IP) and search <namespace>.svc.cluster.local svc.cluster.local cluster.local.
Recovery:
- If CoreDNS is down, restart or scale the deployment:
kubectl rollout restart deployment coredns -n kube-system
- Ensure NetworkPolicies allow egress to the DNS service on port 53. Example egress rule:
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
- If the pod's DNS configuration is wrong, check the pod's
dnsPolicyanddnsConfigfields. For most cases,dnsPolicy: ClusterFirstis correct.
Network Policy Denies Traffic
Symptom: Some pods can connect while others cannot, or connections fail from specific namespaces.
Possible Causes:
- A NetworkPolicy with a restrictive podSelector or namespaceSelector.
- Missing egress rule to allow responses.
Diagnostics:
- Describe NetworkPolicies in the namespace and inspect rules.
kubectl describe networkpolicy -n <namespace>
- Temporarily delete a suspect policy to see if connectivity returns (in a test environment).
kubectl delete networkpolicy <policy-name> -n <namespace>
After deletion, test connectivity; if it works, the policy was the issue.
Recovery:
- Adjust the policy to permit the required traffic. Ensure bidirectional rules are specified if needed. For example, to allow all traffic between pods with label
app: my-stateful-app:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-all-stateful-app
namespace: production
spec:
podSelector:
matchLabels:
app: my-stateful-app
ingress:
- from:
- podSelector:
matchLabels:
app: my-stateful-app
egress:
- to:
- podSelector:
matchLabels:
app: my-stateful-app
Note: This is permissive; adjust ports as needed.
- Test the policy in a staging environment before applying to production.
Headless Service Misconfiguration
Symptom: Individual pod DNS names do not resolve, or they resolve to a single Cluster IP (if Service is not headless).
Diagnostics:
- Check Service YAML for
clusterIP: None.
kubectl get service my-stateful-app -n <namespace> -o jsonpath='{.spec.clusterIP}'
If output is an IP, it's not headless.
- Use
nslookupfor a pod's hostname; if it returns a service IP, the Service is not headless.
Recovery:
- Recreate the Service with
clusterIP: None. Note: changing a Service type may cause brief downtime; plan accordingly.
kubectl delete service my-stateful-app -n <namespace>
kubectl apply -f headless-service.yaml
Pod IP Changes or StatefulSet Scale Issues
Symptom: After a pod restart, other pods cannot reconnect because they cached the old pod IP.
Possible Causes:
- StatefulSet pods have stable hostnames but their IPs can change on restart if the pod is deleted and recreated on a different node.
- Applications may cache IP addresses instead of re-resolving DNS.
Diagnostics:
- Check pod IPs before and after restart:
kubectl get pods -o wide -n <namespace>
Compare the IP column.
- Look for connection errors mentioning stale IPs in application logs.
Recovery:
- Ensure applications use DNS names and refresh DNS lookups periodically.
- If DNS caching is the issue, adjust the application's resolver settings or use a headless Service with
publishNotReadyAddresses: trueto minimize IP churn. - Consider using a StatefulSet with
podManagementPolicy: Parallelto reduce sequential restarts, but be aware of application implications.
Firewall or Node Network Issues
Symptom: All connectivity between pods on different nodes fails, while same-node communication works.
Possible Causes:
- Node-level firewall blocking overlay network ports.
- CNI plugin not routing correctly across nodes.
- Network interface or routing table issues on a node.
Diagnostics:
- Test cross-node connectivity:
kubectl exec -it <pod-on-node-1> -n <namespace> -- ping <pod-ip-on-node-2>
If ping fails but same-node ping works, suspect node networking.
- Check node firewall rules (on the node):
sudo iptables -L -n
- Verify CNI plugin status:
kubectl get pods -n kube-system | grep <cni-name>
Recovery:
- Adjust firewall rules to allow required ports (e.g., VXLAN UDP 8472, BGP TCP 179, etc., depending on CNI).
- Restart CNI pods if they are in error state:
kubectl delete pod -n kube-system <cni-pod>
- Check node routing table and ensure the overlay interface exists.
Rollback and Recovery Checks
When implementing changes to resolve networking issues, always have a rollback plan. Document the original configuration and test changes in a non-production environment. After applying a fix, run the verification steps from the previous section to confirm resolution. Monitor the application for a period to ensure stability.
For example, before applying a NetworkPolicy change, save the current policy YAML:
kubectl get networkpolicy <name> -n <namespace> -o yaml > backup-policy.yaml
If the change causes issues, restore it:
kubectl apply -f backup-policy.yaml
Operations Checklist
Use the following checklist as a repeatable process for operating and reviewing the networking configuration of your replicated stateful application. Running these checks periodically can help catch issues before they impact availability. Assign a single accountable owner for each major decision or checklist item, and revisit at the specified frequency.
Daily Checks
- [ ] All StatefulSet pods are running and ready.
- Owner: On-call engineer (rotating)
- How: Run
kubectl get pods -n <namespace> -l app=my-stateful-appand verify all replicas are 1/1 Ready. - [ ] Service endpoints match the number of replicas.
- Owner: On-call engineer
- How:
kubectl get endpoints my-stateful-app -n <namespace>and confirm the number of endpoints equals desired replicas. - [ ] No recent network-related errors in application logs.
- Owner: Application developer (e.g., Priya Shah, Engineering Lead)
- How: Check centralized logging for patterns like "connection refused", "timeout", "unknown host" within the last 24 hours.
Weekly Checks
- [ ] DNS resolution for all pod hostnames succeeds from multiple pods.
- Owner: DevOps engineer (e.g., John Doe, Platform Engineer)
- How: Run a script that execs into each pod and does
nslookupfor every other pod's hostname. - [ ] TCP connectivity on all required ports between a sample of pods.
- Owner: DevOps engineer
- How: Use
nc -zvfrom a few pods to others, covering all service ports (e.g., 5432, 9104). - [ ] NetworkPolicy rules reviewed and still align with application needs.
- Owner: Security engineer (e.g., Maria Garcia, Security Architect)
- How: Examine each policy in the namespace and compare with documented application requirements. Remove or update stale rules.
Before Any Change
- [ ] Capture current StatefulSet, Service, and NetworkPolicy YAMLs for rollback.
- Owner: Person making the change (must be individually identified)
- How: Use
kubectl get <resource> -o yaml > backup-<resource>.yaml. - [ ] Test configuration changes in a staging cluster.
- Owner: Person making the change
- How: Deploy the same manifest in a staging environment and run the verification steps.
After Any Change
- [ ] Run verification commands (endpoints, DNS, connectivity).
- Owner: Person making the change
- How: Follow the steps in "Verification and Diagnostics" section.
- [ ] Monitor application metrics for expected behavior.
- Owner: On-call engineer
- How: Watch dashboards for 30 minutes post-change; look for anomalies in connections, latency, or errors.
- [ ] Document the change and its impact.
- Owner: Person making the change
- How: Update the change log or ticket with details and outcomes.
Troubleshooting Quick Reference
| Symptom | Likely Cause | First Command to Run |
|---|---|---|
| Pods cannot resolve each other | DNS issues | kubectl exec -it <pod> -- nslookup <other-pod-hostname> |
| Connection refused on app port | Application not listening or wrong port | kubectl exec -it <pod> -- nc -zv <target> <port> |
| Partial connectivity | NetworkPolicy restrictions | kubectl describe networkpolicy -n <namespace> |
| All cross-node traffic fails | CNI or node firewall | kubectl get pods -n kube-system -l k8s-app=calico-node (or your CNI label) |
| Service endpoints empty | Label selector mismatch | kubectl get pods --show-labels -n <namespace> |
Common Pitfalls and How to Avoid Them
Many networking issues in stateful applications stem from a few recurring mistakes. Being aware of these pitfalls can help you avoid them and quickly recognize them when they occur.
1. Using a ClusterIP Service Instead of Headless
Why it happens: Developers may reuse a standard Service template or not understand the need for headless services with StatefulSets.
How to avoid: Always define a headless Service (with clusterIP: None) for StatefulSets that require direct pod-to-pod communication. Document this as a standard.
Recovery: If a non-headless Service is in place, recreate it as headless. Note that this may cause a brief disruption as DNS records update.
2. Overly Restrictive Network Policies
Why it happens: In the name of security, teams may apply broad deny-all policies or too-specific selectors that unintentionally block replication traffic.
How to avoid: Start with a policy that allows required traffic explicitly and test thoroughly in staging. Use labels carefully and ensure both ingress and egress are considered.
Recovery: Temporarily remove or modify the policy to restore connectivity, then refine and reapply with proper rules.
3. Ignoring DNS Caching in Applications
Why it happens: Many applications resolve hostnames once and cache the IP indefinitely. When pods restart and IPs change, connections fail.
How to avoid: Configure applications to use DNS names and refresh lookups appropriately. If possible, set a low TTL for DNS records (Kubernetes default is 30 seconds for headless services). Consider using publishNotReadyAddresses: true to keep IPs stable for a longer time during rolling updates.
Recovery: Restart the application pods to flush DNS caches, or if the application supports it, trigger a manual re-resolution.
4. Mislabeling Pods or Services
Why it happens: Manual edits or CI/CD misconfigurations can cause label mismatches, leading to missing endpoints.
How to avoid: Use declarative manifests and version control. Implement a CI check that validates selectors against pod labels.
Recovery: Correct the labels on pods or service selector and reapply.
5. Not Accounting for CNI-Specific Requirements
Why it happens: Different CNI plugins have different default ports and behaviors. For example, Calico uses BGP (TCP 179) by default, while Flannel uses VXLAN (UDP 8472). Firewall rules may not be updated.
How to avoid: Document the specific CNI in use and its network requirements. Ensure node firewalls allow those ports.
Recovery: Open the required ports on nodes or adjust CNI configuration.
6. Scaling StatefulSet Without Considering Ordering
Why it happens: StatefulSets by default use ordered pod management (OrderedReady), which can cause delays or issues during scale-down if not understood.
How to avoid: If your application can handle unordered scaling, set podManagementPolicy: Parallel to speed up operations. Otherwise, plan for sequential scaling and ensure readiness probes are correct.
Recovery: If a scale-down causes issues, check the StatefulSet's podManagementPolicy and adjust if appropriate.
Conclusion
Networking for replicated stateful applications in Kubernetes requires careful planning and ongoing vigilance. By understanding the role of headless Services, stable pod identities, and NetworkPolicies, you can build a resilient environment. When issues arise, the systematic diagnostic steps outlined in this guide, from verifying DNS resolution to inspecting network policies, will help you identify and resolve problems efficiently.
Remember to keep your environment inventory up to date, test configuration changes in a safe manner, and maintain a rollback plan. Use the operations checklist to incorporate regular checks into your routine. With these practices, you can ensure that your stateful applications remain connected, consistent, and highly available.
For further learning, explore the official Kubernetes documentation on StatefulSets and Services, and practice troubleshooting in a sandbox cluster to build confidence.