E-NO
Kubernetes Run Replicated Stateful Application networking 11 Min Read

Kubernetes Run Replicated Stateful Application Networking: A Practical Troubleshooting Guide

calendar_today Published: 2026-11-19
update Last Updated: 2026-11-19
analytics SEO Efficiency: 100%
Technical guide illustration for Kubernetes Run Replicated Stateful Application Networking: A Practical Troubleshooting Guide.

Intro

Stateful applications such as databases, message queues, and distributed storage systems are critical components of many Kubernetes deployments. Unlike stateless applications, stateful applications require stable network identities, persistent storage, and often direct peer-to-peer communication between replicas. When networking issues arise, they can lead to data inconsistencies, split-brain scenarios, or complete service outages. Troubleshooting these issues requires a systematic approach that understands the unique networking model of Kubernetes StatefulSets and their associated Services.

This guide provides a practical, hands-on approach to diagnosing and resolving networking problems in replicated stateful applications running on Kubernetes. We will cover essential concepts, common failure modes, diagnostic commands, and recovery strategies. Whether you are a developer deploying your first stateful workload or a DevOps engineer managing production clusters, this guide will equip you with the tools and knowledge to keep your stateful applications connected and healthy.

Version and Environment Inventory

Before diving into troubleshooting, it is crucial to establish a clear picture of your environment. This includes Kubernetes versions, container runtime, network plugin, application configuration, and the topology of your StatefulSet and Services. Inconsistent versions or misconfigurations can themselves be the root cause of networking issues.

Kubernetes and Network Plugin Versions

First, verify the version of your Kubernetes cluster and the networking solution in use. Different network plugins (CNI) may have specific behaviors, especially regarding NetworkPolicies, service load balancing, and DNS integration.

kubectl version --short

Expected output (example):

Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.2

To check the CNI plugin, you can inspect the pods in the kube-system namespace:

kubectl get pods -n kube-system | grep -E 'calico|flannel|weave|cilium'

Example output:

calico-node-4x7vz   1/1     Running   0          5d2h
calico-node-8j2kq   1/1     Running   0          5d2h
calico-node-mp9wd   1/1     Running   0          5d2h

StatefulSet and Service Topology

Identify the StatefulSet and its associated governing Service. The governing Service is essential for providing stable network identities to the Pods. Use the following commands to list them:

kubectl get statefulsets -n <namespace>
kubectl get services -n <namespace>

Example output:

NAME               READY   AGE
my-stateful-app    3/3     2d

NAME               TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
my-stateful-app    ClusterIP   None         <none>        5432/TCP   2d

Note that the Service for a StatefulSet is often headless (ClusterIP: None). This is intentional: it allows Pods to be addressed individually via their stable DNS names.

Prerequisites for Troubleshooting

  • Ensure you have kubectl configured with appropriate permissions to exec into pods and inspect resources.
  • Have access to application logs (via kubectl logs) and cluster events (kubectl get events).
  • For advanced network debugging, tools like tcpdump, netcat, curl, and dig should be available inside your application containers or in a separate debug container.

Quick check 1 of 2

According to the passage, which of the following is NOT listed as a reason StatefulSets are valuable?

The passage lists stable unique network identifiers, stable persistent storage, ordered graceful deployment/scaling, and ordered automated rolling updates—but not automatic horizontal scaling across namespaces.

Safe Configuration Path

When configuring networking for replicated stateful applications, certain choices have a significant impact on reliability and ease of troubleshooting. In this section, we discuss scoped implementation choices that minimize risk and improve observability.

Use a Headless Service for Stable Identities

A headless Service (with clusterIP: None) is the standard way to govern a StatefulSet. It does not load-balance traffic but instead returns the individual Pod IPs when queried via DNS. This allows direct pod-to-pod communication, which is often required by stateful applications for replication and consensus protocols.

Example Service definition:

apiVersion: v1
kind: Service
metadata:
  name: my-stateful-app
  namespace: production
spec:
  clusterIP: None
  selector:
    app: my-stateful-app
  ports:
  - name: replication
    port: 5432
    targetPort: 5432
  - name: metrics
    port: 9104
    targetPort: 9104

With this Service, each Pod gets a DNS entry of the form <pod-name>.<service-name>.<namespace>.svc.cluster.local. For example, my-stateful-app-0.my-stateful-app.production.svc.cluster.local.

To ensure the headless service is functioning correctly, you can verify its configuration:

kubectl get service my-stateful-app -n production -o jsonpath='{.spec.clusterIP}'

Expected output:

None

Also, confirm that the service has a selector that matches the StatefulSet pods:

kubectl get service my-stateful-app -n production -o jsonpath='{.spec.selector}'

Expected output (example):

{"app":"my-stateful-app"}

Expose Individual Pods for External Access (if needed)

If external clients need to reach specific pods (e.g., for direct database connections), consider creating a per-pod Service or using a headless Service with an external load balancer in front. However, be cautious: exposing individual pods can bypass the StatefulSet's orderly management and may lead to split-brain if not carefully coordinated.

One approach is to create a separate Service for each pod using a unique label. For example, you can add a label statefulset.kubernetes.io/pod-name: my-stateful-app-0 to each pod and create a Service with that selector. This allows you to expose each pod individually while maintaining stable DNS names.

Implement Network Policies with Care

NetworkPolicies provide segmentation but can inadvertently block required traffic between replicas. When defining policies, be explicit about allowed ingress and egress for the application's ports. Test changes in a staging environment first.

Example NetworkPolicy allowing intra-StatefulSet traffic:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-stateful-app-replication
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: my-stateful-app
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: my-stateful-app
    ports:
    - protocol: TCP
      port: 5432
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: my-stateful-app
    ports:
    - protocol: TCP
      port: 5432

This policy ensures that all pods with label app: my-stateful-app can communicate with each other on TCP port 5432, but only in the specified direction. Note that the policy above allows ingress from pods with the same label and egress to pods with the same label, but only on port 5432. If your application uses additional ports (e.g., for metrics or admin), you must add rules for those as well.

After applying a NetworkPolicy, always verify that it is correctly selected and check the actual rules using:

kubectl describe networkpolicy allow-stateful-app-replication -n production

This will show you the policy's selectors and rules. Ensure that the podSelector matches your StatefulSet pods and that the allowed ports are correct.

Enable Proper Logging and Monitoring

Ensure your stateful application logs include network-related information (e.g., connection attempts, errors). Integrate with cluster monitoring to track network metrics such as connection latency, packet loss, and dropped packets. This facilitates faster diagnosis when issues arise.

For example, you can configure your application to log at DEBUG level for network components. Additionally, use a monitoring stack like Prometheus and Grafana to scrape metrics from your application and the CNI plugin. Key metrics to watch include:

  • node_network_receive_drop_total and node_network_transmit_drop_total for packet drops.
  • container_network_receive_bytes_total and container_network_transmit_bytes_total for pod traffic.
  • Application-specific metrics like connection counts or replication lag.

Verification and Diagnostics

Once your StatefulSet and Services are deployed, you should verify that networking is functioning correctly. This section provides a series of checks and diagnostic steps, each with expected results.

Verify Pod Readiness and Service Endpoints

Start by ensuring all pods are running and ready:

kubectl get pods -n <namespace> -l app=my-stateful-app

Expected output:

NAME                 READY   STATUS    RESTARTS   AGE
my-stateful-app-0    1/1     Running   0          5m
my-stateful-app-1    1/1     Running   0          4m
my-stateful-app-2    1/1     Running   0          3m

Check that the Service has endpoints corresponding to each pod's IP:

kubectl get endpoints my-stateful-app -n <namespace>

Expected output:

NAME               ENDPOINTS                                           AGE
my-stateful-app    10.244.1.5:5432,10.244.2.3:5432,10.244.3.7:5432     5m

If the endpoints list is empty or missing some pods, investigate the Service's selector and pod labels. Confirm that the labels on the pods match the service selector:

kubectl get pods -n <namespace> --show-labels

Look for the app: my-stateful-app label on each pod. If labels are missing, the service will not select them, and endpoints will be empty.

Test DNS Resolution

DNS is a critical component for pod-to-pod communication. Use nslookup or dig from within a pod to verify that each pod's hostname resolves to the correct IP address. Execute commands in one of the pods:

kubectl exec -it my-stateful-app-0 -n <namespace> -- nslookup my-stateful-app-1.my-stateful-app.<namespace>.svc.cluster.local

Expected output (example):

Server:         10.96.0.10
Address:        10.96.0.10#53

Name:   my-stateful-app-1.my-stateful-app.<namespace>.svc.cluster.local
Address: 10.244.2.3

The IP address should match the pod IP from kubectl get pods -o wide.

For more detailed DNS debugging, you can use dig:

kubectl exec -it my-stateful-app-0 -n <namespace> -- dig +short my-stateful-app-1.my-stateful-app.<namespace>.svc.cluster.local

This should return just the IP address.

Also test the service name without the pod prefix to see all IPs:

kubectl exec -it my-stateful-app-0 -n <namespace> -- nslookup my-stateful-app.<namespace>.svc.cluster.local

For a headless service, this should return multiple A records for each pod.

Check Connectivity Between Pods

Use telnet, nc, or curl to test TCP connectivity on the application port. For example, from pod-0 connect to pod-1 on port 5432:

kubectl exec -it my-stateful-app-0 -n <namespace> -- nc -zv my-stateful-app-1.my-stateful-app.<namespace>.svc.cluster.local 5432

Expected output:

Connection to my-stateful-app-1.my-stateful-app.<namespace>.svc.cluster.local 5432 port [tcp/*] succeeded!

If the connection fails, proceed to the failure modes section.

For a more thorough test, you can use curl if the application speaks HTTP, or use a custom script to send application-specific probes.

Inspect Network Policies

If NetworkPolicies are in place, verify that they allow the required traffic. List all policies in the namespace:

kubectl get networkpolicies -n <namespace>

Describe a specific policy to review its rules:

kubectl describe networkpolicy allow-stateful-app-replication -n <namespace>

Look for any PodSelector or NamespaceSelector that might exclude the pods. Ensure the policy's policyTypes includes both Ingress and Egress if bidirectional communication is needed.

Also, check if any network policy is applied to the pods by examining the pod's labels and the policy's podSelector. You can use:

kubectl get pods -n <namespace> --show-labels

and compare with the policy's selector.

Validate Service Type and ClusterIP

Ensure the Service is headless if you rely on individual pod DNS. Check the Service details:

kubectl get service my-stateful-app -n <namespace> -o yaml

In the output, verify clusterIP: None. If it is not headless, the DNS query for <pod-name>.<service-name> will not resolve to the pod; instead, it may resolve to the service's cluster IP and load balance, which may break stateful communication.

You can also check via command:

kubectl get service my-stateful-app -n <namespace> -o jsonpath='{.spec.clusterIP}'

If the output is an IP address (e.g., 10.96.0.100), then the service is not headless.

Failure Modes and Recovery

Even with careful configuration, networking issues can occur. Understanding common failure modes helps you quickly recover. This section outlines typical problems, their symptoms, diagnostic steps, and recovery actions.

DNS Resolution Failures

Symptom: Pods cannot resolve each other's hostnames; application logs show "unknown host" or connection timeouts.

Possible Causes:

  • CoreDNS pods not running.
  • DNS service (kube-dns) misconfigured.
  • Pod's /etc/resolv.conf has incorrect nameserver.
  • Network policy blocking DNS traffic (UDP/TCP port 53).

Diagnostics:

  • Check CoreDNS pods:
  kubectl get pods -n kube-system -l k8s-app=kube-dns

Expected: all pods Running and Ready.

  • Test DNS from a pod:
  kubectl exec -it <pod> -n <namespace> -- nslookup kubernetes.default

If this fails, the issue is cluster-wide DNS.

  • Inspect /etc/resolv.conf inside the pod:
  kubectl exec -it <pod> -n <namespace> -- cat /etc/resolv.conf

It should contain nameserver 10.96.0.10 (or your cluster's DNS IP) and search <namespace>.svc.cluster.local svc.cluster.local cluster.local.

Recovery:

  • If CoreDNS is down, restart or scale the deployment:
  kubectl rollout restart deployment coredns -n kube-system
  • Ensure NetworkPolicies allow egress to the DNS service on port 53. Example egress rule:
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53
  • If the pod's DNS configuration is wrong, check the pod's dnsPolicy and dnsConfig fields. For most cases, dnsPolicy: ClusterFirst is correct.

Network Policy Denies Traffic

Symptom: Some pods can connect while others cannot, or connections fail from specific namespaces.

Possible Causes:

  • A NetworkPolicy with a restrictive podSelector or namespaceSelector.
  • Missing egress rule to allow responses.

Diagnostics:

  • Describe NetworkPolicies in the namespace and inspect rules.
  kubectl describe networkpolicy -n <namespace>
  • Temporarily delete a suspect policy to see if connectivity returns (in a test environment).
  kubectl delete networkpolicy <policy-name> -n <namespace>

After deletion, test connectivity; if it works, the policy was the issue.

Recovery:

  • Adjust the policy to permit the required traffic. Ensure bidirectional rules are specified if needed. For example, to allow all traffic between pods with label app: my-stateful-app:
  apiVersion: networking.k8s.io/v1
  kind: NetworkPolicy
  metadata:
    name: allow-all-stateful-app
    namespace: production
  spec:
    podSelector:
      matchLabels:
        app: my-stateful-app
    ingress:
    - from:
      - podSelector:
          matchLabels:
            app: my-stateful-app
    egress:
    - to:
      - podSelector:
          matchLabels:
            app: my-stateful-app

Note: This is permissive; adjust ports as needed.

  • Test the policy in a staging environment before applying to production.

Headless Service Misconfiguration

Symptom: Individual pod DNS names do not resolve, or they resolve to a single Cluster IP (if Service is not headless).

Diagnostics:

  • Check Service YAML for clusterIP: None.
  kubectl get service my-stateful-app -n <namespace> -o jsonpath='{.spec.clusterIP}'

If output is an IP, it's not headless.

  • Use nslookup for a pod's hostname; if it returns a service IP, the Service is not headless.

Recovery:

  • Recreate the Service with clusterIP: None. Note: changing a Service type may cause brief downtime; plan accordingly.
  kubectl delete service my-stateful-app -n <namespace>
  kubectl apply -f headless-service.yaml

Pod IP Changes or StatefulSet Scale Issues

Symptom: After a pod restart, other pods cannot reconnect because they cached the old pod IP.

Possible Causes:

  • StatefulSet pods have stable hostnames but their IPs can change on restart if the pod is deleted and recreated on a different node.
  • Applications may cache IP addresses instead of re-resolving DNS.

Diagnostics:

  • Check pod IPs before and after restart:
  kubectl get pods -o wide -n <namespace>

Compare the IP column.

  • Look for connection errors mentioning stale IPs in application logs.

Recovery:

  • Ensure applications use DNS names and refresh DNS lookups periodically.
  • If DNS caching is the issue, adjust the application's resolver settings or use a headless Service with publishNotReadyAddresses: true to minimize IP churn.
  • Consider using a StatefulSet with podManagementPolicy: Parallel to reduce sequential restarts, but be aware of application implications.

Firewall or Node Network Issues

Symptom: All connectivity between pods on different nodes fails, while same-node communication works.

Possible Causes:

  • Node-level firewall blocking overlay network ports.
  • CNI plugin not routing correctly across nodes.
  • Network interface or routing table issues on a node.

Diagnostics:

  • Test cross-node connectivity:
  kubectl exec -it <pod-on-node-1> -n <namespace> -- ping <pod-ip-on-node-2>

If ping fails but same-node ping works, suspect node networking.

  • Check node firewall rules (on the node):
  sudo iptables -L -n
  • Verify CNI plugin status:
  kubectl get pods -n kube-system | grep <cni-name>

Recovery:

  • Adjust firewall rules to allow required ports (e.g., VXLAN UDP 8472, BGP TCP 179, etc., depending on CNI).
  • Restart CNI pods if they are in error state:
  kubectl delete pod -n kube-system <cni-pod>
  • Check node routing table and ensure the overlay interface exists.

Rollback and Recovery Checks

When implementing changes to resolve networking issues, always have a rollback plan. Document the original configuration and test changes in a non-production environment. After applying a fix, run the verification steps from the previous section to confirm resolution. Monitor the application for a period to ensure stability.

For example, before applying a NetworkPolicy change, save the current policy YAML:

kubectl get networkpolicy <name> -n <namespace> -o yaml > backup-policy.yaml

If the change causes issues, restore it:

kubectl apply -f backup-policy.yaml

Quick check 2 of 2

In the hostname pattern for StatefulSet Pods, how is each Pod's hostname constructed?

The passage states the hostname pattern is `$(statefulset name)-$(ordinal)`, e.g. web-0, web-1, web-2.

Operations Checklist

Use the following checklist as a repeatable process for operating and reviewing the networking configuration of your replicated stateful application. Running these checks periodically can help catch issues before they impact availability. Assign a single accountable owner for each major decision or checklist item, and revisit at the specified frequency.

Daily Checks

  • [ ] All StatefulSet pods are running and ready.
  • Owner: On-call engineer (rotating)
  • How: Run kubectl get pods -n <namespace> -l app=my-stateful-app and verify all replicas are 1/1 Ready.
  • [ ] Service endpoints match the number of replicas.
  • Owner: On-call engineer
  • How: kubectl get endpoints my-stateful-app -n <namespace> and confirm the number of endpoints equals desired replicas.
  • [ ] No recent network-related errors in application logs.
  • Owner: Application developer (e.g., Priya Shah, Engineering Lead)
  • How: Check centralized logging for patterns like "connection refused", "timeout", "unknown host" within the last 24 hours.

Weekly Checks

  • [ ] DNS resolution for all pod hostnames succeeds from multiple pods.
  • Owner: DevOps engineer (e.g., John Doe, Platform Engineer)
  • How: Run a script that execs into each pod and does nslookup for every other pod's hostname.
  • [ ] TCP connectivity on all required ports between a sample of pods.
  • Owner: DevOps engineer
  • How: Use nc -zv from a few pods to others, covering all service ports (e.g., 5432, 9104).
  • [ ] NetworkPolicy rules reviewed and still align with application needs.
  • Owner: Security engineer (e.g., Maria Garcia, Security Architect)
  • How: Examine each policy in the namespace and compare with documented application requirements. Remove or update stale rules.

Before Any Change

  • [ ] Capture current StatefulSet, Service, and NetworkPolicy YAMLs for rollback.
  • Owner: Person making the change (must be individually identified)
  • How: Use kubectl get <resource> -o yaml > backup-<resource>.yaml.
  • [ ] Test configuration changes in a staging cluster.
  • Owner: Person making the change
  • How: Deploy the same manifest in a staging environment and run the verification steps.

After Any Change

  • [ ] Run verification commands (endpoints, DNS, connectivity).
  • Owner: Person making the change
  • How: Follow the steps in "Verification and Diagnostics" section.
  • [ ] Monitor application metrics for expected behavior.
  • Owner: On-call engineer
  • How: Watch dashboards for 30 minutes post-change; look for anomalies in connections, latency, or errors.
  • [ ] Document the change and its impact.
  • Owner: Person making the change
  • How: Update the change log or ticket with details and outcomes.

Troubleshooting Quick Reference

SymptomLikely CauseFirst Command to Run
Pods cannot resolve each otherDNS issueskubectl exec -it <pod> -- nslookup <other-pod-hostname>
Connection refused on app portApplication not listening or wrong portkubectl exec -it <pod> -- nc -zv <target> <port>
Partial connectivityNetworkPolicy restrictionskubectl describe networkpolicy -n <namespace>
All cross-node traffic failsCNI or node firewallkubectl get pods -n kube-system -l k8s-app=calico-node (or your CNI label)
Service endpoints emptyLabel selector mismatchkubectl get pods --show-labels -n <namespace>

Common Pitfalls and How to Avoid Them

Many networking issues in stateful applications stem from a few recurring mistakes. Being aware of these pitfalls can help you avoid them and quickly recognize them when they occur.

1. Using a ClusterIP Service Instead of Headless

Why it happens: Developers may reuse a standard Service template or not understand the need for headless services with StatefulSets.

How to avoid: Always define a headless Service (with clusterIP: None) for StatefulSets that require direct pod-to-pod communication. Document this as a standard.

Recovery: If a non-headless Service is in place, recreate it as headless. Note that this may cause a brief disruption as DNS records update.

2. Overly Restrictive Network Policies

Why it happens: In the name of security, teams may apply broad deny-all policies or too-specific selectors that unintentionally block replication traffic.

How to avoid: Start with a policy that allows required traffic explicitly and test thoroughly in staging. Use labels carefully and ensure both ingress and egress are considered.

Recovery: Temporarily remove or modify the policy to restore connectivity, then refine and reapply with proper rules.

3. Ignoring DNS Caching in Applications

Why it happens: Many applications resolve hostnames once and cache the IP indefinitely. When pods restart and IPs change, connections fail.

How to avoid: Configure applications to use DNS names and refresh lookups appropriately. If possible, set a low TTL for DNS records (Kubernetes default is 30 seconds for headless services). Consider using publishNotReadyAddresses: true to keep IPs stable for a longer time during rolling updates.

Recovery: Restart the application pods to flush DNS caches, or if the application supports it, trigger a manual re-resolution.

4. Mislabeling Pods or Services

Why it happens: Manual edits or CI/CD misconfigurations can cause label mismatches, leading to missing endpoints.

How to avoid: Use declarative manifests and version control. Implement a CI check that validates selectors against pod labels.

Recovery: Correct the labels on pods or service selector and reapply.

5. Not Accounting for CNI-Specific Requirements

Why it happens: Different CNI plugins have different default ports and behaviors. For example, Calico uses BGP (TCP 179) by default, while Flannel uses VXLAN (UDP 8472). Firewall rules may not be updated.

How to avoid: Document the specific CNI in use and its network requirements. Ensure node firewalls allow those ports.

Recovery: Open the required ports on nodes or adjust CNI configuration.

6. Scaling StatefulSet Without Considering Ordering

Why it happens: StatefulSets by default use ordered pod management (OrderedReady), which can cause delays or issues during scale-down if not understood.

How to avoid: If your application can handle unordered scaling, set podManagementPolicy: Parallel to speed up operations. Otherwise, plan for sequential scaling and ensure readiness probes are correct.

Recovery: If a scale-down causes issues, check the StatefulSet's podManagementPolicy and adjust if appropriate.

Conclusion

Networking for replicated stateful applications in Kubernetes requires careful planning and ongoing vigilance. By understanding the role of headless Services, stable pod identities, and NetworkPolicies, you can build a resilient environment. When issues arise, the systematic diagnostic steps outlined in this guide, from verifying DNS resolution to inspecting network policies, will help you identify and resolve problems efficiently.

Remember to keep your environment inventory up to date, test configuration changes in a safe manner, and maintain a rollback plan. Use the operations checklist to incorporate regular checks into your routine. With these practices, you can ensure that your stateful applications remain connected, consistent, and highly available.

For further learning, explore the official Kubernetes documentation on StatefulSets and Services, and practice troubleshooting in a sandbox cluster to build confidence.

Related Research

Article Quality Score

Reader usefulness 100%
  • check_circle Reader-ready guide
  • check_circle Practical examples included
  • check_circle Clean SEO article URL