Kubernetes Network Policy Troubleshooting: A Practical Guide with Examples

Intro

When network traffic is unexpectedly blocked or allowed in Kubernetes, Network Policies are often the first suspect. Yet troubleshooting them can be frustrating because they are silent: no logs, no events, no obvious error messages. This guide provides a practical, example-driven approach to diagnosing and fixing Network Policy issues, aimed at developers, DevOps engineers, and platform teams.

We will start by understanding the prerequisites and environment, then move through safe configuration practices, verification techniques, failure modes, and a repeatable troubleshooting checklist. Along the way, we provide concrete kubectl commands, YAML examples, and expected outputs to help you move from problem to resolution quickly and confidently.

By the end, you will be able to:

• Determine whether a Network Policy is the actual cause of a connectivity problem.
• Inspect and validate Network Policy resources.
• Test connectivity in a controlled manner.
• Implement fixes with minimal blast radius.
• Verify that the fix works and document recovery steps.

We assume you have a working Kubernetes cluster and basic familiarity with kubectl.

Version and Environment Inventory

Before touching any configuration, establish the exact environment. This prevents misdiagnosis and ensures any fix is appropriate for your cluster version and network plugin.

Key steps:

1. Check Kubernetes version:

   kubectl version --short

Expected output (example):

   Client Version: v1.28.2
   Server Version: v1.28.2

1. Identify the CNI plugin and verify Network Policy support: Network Policies are enforced by the CNI provider (e.g., Calico, Cilium, Weave, Antrea). Some plugins have limitations. Check the plugin pods:

   kubectl get pods -n kube-system | grep -E 'calico|cilium|weave|antrea'

Example output:

   calico-node-abcde                  1/1     Running   0          10d
   calico-kube-controllers-12345      1/1     Running   0          10d

1. Confirm NetworkPolicy API availability:

   kubectl api-resources | grep networkpolicies

Example output:

   networkpolicies                   netpol       networking.k8s.io/v1       true         NetworkPolicy

1. List existing Network Policies in the relevant namespace:

   kubectl get networkpolicies -n <namespace>

Example (namespace default):

   NAME            POD-SELECTOR          AGE
   deny-all        <none>                5d
   allow-web       app=web               2d

1. Check the namespace labels: Network Policies often rely on namespace selectors. Verify labels:

   kubectl get namespace <namespace> --show-labels

1. Inspect the pods involved in the connectivity issue:

   kubectl get pods -n <namespace> -o wide

Note the pod IPs and node placement. If pods are on different nodes, ensure the CNI is functioning across nodes.

A common pitfall is assuming the Network Policy is the problem when in fact the CNI is not enforcing it (e.g., because the plugin is not installed or misconfigured). Always verify the CNI pods are healthy and the policy CRD is present.

Safe Configuration Path

Making changes to Network Policies can have immediate effects on production traffic. Follow a safe, incremental approach:

1. Start with a read-only assessment: Use kubectl describe and kubectl get -o yaml to understand current policies.

   kubectl describe networkpolicy <policy-name> -n <namespace>

Example output snippet:

   Spec:
     PodSelector:     app=web
     Ingress:
       From:
         PodSelector: app=api
       Ports:
         Port: 80
         Protocol: TCP

1. Back up existing policies before changes:

   kubectl get networkpolicy <policy-name> -n <namespace> -o yaml > policy-backup.yaml

1. Make minimal changes: Instead of rewriting the entire policy, adjust only the necessary selector or rule. For example, if you need to allow traffic from a new namespace, add an ipBlock or namespaceSelector without disrupting existing rules.

1. Test in a non-production namespace first: Duplicate the setup in a test namespace and verify behavior before applying to production.

1. Apply the change and monitor: Use kubectl apply -f new-policy.yaml and then immediately test connectivity.

Example: Adding an allow rule for a monitoring namespace

Original policy (denies all ingress to app=web):

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all
  namespace: default
spec:
  podSelector:
    matchLabels:
      app: web
  policyTypes:
  - Ingress

Revised policy to allow ingress from namespace monitoring on port 9090:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all
  namespace: default
spec:
  podSelector:
    matchLabels:
      app: web
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          purpose: monitoring
    ports:
    - port: 9090
      protocol: TCP
  policyTypes:
  - Ingress

Apply and verify:

kubectl apply -f revised-policy.yaml
kubectl get networkpolicy deny-all -n default -o yaml

Pitfall: Forgetting that Network Policies are additive but their selectors are ORed within a rule and ANDed across rules. A small change can inadvertently open or close traffic. Always review the full policy after editing.

Verification and Diagnostics

To confirm whether a Network Policy is causing an issue, you need to test connectivity from one pod to another. Use these techniques:

1. Create a test pod with network tools:

   kubectl run test-pod --rm -it --image=nicolaka/netshoot -- /bin/bash

Inside the pod, use curl, wget, nc, or telnet to test connections.

1. Test connectivity to a service from within the cluster:

   kubectl exec -it <source-pod> -n <namespace> -- curl -v http://<service-name>.<namespace>.svc.cluster.local:80

Expected output if blocked:

   *   Trying 10.96.0.10:80...
   * connect to 10.96.0.10 port 80 failed: Connection timed out

Expected output if allowed:

   * Connected to 10.96.0.10 (10.96.0.10) port 80
   > GET / HTTP/1.1
   ...
   < HTTP/1.1 200 OK

Get the target pod IP:

1. Check if the traffic is blocked by testing directly to pod IP:

   kubectl get pod <target-pod> -n <namespace> -o wide

Then from the source pod:

   curl http://<target-pod-ip>:80

If Service works but direct pod IP fails, the policy might be selecting the service's endpoints differently.

1. Inspect Network Policy events and logs: Network Policies themselves do not produce events, but the CNI may log drops. For Calico, check calico-node logs:

   kubectl logs -n kube-system <calico-node-pod> | grep -i drop | tail -20

This may show denied flows with source and destination IPs.

The effective policy is not dynamic; it depends on the selectors. Verify that the target pod matches the policy's podSelector.

1. Use kubectl describe to see if the policy is applied to the correct pods:

   kubectl get pods -n <namespace> --show-labels
   kubectl describe networkpolicy <policy-name> -n <namespace>

Real-world scenario: A developer complained that a web pod could not reach a database pod. We found that the database pod had label app=db, but the Network Policy selected app=database. The policy did not apply, and another deny-all policy blocked traffic. Fixing the label resolved the issue.

Failure Modes and Recovery

Let's outline common failure modes and how to recover from them.

1. Policy Not Enforced (Traffic Still Flows)

Symptom: You applied a deny-all policy, but traffic still succeeds.

Causes:

• CNI does not support Network Policies.
• Policy is in a different namespace than the targeted pods.
• Policy's podSelector does not match any pods.
• Policy is missing policyTypes (default is only Ingress).

Diagnosis:

kubectl get networkpolicy -A
kubectl describe networkpolicy <policy-name> -n <namespace>
kubectl get pods -n <namespace> --show-labels

Recovery: Correct the selector or namespace. Ensure the policy's podSelector matches the intended pods. If using a namespace selector, ensure the namespace has the correct labels.

2. Traffic Unexpectedly Blocked

Symptom: After applying a policy, legitimate traffic fails.

Causes:

• Missing ingress rule for a required source.
• Ingress rule with an empty from field (which means allow nothing, not allow all).
• Port mismatch (e.g., service uses 8080 but policy allows 80).
• Using ipBlock with wrong CIDR.

Diagnosis: Test connectivity, review policy YAML carefully, compare with the service definition.

Recovery: Modify the policy to add the necessary rule. If in doubt, temporarily delete the policy to restore traffic, then reapply incrementally.

Example: A policy intended to allow only from within the namespace but accidentally included a podSelector: {} (empty), which selects all pods in the namespace, effectively allowing all namespace traffic. This is a common mistake. To allow only from specific pods, use their labels.

3. Policy Applies but Partial Connectivity

Symptom: Some traffic works, some fails.

Causes: Multiple policies combine; some rules conflict. Egress rules may be missing if you applied an egress-deny but did not allow DNS (UDP 53) or required IPs.

Diagnosis: List all policies in the namespace and examine their combined effect. Use kubectl get networkpolicy -n <namespace> and then describe each. Tools like kubectl-neat or visualizers (e.g., Octant, Lens) can help.

Recovery: Adjust policies to include missing rules. For egress, remember to allow DNS and any necessary external endpoints.

4. CNI Logs Show Drops but Policy Seems Correct

Symptom: Calico/Cilium logs show denied packets, but the policy appears to allow them.

Causes: Policy ordering? Not applicable, Network Policies are additive. More likely: source or destination IP in the drop log does not match the pods you think are communicating. Could be a Service IP (ClusterIP) being used as destination; Network Policy applies to pod IPs, not service IPs, but the CNI may drop if the backend pod is not allowed.

Diagnosis: Check the actual pod IPs and the flow. Use kubectl get endpoints <service> to see backend pod IPs.

Recovery: Ensure the policy allows traffic to the backend pods from the source pods.

Operations Checklist

Use this checklist for systematic troubleshooting:

Owner and frequency: The platform or network team typically owns Network Policy definitions. Policies should be reviewed quarterly and after any significant change to application architecture. A designated engineer (e.g., the network lead or platform engineer) should own the policy repository and review process.

Common Pitfalls

Here are the most frequent mistakes and how to avoid them:

1. Barren podSelector: {}: An empty pod selector selects all pods in the namespace. This is rarely intended, especially in a deny-all policy. Always specify labels.

1. Forgetting policyTypes: If you omit policyTypes, only Ingress is enforced. To control egress, you must explicitly include Egress.

1. Misunderstanding selector semantics: Within a single rule, from items are ORed: if any matches, the rule applies. Across different rules, they are ANDed: traffic must satisfy all rules. This often leads to over-restriction.

1. Using ipBlock incorrectly: ipBlock cannot be used with podSelector or namespaceSelector in the same from item; they are separate array elements. Also, ipBlock CIDR must include the pod CIDR or external IP exactly.

1. Not accounting for namespace isolation: A policy in one namespace does not affect pods in another unless the other namespace's labels are used. Ensure cross-namespace traffic rules are defined in the target pod's namespace.

1. Ignoring DNS in egress policies: If you create a deny-all egress policy without allowing UDP 53 to your DNS server, all external resolution fails, causing cascading issues.

1. Not testing after changes: Always test from a known source pod to verify the intended behavior. Use ephemeral test containers to avoid impacting production.

1. Assuming Network Policies are enforced by default: They are not. You need a CNI that supports them and the policies must be applied.

Conclusion

Kubernetes Network Policy troubleshooting is a systematic process that requires careful observation, incremental changes, and thorough verification. By building a solid understanding of your environment, using safe configuration practices, and employing the diagnostic techniques outlined here, you can quickly resolve connectivity issues caused by network policies.

Remember to always:

• Verify that the CNI is functioning and Network Policies are supported.
• Inspect the actual labels and selectors before assuming.
• Test connectivity from inside the cluster with tools like netshoot.
• Make minimal changes and back up policies before editing.
• Document every change and its effect for future reference.

With practice, these steps become second nature, turning network policy troubleshooting from a daunting task into a routine procedure. For further learning, explore the Kubernetes documentation on Network Policies and your CNI provider's specifics, as implementations vary.