## Intro When network traffic is unexpectedly blocked or allowed in Kubernetes, Network Policies are often the first suspect. Yet troubleshooting them can be frustrating because they are silent: no logs, no events, no obvious error messages. This guide provides a practical, example-driven approach to diagnosing and fixing Network Policy issues, aimed at developers, DevOps engineers, and platform teams. We will start by understanding the prerequisites and environment, then move through safe configuration practices, verification techniques, failure modes, and a repeatable troubleshooting checklist. Along the way, we provide concrete `kubectl` commands, YAML examples, and expected outputs to help you move from problem to resolution quickly and confidently. By the end, you will be able to: - Determine whether a Network Policy is the actual cause of a connectivity problem. - Inspect and validate Network Policy resources. - Test connectivity in a controlled manner. - Implement fixes with minimal blast radius. - Verify that the fix works and document recovery steps. We assume you have a working Kubernetes cluster and basic familiarity with `kubectl`. ## Version and Environment Inventory Before touching any configuration, establish the exact environment. This prevents misdiagnosis and ensures any fix is appropriate for your cluster version and network plugin. **Key steps:** 1. **Check Kubernetes version:** ```bash kubectl version --short ``` Expected output (example): ``` Client Version: v1.28.2 Server Version: v1.28.2 ``` 2. **Identify the CNI plugin and verify Network Policy support:** Network Policies are enforced by the CNI provider (e.g., Calico, Cilium, Weave, Antrea). Some plugins have limitations. Check the plugin pods: ```bash kubectl get pods -n kube-system | grep -E 'calico|cilium|weave|antrea' ``` Example output: ``` calico-node-abcde 1/1 Running 0 10d calico-kube-controllers-12345 1/1 Running 0 10d ``` 3. **Confirm NetworkPolicy API availability:** ```bash kubectl api-resources | grep networkpolicies ``` Example output: ``` networkpolicies netpol networking.k8s.io/v1 true NetworkPolicy ``` 4. **List existing Network Policies in the relevant namespace:** ```bash kubectl get networkpolicies -n ``` Example (namespace `default`): ``` NAME POD-SELECTOR AGE deny-all 5d allow-web app=web 2d ``` 5. **Check the namespace labels:** Network Policies often rely on namespace selectors. Verify labels: ```bash kubectl get namespace --show-labels ``` 6. **Inspect the pods involved in the connectivity issue:** ```bash kubectl get pods -n -o wide ``` Note the pod IPs and node placement. If pods are on different nodes, ensure the CNI is functioning across nodes. A common pitfall is assuming the Network Policy is the problem when in fact the CNI is not enforcing it (e.g., because the plugin is not installed or misconfigured). Always verify the CNI pods are healthy and the policy CRD is present. ## Safe Configuration Path Making changes to Network Policies can have immediate effects on production traffic. Follow a safe, incremental approach: 1. **Start with a read-only assessment:** Use `kubectl describe` and `kubectl get -o yaml` to understand current policies. ```bash kubectl describe networkpolicy -n ``` Example output snippet: ``` Spec: PodSelector: app=web Ingress: From: PodSelector: app=api Ports: Port: 80 Protocol: TCP ``` 2. **Back up existing policies before changes:** ```bash kubectl get networkpolicy -n -o yaml > policy-backup.yaml ``` 3. **Make minimal changes:** Instead of rewriting the entire policy, adjust only the necessary selector or rule. For example, if you need to allow traffic from a new namespace, add an `ipBlock` or `namespaceSelector` without disrupting existing rules. 4. **Test in a non-production namespace first:** Duplicate the setup in a test namespace and verify behavior before applying to production. 5. **Apply the change and monitor:** Use `kubectl apply -f new-policy.yaml` and then immediately test connectivity. **Example: Adding an allow rule for a monitoring namespace** Original policy (denies all ingress to `app=web`): ```yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: deny-all namespace: default spec: podSelector: matchLabels: app: web policyTypes: - Ingress ``` Revised policy to allow ingress from namespace `monitoring` on port 9090: ```yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: deny-all namespace: default spec: podSelector: matchLabels: app: web ingress: - from: - namespaceSelector: matchLabels: purpose: monitoring ports: - port: 9090 protocol: TCP policyTypes: - Ingress ``` Apply and verify: ```bash kubectl apply -f revised-policy.yaml kubectl get networkpolicy deny-all -n default -o yaml ``` **Pitfall:** Forgetting that Network Policies are additive but their selectors are ORed within a rule and ANDed across rules. A small change can inadvertently open or close traffic. Always review the full policy after editing. ## Verification and Diagnostics To confirm whether a Network Policy is causing an issue, you need to test connectivity from one pod to another. Use these techniques: 1. **Create a test pod with network tools:** ```bash kubectl run test-pod --rm -it --image=nicolaka/netshoot -- /bin/bash ``` Inside the pod, use `curl`, `wget`, `nc`, or `telnet` to test connections. 2. **Test connectivity to a service from within the cluster:** ```bash kubectl exec -it -n -- curl -v http://..svc.cluster.local:80 ``` Expected output if blocked: ``` * Trying 10.96.0.10:80... * connect to 10.96.0.10 port 80 failed: Connection timed out ``` Expected output if allowed: ``` * Connected to 10.96.0.10 (10.96.0.10) port 80 > GET / HTTP/1.1 ... < HTTP/1.1 200 OK ``` 3. **Check if the traffic is blocked by testing directly to pod IP:** Get the target pod IP: ```bash kubectl get pod -n -o wide ``` Then from the source pod: ```bash curl http://:80 ``` If Service works but direct pod IP fails, the policy might be selecting the service's endpoints differently. 4. **Inspect Network Policy events and logs:** Network Policies themselves do not produce events, but the CNI may log drops. For Calico, check `calico-node` logs: ```bash kubectl logs -n kube-system | grep -i drop | tail -20 ``` This may show denied flows with source and destination IPs. 5. **Use `kubectl describe` to see if the policy is applied to the correct pods:** The effective policy is not dynamic; it depends on the selectors. Verify that the target pod matches the policy's `podSelector`. ```bash kubectl get pods -n --show-labels kubectl describe networkpolicy -n ``` **Real-world scenario:** A developer complained that a web pod could not reach a database pod. We found that the database pod had label `app=db`, but the Network Policy selected `app=database`. The policy did not apply, and another deny-all policy blocked traffic. Fixing the label resolved the issue. ## Failure Modes and Recovery Let's outline common failure modes and how to recover from them. ### 1. Policy Not Enforced (Traffic Still Flows) **Symptom:** You applied a deny-all policy, but traffic still succeeds. **Causes:** - CNI does not support Network Policies. - Policy is in a different namespace than the targeted pods. - Policy's `podSelector` does not match any pods. - Policy is missing `policyTypes` (default is only `Ingress`). **Diagnosis:** ```bash kubectl get networkpolicy -A kubectl describe networkpolicy -n kubectl get pods -n --show-labels ``` **Recovery:** Correct the selector or namespace. Ensure the policy's `podSelector` matches the intended pods. If using a namespace selector, ensure the namespace has the correct labels. ### 2. Traffic Unexpectedly Blocked **Symptom:** After applying a policy, legitimate traffic fails. **Causes:** - Missing ingress rule for a required source. - Ingress rule with an empty `from` field (which means allow nothing, not allow all). - Port mismatch (e.g., service uses 8080 but policy allows 80). - Using `ipBlock` with wrong CIDR. **Diagnosis:** Test connectivity, review policy YAML carefully, compare with the service definition. **Recovery:** Modify the policy to add the necessary rule. If in doubt, temporarily delete the policy to restore traffic, then reapply incrementally. **Example:** A policy intended to allow only from within the namespace but accidentally included a `podSelector: {}` (empty), which selects all pods in the namespace, effectively allowing all namespace traffic. This is a common mistake. To allow only from specific pods, use their labels. ### 3. Policy Applies but Partial Connectivity **Symptom:** Some traffic works, some fails. **Causes:** Multiple policies combine; some rules conflict. Egress rules may be missing if you applied an egress-deny but did not allow DNS (UDP 53) or required IPs. **Diagnosis:** List all policies in the namespace and examine their combined effect. Use `kubectl get networkpolicy -n ` and then `describe` each. Tools like `kubectl-neat` or visualizers (e.g., Octant, Lens) can help. **Recovery:** Adjust policies to include missing rules. For egress, remember to allow DNS and any necessary external endpoints. ### 4. CNI Logs Show Drops but Policy Seems Correct **Symptom:** Calico/Cilium logs show denied packets, but the policy appears to allow them. **Causes:** Policy ordering? Not applicable, Network Policies are additive. More likely: source or destination IP in the drop log does not match the pods you think are communicating. Could be a Service IP (ClusterIP) being used as destination; Network Policy applies to pod IPs, not service IPs, but the CNI may drop if the backend pod is not allowed. **Diagnosis:** Check the actual pod IPs and the flow. Use `kubectl get endpoints ` to see backend pod IPs. **Recovery:** Ensure the policy allows traffic to the backend pods from the source pods. ## Operations Checklist Use this checklist for systematic troubleshooting: | Step | Action | Command / Check | Expected Result | |------|--------|-----------------|-----------------| | 1 | Verify cluster and CNI | `kubectl get pods -n kube-system` | All CNI pods Running | | 2 | Confirm NetworkPolicy API | `kubectl api-resources \| grep networkpolicies` | Resource listed | | 3 | Identify affected pods and labels | `kubectl get pods -n --show-labels` | Labels match policy selectors | | 4 | List all policies in namespace | `kubectl get networkpolicy -n ` | Policies present as expected | | 5 | Describe the relevant policy | `kubectl describe networkpolicy -n ` | Rules match intended access | | 6 | Test connectivity from source pod | `kubectl exec -- curl -v ` | Expected HTTP response or timeout | | 7 | Test connectivity directly to pod IP | `kubectl exec -- curl :port` | Same as service test, helps isolate | | 8 | Check CNI logs for drops | `kubectl logs -n kube-system \| grep -i drop` | No unexpected drops or drops match policy | | 9 | Verify namespace labels if using namespaceSelector | `kubectl get namespace --show-labels` | Required labels present | | 10 | Apply fix and re-test | `kubectl apply -f fix.yaml` | Connectivity restored | **Owner and frequency:** The platform or network team typically owns Network Policy definitions. Policies should be reviewed quarterly and after any significant change to application architecture. A designated engineer (e.g., the network lead or platform engineer) should own the policy repository and review process. ## Common Pitfalls Here are the most frequent mistakes and how to avoid them: 1. **Barren `podSelector: {}`:** An empty pod selector selects all pods in the namespace. This is rarely intended, especially in a deny-all policy. Always specify labels. 2. **Forgetting `policyTypes`:** If you omit `policyTypes`, only Ingress is enforced. To control egress, you must explicitly include `Egress`. 3. **Misunderstanding selector semantics:** Within a single rule, `from` items are ORed: if any matches, the rule applies. Across different rules, they are ANDed: traffic must satisfy all rules. This often leads to over-restriction. 4. **Using `ipBlock` incorrectly:** `ipBlock` cannot be used with `podSelector` or `namespaceSelector` in the same `from` item; they are separate array elements. Also, `ipBlock` CIDR must include the pod CIDR or external IP exactly. 5. **Not accounting for namespace isolation:** A policy in one namespace does not affect pods in another unless the other namespace's labels are used. Ensure cross-namespace traffic rules are defined in the target pod's namespace. 6. **Ignoring DNS in egress policies:** If you create a deny-all egress policy without allowing UDP 53 to your DNS server, all external resolution fails, causing cascading issues. 7. **Not testing after changes:** Always test from a known source pod to verify the intended behavior. Use ephemeral test containers to avoid impacting production. 8. **Assuming Network Policies are enforced by default:** They are not. You need a CNI that supports them and the policies must be applied. ## Conclusion Kubernetes Network Policy troubleshooting is a systematic process that requires careful observation, incremental changes, and thorough verification. By building a solid understanding of your environment, using safe configuration practices, and employing the diagnostic techniques outlined here, you can quickly resolve connectivity issues caused by network policies. Remember to always: - Verify that the CNI is functioning and Network Policies are supported. - Inspect the actual labels and selectors before assuming. - Test connectivity from inside the cluster with tools like `netshoot`. - Make minimal changes and back up policies before editing. - Document every change and its effect for future reference. With practice, these steps become second nature, turning network policy troubleshooting from a daunting task into a routine procedure. For further learning, explore the Kubernetes documentation on Network Policies and your CNI provider's specifics, as implementations vary.