## Intro Kubernetes Network Policies are essential for controlling pod-to-pod traffic and enforcing zero-trust networking inside a cluster. However, defining policies is only half the battle. Without effective monitoring and alerting, you cannot know whether policies are actually enforced, misconfigured, or silently dropping legitimate traffic. This guide provides a practical, step-by-step approach to monitoring Kubernetes Network Policies using native tools, Prometheus, and alerting systems. You will learn how to collect metrics, build dashboards, set up alerts, and respond to incidents with real examples and commands. We will focus on common scenarios such as detecting denied traffic, auditing policy changes, and troubleshooting connectivity issues caused by overly restrictive rules. The article is aimed at platform engineers, SREs, and DevOps teams running Kubernetes in production. Before making any changes, always establish observability. The goal is to make network policy behavior visible, enable rapid diagnosis, and ensure that any configuration change is safe and reversible. ## Version and Environment Inventory Before implementing monitoring, document your environment. This inventory will help you choose the right tools and understand compatibility. ### Cluster and CNI Details Network Policy enforcement depends on the CNI plugin. Common CNIs that support Network Policies include Calico, Cilium, Weave Net, and Antrea. Verify your CNI and its version: kubectl get nodes -o wide # Check CNI pods kubectl get pods -n kube-system | grep -E 'calico|cilium|weave|antrea' Example output (Calico): calico-node-abcde 1/1 Running 0 5d calico-kube-controllers-67890 1/1 Running 0 5d Check the CNI configuration in the node if needed, but the pod image tag often reveals the version: kubectl get ds -n kube-system calico-node -o jsonpath='{.spec.template.spec.containers[0].image}' ### Kubernetes Version and API Availability NetworkPolicy is a stable API (networking.k8s.io/v1) since Kubernetes 1.7. Ensure your cluster is recent enough: kubectl version --short Example: Client Version: v1.28.2 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.28.3 ### Existing Network Policies List all Network Policies across namespaces to understand your current state: kubectl get networkpolicies -A Example output: NAMESPACE NAME POD-SELECTOR AGE default deny-all 10d kube-system allow-dns app=kube-dns 10d app-ns allow-frontend-tiers tier=frontend 2d Note any default-deny policies and their selectors. This is your baseline. ### Monitoring Tools Available Check if Prometheus is installed: kubectl get pods -n monitoring | grep prometheus If not, you may need to install Prometheus and configure it to scrape metrics from your CNI or use a ServiceMonitor. ## Safe Configuration Path Monitoring Network Policies should be done incrementally to avoid disrupting production traffic. Follow a safe path: observe, enable metrics, test, and then alert. ### Step 1: Enable Metrics Collection from CNI Most CNIs expose Prometheus metrics. For Calico, the Felix agent (running on each node) provides detailed policy metrics. Enable metrics reporting by applying the following configuration if not already enabled. For Calico, you can use the built-in FelixConfiguration: apiVersion: crd.projectcalico.org/v1 kind: FelixConfiguration metadata: name: default spec: prometheusMetricsEnabled: true prometheusMetricsPort: 9091 Apply with kubectl apply -f felix-config.yaml . Verify by checking a node's Felix port: kubectl get felixconfigurations default -o yaml | grep prometheus For Cilium, metrics are enabled by default, but verify: kubectl -n kube-system exec ds/cilium -- cilium status | grep Metrics ### Step 2: Create a ServiceMonitor for Prometheus If you use the Prometheus Operator, create a ServiceMonitor to scrape Felix metrics: apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: calico-felix namespace: monitoring spec: selector: matchLabels: k8s-app: calico-node endpoints: - port: metrics interval: 30s Ensure the calico-node service exposes the metrics port. If not, create a service: apiVersion: v1 kind: Service metadata: name: calico-node-metrics namespace: kube-system labels: k8s-app: calico-node spec: selector: k8s-app: calico-node ports: - name: metrics port: 9091 targetPort: 9091 Apply and verify Prometheus targets: kubectl -n monitoring port-forward svc/prometheus-operated 9090:9090 # Then open http://localhost:9090/targets and confirm calico-felix is up. ### Step 3: Define Minimal Test Traffic Before relying on alerts, generate known traffic that should be allowed or denied by a test policy. Create a test namespace and pods: kubectl create ns policy-test kubectl run allowed-client --image=busybox --restart=Never -- sleep 3600 kubectl run denied-client --image=busybox --restart=Never -- sleep 3600 kubectl run server --image=nginx --restart=Never Apply a NetworkPolicy that allows only allowed-client to access server: apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: test-allow-specific namespace: policy-test spec: podSelector: matchLabels: run: server ingress: - from: - podSelector: matchLabels: run: allowed-client ports: - port: 80 Now test connectivity: kubectl exec -it allowed-client -- wget -qO- --timeout=2 http://server # Should return HTML kubectl exec -it denied-client -- wget -qO- --timeout=2 http://server # Should fail or time out This baseline helps validate that alerts correspond to actual traffic. ## Verification and Diagnostics Once metrics are scraped and test traffic is flowing, you need to verify that policy enforcement is visible and diagnose any issues. ### Key Metrics to Monitor Different CNIs expose different metric names. For Calico Felix, important metrics include: - felix_iptables_restore_errors : errors applying iptables rules. - felix_denied_packets_total : count of packets denied by policy (if iptables action is set to LOG). - felix_ipset_errors : errors manipulating IP sets. - felix_active_local_endpoints : number of local endpoints. For Cilium: - cilium_drop_count_total : dropped packets due to policy. - cilium_policy_endpoint_enforcement_status : enforcement status per endpoint. Query examples in Prometheus: # Total denied packets in last 5 minutes sum(increase(felix_denied_packets_total[5m])) # Rate of drops in Cilium sum(rate(cilium_drop_count_total[5m])) by (reason) ### PromQL for Policy Auditing To see which pods are generating drops, use labels. Calico metrics often include labels like source_namespace , source_pod , dest_namespace , dest_pod . Example: topk(10, sum by (source_namespace, source_pod) (rate(felix_denied_packets_total[5m]))) This gives you the top sources of denied traffic. ### Using kubectl to Inspect Policy Status While CNI metrics show enforcement, you can also inspect policy objects and endpoints: kubectl describe networkpolicy test-allow-specific -n policy-test Output includes the pod selector and ingress rules. To see if a policy is applied to a pod, check the pod's annotations or use CNI CLI tools: For Calico: kubectl exec -n kube-system calicoctl -- calicoctl get profile -A For Cilium, use cilium endpoint list to see policy enforcement status: kubectl -n kube-system exec ds/cilium -- cilium endpoint list Look for Policy enforcement column: Ingress, Egress, or Both. ### Diagnosing Connectivity Issues If traffic is unexpectedly blocked, follow this sequence: - Check if NetworkPolicy exists and selects the pod: kubectl get networkpolicy -n - Describe the policy and verify rules: kubectl describe networkpolicy -n For Calico Felix: - Test with a known client pod (as earlier). - Check metrics for drops: query Prometheus for denied packets in that namespace. - Check CNI logs: kubectl logs -n kube-system ds/calico-node -c calico-node --tail=50 For Cilium: kubectl -n kube-system logs ds/cilium --tail=50 Quick check 2 of 2 What is recommended to start with when strict network isolation between tenants is required? - A policy that allows all traffic within a namespace - A default policy that denies communication between pods with another rule allowing DNS queries - A policy that allows all traffic between namespaces - A policy that only allows HTTPS traffic

Reference passage [1] says: 'In a multi-tenant environment where strict network isolation between tenants is required, starting with a default policy that denies communication between pods is recommended with another rule that allows all pods to query the DNS server for name resolution.'

## Failure Modes and Recovery Monitoring itself can fail, and policies can cause unintended outages. This section covers common failure modes and how to recover safely. ### Typical Failure Scenarios ### 1. Overly Restrictive Network Policy A new policy may block DNS or health checks, causing pods to fail readiness probes. Symptom: pods stuck in ContainerCreating or not ready. Check pod events: kubectl describe pod -n If events show connection refused to kube-dns or similar, the policy is likely blocking DNS. Quick recovery: delete or modify the policy to allow DNS (port 53) to kube-dns. Example fix: apply an egress rule for DNS: apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-dns namespace: myapp spec: podSelector: {} egress: - to: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: kube-system ports: - protocol: UDP port: 53 - protocol: TCP port: 53 policyTypes: - Egress ### 2. Metrics Endpoint Not Scraped If Prometheus cannot scrape Felix metrics, alerts won't fire. Check Prometheus targets for errors: https://prometheus.example.com/targets Common issues: service labels mismatch, network policy blocking Prometheus, or missing ServiceMonitor. Fix by aligning labels in ServiceMonitor and Service, and ensure Prometheus can reach the metrics port. ### 3. Log Noise from Denied Packets If you enable iptables logging for denied packets (via FelixConfiguration logSeverityScreen or similar), logs may flood. Set appropriate log levels and rotate logs. Monitor your log pipeline. ### Recovery Procedures ### Rolling Back a Network Policy Always back up policies before changing. Use kubectl get networkpolicy -n -o yaml > policy-backup.yaml . To rollback, apply the backup. ### Temporarily Disabling Enforcement If a policy is causing an outage, you can quickly delete it: kubectl delete networkpolicy -n But to be safer, you can annotate the policy to disable enforcement if supported by CNI. For Calico, you can set the policy order or use applyOnForward ? Not all CNIs support this. Deletion is often the quickest. ## Operations Checklist Use this checklist to ensure ongoing monitoring and quick response. ### Daily Checks - [ ] Prometheus target calico-felix is UP (or your CNI equivalent). - [ ] No critical alerts firing for policy drops or errors. - [ ] CNI pods are Running and not restarting frequently. Command: kubectl get pods -n kube-system | grep -E 'calico|cilium|antrea|weave' ### Weekly Checks - [ ] Review top denied traffic sources using PromQL and investigate if legitimate. - [ ] Audit NetworkPolicy changes: kubectl get events --all-namespaces | grep NetworkPolicy . - [ ] Check Prometheus storage and scrape duration to avoid performance issues. ### When a New NetworkPolicy is Applied - Before applying, simulate expected traffic with a test pod. - Apply policy in a staging namespace first if possible. - Monitor metrics for unexpected drops in the first hour. - Have a rollback plan: backup YAML. ### Alert Runbook: High Denied Traffic Alert definition example (Prometheus rule): groups: - name: network-policy rules: - alert: HighDeniedTraffic expr: sum(rate(felix_denied_packets_total[5m])) > 100 for: 5m labels: severity: warning annotations: summary: "High rate of denied packets" description: "Denied packets rate is {{ $value }} per second. Investigate potential misconfigurations." Runbook steps: - Check Prometheus to identify top namespaces and pods causing denies. - If legitimate traffic, adjust NetworkPolicy to allow it. - If malicious, investigate source and consider additional security measures. - If noise, adjust logging or alert threshold. ## Conclusion Kubernetes Network Policy monitoring is crucial for maintaining a secure and reliable cluster. By leveraging CNI metrics, Prometheus, and well-defined alerts, you can detect misconfigurations early, troubleshoot connectivity issues quickly, and ensure policies are enforced as intended. This guide provided concrete steps for environment inventory, safe configuration, verification, failure recovery, and operational checklists. Start with the basics: enable metrics, create a dashboard, and set up simple alerts. Then iterate to refine your monitoring as your policies evolve. Remember, the goal is not just to enforce policies but to understand their impact on your applications and respond effectively when things go wrong.