Kubernetes Network Policy Monitoring and Alerts: A Practical Implementation Guide

Intro

Kubernetes Network Policies are essential for controlling pod-to-pod traffic and enforcing zero-trust networking inside a cluster. However, defining policies is only half the battle. Without effective monitoring and alerting, you cannot know whether policies are actually enforced, misconfigured, or silently dropping legitimate traffic. This guide provides a practical, step-by-step approach to monitoring Kubernetes Network Policies using native tools, Prometheus, and alerting systems. You will learn how to collect metrics, build dashboards, set up alerts, and respond to incidents with real examples and commands.

We will focus on common scenarios such as detecting denied traffic, auditing policy changes, and troubleshooting connectivity issues caused by overly restrictive rules. The article is aimed at platform engineers, SREs, and DevOps teams running Kubernetes in production.

Before making any changes, always establish observability. The goal is to make network policy behavior visible, enable rapid diagnosis, and ensure that any configuration change is safe and reversible.

Version and Environment Inventory

Before implementing monitoring, document your environment. This inventory will help you choose the right tools and understand compatibility.

Cluster and CNI Details

Network Policy enforcement depends on the CNI plugin. Common CNIs that support Network Policies include Calico, Cilium, Weave Net, and Antrea. Verify your CNI and its version:

kubectl get nodes -o wide
# Check CNI pods
kubectl get pods -n kube-system | grep -E 'calico|cilium|weave|antrea'

Example output (Calico):

calico-node-abcde                    1/1     Running   0          5d
calico-kube-controllers-67890        1/1     Running   0          5d

Check the CNI configuration in the node if needed, but the pod image tag often reveals the version:

kubectl get ds -n kube-system calico-node -o jsonpath='{.spec.template.spec.containers[0].image}'

Kubernetes Version and API Availability

NetworkPolicy is a stable API (networking.k8s.io/v1) since Kubernetes 1.7. Ensure your cluster is recent enough:

kubectl version --short

Example:

Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.3

Existing Network Policies

List all Network Policies across namespaces to understand your current state:

kubectl get networkpolicies -A

Example output:

NAMESPACE     NAME                   POD-SELECTOR       AGE
default       deny-all               <none>             10d
kube-system   allow-dns              app=kube-dns       10d
app-ns        allow-frontend-tiers   tier=frontend      2d

Note any default-deny policies and their selectors. This is your baseline.

Monitoring Tools Available

Check if Prometheus is installed:

kubectl get pods -n monitoring | grep prometheus

If not, you may need to install Prometheus and configure it to scrape metrics from your CNI or use a ServiceMonitor.

Safe Configuration Path

Monitoring Network Policies should be done incrementally to avoid disrupting production traffic. Follow a safe path: observe, enable metrics, test, and then alert.

Step 1: Enable Metrics Collection from CNI

Most CNIs expose Prometheus metrics. For Calico, the Felix agent (running on each node) provides detailed policy metrics. Enable metrics reporting by applying the following configuration if not already enabled.

For Calico, you can use the built-in FelixConfiguration:

apiVersion: crd.projectcalico.org/v1
kind: FelixConfiguration
metadata:
  name: default
spec:
  prometheusMetricsEnabled: true
  prometheusMetricsPort: 9091

Apply with kubectl apply -f felix-config.yaml. Verify by checking a node's Felix port:

kubectl get felixconfigurations default -o yaml | grep prometheus

For Cilium, metrics are enabled by default, but verify:

kubectl -n kube-system exec ds/cilium -- cilium status | grep Metrics

Step 2: Create a ServiceMonitor for Prometheus

If you use the Prometheus Operator, create a ServiceMonitor to scrape Felix metrics:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: calico-felix
  namespace: monitoring
spec:
  selector:
    matchLabels:
      k8s-app: calico-node
  endpoints:
  - port: metrics
    interval: 30s

Ensure the calico-node service exposes the metrics port. If not, create a service:

apiVersion: v1
kind: Service
metadata:
  name: calico-node-metrics
  namespace: kube-system
  labels:
    k8s-app: calico-node
spec:
  selector:
    k8s-app: calico-node
  ports:
  - name: metrics
    port: 9091
    targetPort: 9091

Apply and verify Prometheus targets:

kubectl -n monitoring port-forward svc/prometheus-operated 9090:9090
# Then open http://localhost:9090/targets and confirm calico-felix is up.

Step 3: Define Minimal Test Traffic

Before relying on alerts, generate known traffic that should be allowed or denied by a test policy. Create a test namespace and pods:

kubectl create ns policy-test
kubectl run allowed-client --image=busybox --restart=Never -- sleep 3600
kubectl run denied-client --image=busybox --restart=Never -- sleep 3600
kubectl run server --image=nginx --restart=Never

Apply a NetworkPolicy that allows only allowed-client to access server:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: test-allow-specific
  namespace: policy-test
spec:
  podSelector:
    matchLabels:
      run: server
  ingress:
  - from:
    - podSelector:
        matchLabels:
          run: allowed-client
    ports:
    - port: 80

Now test connectivity:

kubectl exec -it allowed-client -- wget -qO- --timeout=2 http://server
# Should return HTML
kubectl exec -it denied-client -- wget -qO- --timeout=2 http://server
# Should fail or time out

This baseline helps validate that alerts correspond to actual traffic.

Verification and Diagnostics

Once metrics are scraped and test traffic is flowing, you need to verify that policy enforcement is visible and diagnose any issues.

Key Metrics to Monitor

Different CNIs expose different metric names. For Calico Felix, important metrics include:

• felix_iptables_restore_errors: errors applying iptables rules.
• felix_denied_packets_total: count of packets denied by policy (if iptables action is set to LOG).
• felix_ipset_errors: errors manipulating IP sets.
• felix_active_local_endpoints: number of local endpoints.

For Cilium:

• cilium_drop_count_total: dropped packets due to policy.
• cilium_policy_endpoint_enforcement_status: enforcement status per endpoint.

Query examples in Prometheus:

# Total denied packets in last 5 minutes
sum(increase(felix_denied_packets_total[5m]))

# Rate of drops in Cilium
sum(rate(cilium_drop_count_total[5m])) by (reason)

PromQL for Policy Auditing

To see which pods are generating drops, use labels. Calico metrics often include labels like source_namespace, source_pod, dest_namespace, dest_pod. Example:

topk(10, sum by (source_namespace, source_pod) (rate(felix_denied_packets_total[5m])))

This gives you the top sources of denied traffic.

Using kubectl to Inspect Policy Status

While CNI metrics show enforcement, you can also inspect policy objects and endpoints:

kubectl describe networkpolicy test-allow-specific -n policy-test

Output includes the pod selector and ingress rules. To see if a policy is applied to a pod, check the pod's annotations or use CNI CLI tools:

For Calico:

kubectl exec -n kube-system calicoctl -- calicoctl get profile -A

For Cilium, use cilium endpoint list to see policy enforcement status:

kubectl -n kube-system exec ds/cilium -- cilium endpoint list

Look for Policy enforcement column: Ingress, Egress, or Both.

Diagnosing Connectivity Issues

If traffic is unexpectedly blocked, follow this sequence:

1. Check if NetworkPolicy exists and selects the pod:

   kubectl get networkpolicy -n <namespace>

1. Describe the policy and verify rules:

   kubectl describe networkpolicy <name> -n <namespace>

For Calico Felix:

1. Test with a known client pod (as earlier).
2. Check metrics for drops: query Prometheus for denied packets in that namespace.
3. Check CNI logs:

   kubectl logs -n kube-system ds/calico-node -c calico-node --tail=50

For Cilium:

   kubectl -n kube-system logs ds/cilium --tail=50

Failure Modes and Recovery

Monitoring itself can fail, and policies can cause unintended outages. This section covers common failure modes and how to recover safely.

Typical Failure Scenarios

1. Overly Restrictive Network Policy

A new policy may block DNS or health checks, causing pods to fail readiness probes. Symptom: pods stuck in ContainerCreating or not ready.

Check pod events:

kubectl describe pod <pod-name> -n <namespace>

If events show connection refused to kube-dns or similar, the policy is likely blocking DNS. Quick recovery: delete or modify the policy to allow DNS (port 53) to kube-dns.

Example fix: apply an egress rule for DNS:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: myapp
spec:
  podSelector: {}
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53
  policyTypes:
  - Egress

2. Metrics Endpoint Not Scraped

If Prometheus cannot scrape Felix metrics, alerts won't fire. Check Prometheus targets for errors:

https://prometheus.example.com/targets

Common issues: service labels mismatch, network policy blocking Prometheus, or missing ServiceMonitor. Fix by aligning labels in ServiceMonitor and Service, and ensure Prometheus can reach the metrics port.

3. Log Noise from Denied Packets

If you enable iptables logging for denied packets (via FelixConfiguration logSeverityScreen or similar), logs may flood. Set appropriate log levels and rotate logs. Monitor your log pipeline.

Recovery Procedures

Rolling Back a Network Policy

Always back up policies before changing. Use kubectl get networkpolicy <name> -n <ns> -o yaml > policy-backup.yaml. To rollback, apply the backup.

Temporarily Disabling Enforcement

If a policy is causing an outage, you can quickly delete it:

kubectl delete networkpolicy <name> -n <namespace>

But to be safer, you can annotate the policy to disable enforcement if supported by CNI. For Calico, you can set the policy order or use applyOnForward? Not all CNIs support this. Deletion is often the quickest.

Operations Checklist

Use this checklist to ensure ongoing monitoring and quick response.

Daily Checks

• [ ] Prometheus target calico-felix is UP (or your CNI equivalent).
• [ ] No critical alerts firing for policy drops or errors.
• [ ] CNI pods are Running and not restarting frequently.

Command:

kubectl get pods -n kube-system | grep -E 'calico|cilium|antrea|weave'

Weekly Checks

• [ ] Review top denied traffic sources using PromQL and investigate if legitimate.
• [ ] Audit NetworkPolicy changes: kubectl get events --all-namespaces | grep NetworkPolicy.
• [ ] Check Prometheus storage and scrape duration to avoid performance issues.

When a New NetworkPolicy is Applied

1. Before applying, simulate expected traffic with a test pod.
2. Apply policy in a staging namespace first if possible.
3. Monitor metrics for unexpected drops in the first hour.
4. Have a rollback plan: backup YAML.

Alert Runbook: High Denied Traffic

Alert definition example (Prometheus rule):

groups:
- name: network-policy
  rules:
  - alert: HighDeniedTraffic
    expr: sum(rate(felix_denied_packets_total[5m])) > 100
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "High rate of denied packets"
      description: "Denied packets rate is {{ $value }} per second. Investigate potential misconfigurations."

Runbook steps:

1. Check Prometheus to identify top namespaces and pods causing denies.
2. If legitimate traffic, adjust NetworkPolicy to allow it.
3. If malicious, investigate source and consider additional security measures.
4. If noise, adjust logging or alert threshold.

Conclusion

Kubernetes Network Policy monitoring is crucial for maintaining a secure and reliable cluster. By leveraging CNI metrics, Prometheus, and well-defined alerts, you can detect misconfigurations early, troubleshoot connectivity issues quickly, and ensure policies are enforced as intended. This guide provided concrete steps for environment inventory, safe configuration, verification, failure recovery, and operational checklists. Start with the basics: enable metrics, create a dashboard, and set up simple alerts. Then iterate to refine your monitoring as your policies evolve. Remember, the goal is not just to enforce policies but to understand their impact on your applications and respond effectively when things go wrong.