Intro
Kubernetes authorization and networking are deeply intertwined. Even with correct RBAC rules, a request can fail if the network path between components is broken. Common issues include DNS resolution failures, blocked ports, misconfigured routing, or firewall rules that silently drop traffic. This guide walks through a structured troubleshooting approach with practical command examples, expected outputs, and safe diagnosis steps. You will learn how to verify connectivity, pinpoint authorization-related network failures, and restore service without guesswork.
By the end, you will have a repeatable checklist to troubleshoot Kubernetes authorization networking issues in your own clusters. We cover version inventory, safe configuration, verification, failure modes, and recovery, with concrete commands and tables for quick reference.
Version and Environment Inventory
Before troubleshooting, establish the exact environment. Run these commands on a control plane node or via kubectl with cluster-admin access. Record the Kubernetes server version, networking plugin, DNS service IP, and API server endpoint.
# Kubernetes version
kubectl version --short
# Expected output (example):
# Client Version: v1.28.2
# Server Version: v1.28.2
# API server endpoint
kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}'
# Expected output:
# https://192.168.1.100:6443
# DNS service IP (usually the cluster IP of kube-dns or CoreDNS)
kubectl get svc -n kube-system -l k8s-app=kube-dns
# Expected output (example):
# NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
# kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP 10d
# Networking plugin pods (example for Calico)
kubectl get pods -n kube-system -l k8s-app=calico-node
# Expected output:
# NAME READY STATUS RESTARTS AGE
# calico-node-xxxxx 1/1 Running 0 10d
Topology assumptions: A standard cluster with one control plane node and multiple worker nodes, using CoreDNS for cluster DNS, and a CNI plugin such as Calico, Flannel, or Cilium. Firewalls may exist between nodes and between clients and the API server.
Prerequisites for troubleshooting:
- kubectl configured with appropriate permissions (at least read access to resources and ability to run exec in kube-system).
- Network utilities: curl, dig or nslookup, nc or telnet, traceroute, ip, iptables (on nodes).
- Access to node shells (via SSH or kubectl debug node).
Document the following in a table to keep track:
| Component | Version / Address | Notes |
|---|---|---|
| Kubernetes API server | https://192.168.1.100:6443 | Verify with kubectl cluster-info |
| CoreDNS service IP | 10.96.0.10 | ClusterIP of kube-dns service |
| CNI plugin | Calico v3.26.1 | Check with kubectl get pods -n kube-system |
| Node CIDR range | 10.244.0.0/16 | From CNI config |
Safe Configuration Path
When changes are needed, start with the smallest, reversible configuration update. For authorization networking, common adjustments include:
- Updating RBAC roles to allow DNS lookups (if using custom service accounts).
- Adjusting API server flags for authorization modes or network settings.
- Modifying CNI network policies that may block required ports.
- Adding firewall rules for node-to-node and pod-to-API-server traffic.
Example: Adding a missing RBAC permission for DNS resolution
A pod's service account may lack permission to create EndpointSlices or Services, causing DNS resolution to fail intermittently. To check current permissions:
kubectl auth can-i list endpointslices --as=system:serviceaccount:default:my-app
# Expected output if denied:
# no
If the service account needs this, create a Role and RoleBinding scoped to the necessary namespace:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: default
name: app-dns-reader
rules:
- apiGroups: ["discovery.k8s.io"]
resources: ["endpointslices"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
namespace: default
name: app-dns-reader-binding
subjects:
- kind: ServiceAccount
name: my-app
namespace: default
roleRef:
kind: Role
name: app-dns-reader
apiGroup: rbac.authorization.k8s.io
Apply with:
kubectl apply -f rbac-dns.yaml
# Expected output:
# role.rbac.authorization.k8s.io/app-dns-reader created
# rolebinding.rbac.authorization.k8s.io/app-dns-reader-binding created
Verify permission now:
kubectl auth can-i list endpointslices --as=system:serviceaccount:default:my-app
# Expected output:
# yes
If the issue is network policy blocking DNS traffic, define a policy scoped to the necessary pods:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-dns
namespace: default
spec:
podSelector:
matchLabels:
app: my-app
policyTypes:
- Egress
egress:
- to:
- namespaceSelector: {}
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
Apply and verify with kubectl get networkpolicy.
Safe practice: Always test changes in a non-production namespace or a test cluster first. Use kubectl diff -f <file> to preview changes before applying.
Verification and Diagnostics
This section provides step-by-step commands to verify each layer of the authorization networking stack. Start from the client side and move toward the API server.
1. API Server Connectivity
From a node or workstation, test TCP connectivity to the API server:
nc -zv 192.168.1.100 6443
# Expected output:
# Connection to 192.168.1.100 6443 port [tcp/*] succeeded!
If it fails, check firewall rules between the source and the API server. Also verify the API server is listening:
# On control plane node
ss -tlnp | grep 6443
# Expected output:
# LISTEN 0 4096 *:6443 *:* users:(("kube-apiserver",pid=1234,fd=7))
2. DNS Resolution
Check DNS resolution from a pod. Create a temporary pod if needed:
kubectl run -it --rm dnsutils --image=infoblox/dnstools:latest --restart=Never -- /bin/sh
# Inside the pod:
nslookup kubernetes.default.svc.cluster.local
# Expected output (example):
# Server: 10.96.0.10
# Address: 10.96.0.10#53
#
# Name: kubernetes.default.svc.cluster.local
# Address: 10.96.0.1
If the address is not the expected service IP or the lookup times out, investigate CoreDNS. Check CoreDNS pods and logs:
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=20
3. Service Routing
Verify that a Service routes to its endpoints. For a service named my-svc in namespace default:
kubectl get svc my-svc -n default -o wide
# Expected output:
# NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
# my-svc ClusterIP 10.96.100.10 <none> 80/TCP 1h app=my-app
kubectl get endpoints my-svc -n default
# Expected output:
# NAME ENDPOINTS AGE
# my-svc 10.244.1.2:8080 1h
If endpoints are empty, the service selector may not match any pods, or the pods are not ready. Check the pods:
kubectl get pods -n default -l app=my-app
4. Authorization Decision Verification
Use kubectl auth can-i to test specific permissions without making actual requests:
kubectl auth can-i create deployments --as=system:serviceaccount:default:my-app -n default
# Expected output:
# yes (or no)
For a deeper check, use kubectl auth reconcile to see differences, but be cautious as it can modify RBAC.
To see the actual RBAC rules for a user or service account, use kubectl get rolebindings and inspect:
kubectl get rolebinding -n default -o yaml | grep -A 10 subjects
5. Network Policy Verification
Check if any NetworkPolicies are blocking traffic. List policies in the namespace:
kubectl get networkpolicies -n default
For a specific pod, verify which policies apply:
kubectl describe networkpolicy -n default <policy-name>
Also check if the CNI plugin enforces network policies (e.g., Calico uses calicoctl for troubleshooting). For Calico:
calicoctl get networkpolicy -n default
6. Logs and Audit
Enable API server audit logging to capture authorization decisions. In the API server manifest, add audit policy and log backend. Then check logs for forbidden or unauthorized entries:
grep -i forbidden /var/log/kubernetes/audit.log | tail -5
If using kubeadm, audit logs may be in /var/log/kubernetes/audit/.
Failure Modes and Recovery
Common failure modes and their recovery steps:
1. Missing RBAC Permissions
Symptom: API calls return 403 Forbidden despite correct network connectivity.
Diagnosis:
kubectl auth can-i --list --as=system:serviceaccount:default:my-app -n default
Recovery: Apply the missing Role or ClusterRole with appropriate rules. Document changes in a version-controlled YAML file.
2. DNS Resolution Failure
Symptom: Pods cannot resolve internal service names (nslookup returns "server can't find ..." or times out).
Diagnosis: Check CoreDNS pod status and logs:
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50
# Look for errors like "plugin/errors: 2 ... SERVFAIL"
Also test DNS from a pod using the cluster IP directly:
kubectl run -it --rm dnsutils --image=infoblox/dnstools:latest -- nslookup kubernetes.default 10.96.0.10
Recovery: If CoreDNS is down, restart the deployment. If configuration is corrupt, restore from backup. Check CoreDNS ConfigMap for correctness:
kubectl get configmap coredns -n kube-system -o yaml
3. Network Policy Blocking
Symptom: Pod A cannot reach Pod B or a service, even though DNS resolves and endpoints exist.
Diagnosis: Describe network policies in the namespace and check for matching selectors that may unintentionally apply.
Recovery: Temporarily delete or edit the network policy to allow traffic, after confirming the policy is the cause. For example:
kubectl delete networkpolicy deny-all -n default
If deletion restores connectivity, recreate the policy with correct rules.
4. API Server Endpoint Misconfiguration
Symptom: kubectl commands hang or return connection refused from nodes or pods.
Diagnosis: Check the API server service and endpoints:
kubectl get svc kubernetes -n default
kubectl get endpoints kubernetes -n default
Recovery: Ensure the API server pod is running and the service selector matches. If the endpoint IP is wrong, restart kube-apiserver.
5. Firewall Dropping Traffic
Symptom: Ports open locally but connections time out between nodes.
Diagnosis: Use telnet or nc from one node to another on required ports (e.g., 6443 for API server, 10250 for kubelet, 8472 for Flannel VXLAN).
Recovery: Add firewall rules to allow the necessary ports based on your CNI and Kubernetes requirements. Document the changes in infrastructure-as-code.
Common Pitfalls and How to Avoid Them
Here are common mistakes people make when troubleshooting Kubernetes authorization networking, along with why they happen and how to avoid or recover from them.
A 401 Unauthorized error is often misread as an RBAC problem, but it actually means authentication failed (e.g., expired token, invalid certificate). Use kubectl auth can-i to test authorization only after confirming authentication works. Check the API server logs for authentication errors.
- Ignoring the distinction between authentication and authorization
Running kubectl auth can-i as the cluster admin always returns yes and hides the real issue. Always use --as or --as-group to impersonate the actual user or service account. For example, kubectl auth can-i list pods --as=system:serviceaccount:default:my-app.
- Testing from the wrong context
Many authorization checks involve service accounts that need to resolve service names. If CoreDNS is failing, authentication and authorization may appear broken because the API server cannot validate tokens or the client cannot reach it. Always test DNS early in the troubleshooting process.
- Overlooking DNS as a dependency
Some CNI plugins require explicit enabling of network policy enforcement. If policies are defined but not enforced, they will not cause issues; conversely, if you think policies are not enforced but they are, traffic may be blocked unexpectedly. Verify the CNI plugin's network policy support and check its logs.
- Assuming network policies are not enabled
In a panic, deleting network policies or modifying RBAC rules globally can cause more harm. Always test changes in a staging environment, use kubectl diff to preview, and back up current configurations before applying.
- Making broad, irreversible changes
Audit logs provide the exact reason for authorization decisions. Without them, you may guess wrong. Enable audit logging if not already on, and review logs for the specific request that failed.
- Not checking API server audit logs
The kubelet has its own authorization mode (e.g., AlwaysAllow, Webhook). If the API server uses RBAC but the kubelet uses AlwaysAllow, it may accept unauthorized requests. Check kubelet configuration with ps aux | grep kubelet or systemctl status kubelet for flags.
- Ignoring the kubelet's authorization mode
Pod-to-pod, pod-to-service, and external-to-service traffic may traverse different network paths (e.g., iptables, IPVS, eBPF). A connectivity issue may only affect one path. Test each path separately to isolate the problem.
- Assuming all traffic uses the same network path
Operations Checklist
Use this checklist when troubleshooting Kubernetes authorization networking. Each item includes a concrete command or action, and we recommend assigning an owner and a review frequency for recurring issues.
- [ ] Confirm Kubernetes and CNI versions (
kubectl version, CNI pod logs). Owner: platform engineer (e.g., Priya Shah). Review: before every troubleshooting session. - [ ] Verify API server reachability (
nc -zv <api-server-ip> 6443). Owner: network admin. Review: as needed. - [ ] Test DNS resolution from a pod (
nslookup kubernetes.default). Owner: developer on-call. Review: during pod connectivity issues. - [ ] Check CoreDNS pods and logs (
kubectl logs -n kube-system -l k8s-app=kube-dns). Owner: cluster operator. Review: when DNS fails. - [ ] Verify service endpoints are non-empty (
kubectl get endpoints <svc>). Owner: application owner. Review: when service routing is suspect. - [ ] Test authorization as the affected user (
kubectl auth can-i --as=...). Owner: security engineer. Review: when access is denied. - [ ] List and inspect NetworkPolicies in the namespace (
kubectl get networkpolicies). Owner: network admin. Review: when traffic is blocked. - [ ] Review API server audit logs for forbidden requests. Owner: security engineer. Review: monthly or after incidents.
- [ ] Check node firewall rules for required ports (6443, 10250, CNI ports). Owner: infrastructure engineer. Review: quarterly or after network changes.
- [ ] Document findings and any configuration changes made. Owner: incident commander. Review: after every incident.
Repeat these steps in a structured manner, moving from network connectivity to authorization decisions. Keep a log of commands executed and their outputs for post-incident review.
Conclusion
Kubernetes authorization networking issues can be complex, but a systematic approach helps isolate the root cause quickly. Start by inventorying your environment, then verify connectivity, DNS, service routing, and authorization decisions in order. Use safe, scoped configuration changes and always test in a non-production environment first. The diagnostics and recovery steps provided here give you concrete commands and expected outputs to resolve common problems. By following the operations checklist, avoiding common pitfalls, and assigning clear ownership, you can reduce downtime and prevent recurrence. Continue to build on this foundation by enabling audit logging and monitoring network policies to catch issues before they impact users.