## Intro

Kubernetes authorization and networking are deeply intertwined. Even with correct RBAC rules, a request can fail if the network path between components is broken. Common issues include DNS resolution failures, blocked ports, misconfigured routing, or firewall rules that silently drop traffic. This guide walks through a structured troubleshooting approach with practical command examples, expected outputs, and safe diagnosis steps. You will learn how to verify connectivity, pinpoint authorization-related network failures, and restore service without guesswork.

By the end, you will have a repeatable checklist to troubleshoot Kubernetes authorization networking issues in your own clusters. We cover version inventory, safe configuration, verification, failure modes, and recovery, with concrete commands and tables for quick reference.

## Version and Environment Inventory

Before troubleshooting, establish the exact environment. Run these commands on a control plane node or via kubectl with cluster-admin access. Record the Kubernetes server version, networking plugin, DNS service IP, and API server endpoint.

```bash
# Kubernetes version
kubectl version --short
# Expected output (example):
# Client Version: v1.28.2
# Server Version: v1.28.2

# API server endpoint
kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}'
# Expected output:
# https://192.168.1.100:6443

# DNS service IP (usually the cluster IP of kube-dns or CoreDNS)
kubectl get svc -n kube-system -l k8s-app=kube-dns
# Expected output (example):
# NAME          TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE
# kube-dns      ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP   10d

# Networking plugin pods (example for Calico)
kubectl get pods -n kube-system -l k8s-app=calico-node
# Expected output:
# NAME                READY   STATUS    RESTARTS   AGE
# calico-node-xxxxx   1/1     Running   0          10d
```

Topology assumptions: A standard cluster with one control plane node and multiple worker nodes, using CoreDNS for cluster DNS, and a CNI plugin such as Calico, Flannel, or Cilium. Firewalls may exist between nodes and between clients and the API server.

Prerequisites for troubleshooting:

- kubectl configured with appropriate permissions (at least read access to resources and ability to run exec in kube-system).
- Network utilities: curl, dig or nslookup, nc or telnet, traceroute, ip, iptables (on nodes).
- Access to node shells (via SSH or kubectl debug node).

Document the following in a table to keep track:

| Component | Version / Address | Notes |
|---|---|---|
| Kubernetes API server | https://192.168.1.100:6443 | Verify with `kubectl cluster-info` |
| CoreDNS service IP | 10.96.0.10 | ClusterIP of kube-dns service |
| CNI plugin | Calico v3.26.1 | Check with `kubectl get pods -n kube-system` |
| Node CIDR range | 10.244.0.0/16 | From CNI config |

## Safe Configuration Path

When changes are needed, start with the smallest, reversible configuration update. For authorization networking, common adjustments include:

- Updating RBAC roles to allow DNS lookups (if using custom service accounts).
- Adjusting API server flags for authorization modes or network settings.
- Modifying CNI network policies that may block required ports.
- Adding firewall rules for node-to-node and pod-to-API-server traffic.

Example: Adding a missing RBAC permission for DNS resolution

A pod's service account may lack permission to create EndpointSlices or Services, causing DNS resolution to fail intermittently. To check current permissions:

```bash
kubectl auth can-i list endpointslices --as=system:serviceaccount:default:my-app
# Expected output if denied:
# no
```

If the service account needs this, create a Role and RoleBinding scoped to the necessary namespace:

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: app-dns-reader
rules:
- apiGroups: ["discovery.k8s.io"]
  resources: ["endpointslices"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: default
  name: app-dns-reader-binding
subjects:
- kind: ServiceAccount
  name: my-app
  namespace: default
roleRef:
  kind: Role
  name: app-dns-reader
  apiGroup: rbac.authorization.k8s.io
```

Apply with:

```bash
kubectl apply -f rbac-dns.yaml
# Expected output:
# role.rbac.authorization.k8s.io/app-dns-reader created
# rolebinding.rbac.authorization.k8s.io/app-dns-reader-binding created
```

Verify permission now:

```bash
kubectl auth can-i list endpointslices --as=system:serviceaccount:default:my-app
# Expected output:
# yes
```

If the issue is network policy blocking DNS traffic, define a policy scoped to the necessary pods:

```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: default
spec:
  podSelector:
    matchLabels:
      app: my-app
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector: {}
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53
```

Apply and verify with `kubectl get networkpolicy`.

Safe practice: Always test changes in a non-production namespace or a test cluster first. Use `kubectl diff -f <file>` to preview changes before applying.

## Verification and Diagnostics

This section provides step-by-step commands to verify each layer of the authorization networking stack. Start from the client side and move toward the API server.

### 1. API Server Connectivity

From a node or workstation, test TCP connectivity to the API server:

```bash
nc -zv 192.168.1.100 6443
# Expected output:
# Connection to 192.168.1.100 6443 port [tcp/*] succeeded!
```

If it fails, check firewall rules between the source and the API server. Also verify the API server is listening:

```bash
# On control plane node
ss -tlnp | grep 6443
# Expected output:
# LISTEN 0 4096 *:6443 *:* users:(("kube-apiserver",pid=1234,fd=7))
```

### 2. DNS Resolution

Check DNS resolution from a pod. Create a temporary pod if needed:

```bash
kubectl run -it --rm dnsutils --image=infoblox/dnstools:latest --restart=Never -- /bin/sh
# Inside the pod:
nslookup kubernetes.default.svc.cluster.local
# Expected output (example):
# Server: 10.96.0.10
# Address: 10.96.0.10#53
#
# Name: kubernetes.default.svc.cluster.local
# Address: 10.96.0.1
```

If the address is not the expected service IP or the lookup times out, investigate CoreDNS. Check CoreDNS pods and logs:

```bash
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=20
```

### 3. Service Routing

Verify that a Service routes to its endpoints. For a service named my-svc in namespace default:

```bash
kubectl get svc my-svc -n default -o wide
# Expected output:
# NAME      TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE   SELECTOR
# my-svc    ClusterIP   10.96.100.10   <none>        80/TCP    1h    app=my-app

kubectl get endpoints my-svc -n default
# Expected output:
# NAME      ENDPOINTS         AGE
# my-svc    10.244.1.2:8080   1h
```

If endpoints are empty, the service selector may not match any pods, or the pods are not ready. Check the pods:

```bash
kubectl get pods -n default -l app=my-app
```

### 4. Authorization Decision Verification

Use `kubectl auth can-i` to test specific permissions without making actual requests:

```bash
kubectl auth can-i create deployments --as=system:serviceaccount:default:my-app -n default
# Expected output:
# yes (or no)
```

For a deeper check, use `kubectl auth reconcile` to see differences, but be cautious as it can modify RBAC.

To see the actual RBAC rules for a user or service account, use `kubectl get rolebindings` and inspect:

```bash
kubectl get rolebinding -n default -o yaml | grep -A 10 subjects
```

### 5. Network Policy Verification

Check if any NetworkPolicies are blocking traffic. List policies in the namespace:

```bash
kubectl get networkpolicies -n default
```

For a specific pod, verify which policies apply:

```bash
kubectl describe networkpolicy -n default <policy-name>
```

Also check if the CNI plugin enforces network policies (e.g., Calico uses calicoctl for troubleshooting). For Calico:

```bash
calicoctl get networkpolicy -n default
```

### 6. Logs and Audit

Enable API server audit logging to capture authorization decisions. In the API server manifest, add audit policy and log backend. Then check logs for forbidden or unauthorized entries:

```bash
grep -i forbidden /var/log/kubernetes/audit.log | tail -5
```

If using kubeadm, audit logs may be in /var/log/kubernetes/audit/.

## Failure Modes and Recovery

Common failure modes and their recovery steps:

### 1. Missing RBAC Permissions

Symptom: API calls return 403 Forbidden despite correct network connectivity.

Diagnosis:

```bash
kubectl auth can-i --list --as=system:serviceaccount:default:my-app -n default
```

Recovery: Apply the missing Role or ClusterRole with appropriate rules. Document changes in a version-controlled YAML file.

### 2. DNS Resolution Failure

Symptom: Pods cannot resolve internal service names (`nslookup` returns "server can't find ..." or times out).

Diagnosis: Check CoreDNS pod status and logs:

```bash
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50
# Look for errors like "plugin/errors: 2 ... SERVFAIL"
```

Also test DNS from a pod using the cluster IP directly:

```bash
kubectl run -it --rm dnsutils --image=infoblox/dnstools:latest -- nslookup kubernetes.default 10.96.0.10
```

Recovery: If CoreDNS is down, restart the deployment. If configuration is corrupt, restore from backup. Check CoreDNS ConfigMap for correctness:

```bash
kubectl get configmap coredns -n kube-system -o yaml
```

### 3. Network Policy Blocking

Symptom: Pod A cannot reach Pod B or a service, even though DNS resolves and endpoints exist.

Diagnosis: Describe network policies in the namespace and check for matching selectors that may unintentionally apply.

Recovery: Temporarily delete or edit the network policy to allow traffic, after confirming the policy is the cause. For example:

```bash
kubectl delete networkpolicy deny-all -n default
```

If deletion restores connectivity, recreate the policy with correct rules.

### 4. API Server Endpoint Misconfiguration

Symptom: kubectl commands hang or return connection refused from nodes or pods.

Diagnosis: Check the API server service and endpoints:

```bash
kubectl get svc kubernetes -n default
kubectl get endpoints kubernetes -n default
```

Recovery: Ensure the API server pod is running and the service selector matches. If the endpoint IP is wrong, restart kube-apiserver.

### 5. Firewall Dropping Traffic

Symptom: Ports open locally but connections time out between nodes.

Diagnosis: Use telnet or nc from one node to another on required ports (e.g., 6443 for API server, 10250 for kubelet, 8472 for Flannel VXLAN).

Recovery: Add firewall rules to allow the necessary ports based on your CNI and Kubernetes requirements. Document the changes in infrastructure-as-code.

## Common Pitfalls and How to Avoid Them

Here are common mistakes people make when troubleshooting Kubernetes authorization networking, along with why they happen and how to avoid or recover from them.

1. **Ignoring the distinction between authentication and authorization**  
   A 401 Unauthorized error is often misread as an RBAC problem, but it actually means authentication failed (e.g., expired token, invalid certificate). Use `kubectl auth can-i` to test authorization only after confirming authentication works. Check the API server logs for authentication errors.

2. **Testing from the wrong context**  
   Running `kubectl auth can-i` as the cluster admin always returns yes and hides the real issue. Always use `--as` or `--as-group` to impersonate the actual user or service account. For example, `kubectl auth can-i list pods --as=system:serviceaccount:default:my-app`.

3. **Overlooking DNS as a dependency**  
   Many authorization checks involve service accounts that need to resolve service names. If CoreDNS is failing, authentication and authorization may appear broken because the API server cannot validate tokens or the client cannot reach it. Always test DNS early in the troubleshooting process.

4. **Assuming network policies are not enabled**  
   Some CNI plugins require explicit enabling of network policy enforcement. If policies are defined but not enforced, they will not cause issues; conversely, if you think policies are not enforced but they are, traffic may be blocked unexpectedly. Verify the CNI plugin's network policy support and check its logs.

5. **Making broad, irreversible changes**  
   In a panic, deleting network policies or modifying RBAC rules globally can cause more harm. Always test changes in a staging environment, use `kubectl diff` to preview, and back up current configurations before applying.

6. **Not checking API server audit logs**  
   Audit logs provide the exact reason for authorization decisions. Without them, you may guess wrong. Enable audit logging if not already on, and review logs for the specific request that failed.

7. **Ignoring the kubelet's authorization mode**  
   The kubelet has its own authorization mode (e.g., AlwaysAllow, Webhook). If the API server uses RBAC but the kubelet uses AlwaysAllow, it may accept unauthorized requests. Check kubelet configuration with `ps aux | grep kubelet` or `systemctl status kubelet` for flags.

8. **Assuming all traffic uses the same network path**  
   Pod-to-pod, pod-to-service, and external-to-service traffic may traverse different network paths (e.g., iptables, IPVS, eBPF). A connectivity issue may only affect one path. Test each path separately to isolate the problem.

## Operations Checklist

Use this checklist when troubleshooting Kubernetes authorization networking. Each item includes a concrete command or action, and we recommend assigning an owner and a review frequency for recurring issues.

- [ ] Confirm Kubernetes and CNI versions (`kubectl version`, CNI pod logs). Owner: platform engineer (e.g., Priya Shah). Review: before every troubleshooting session.
- [ ] Verify API server reachability (`nc -zv <api-server-ip> 6443`). Owner: network admin. Review: as needed.
- [ ] Test DNS resolution from a pod (`nslookup kubernetes.default`). Owner: developer on-call. Review: during pod connectivity issues.
- [ ] Check CoreDNS pods and logs (`kubectl logs -n kube-system -l k8s-app=kube-dns`). Owner: cluster operator. Review: when DNS fails.
- [ ] Verify service endpoints are non-empty (`kubectl get endpoints <svc>`). Owner: application owner. Review: when service routing is suspect.
- [ ] Test authorization as the affected user (`kubectl auth can-i --as=...`). Owner: security engineer. Review: when access is denied.
- [ ] List and inspect NetworkPolicies in the namespace (`kubectl get networkpolicies`). Owner: network admin. Review: when traffic is blocked.
- [ ] Review API server audit logs for forbidden requests. Owner: security engineer. Review: monthly or after incidents.
- [ ] Check node firewall rules for required ports (6443, 10250, CNI ports). Owner: infrastructure engineer. Review: quarterly or after network changes.
- [ ] Document findings and any configuration changes made. Owner: incident commander. Review: after every incident.

Repeat these steps in a structured manner, moving from network connectivity to authorization decisions. Keep a log of commands executed and their outputs for post-incident review.

## Conclusion

Kubernetes authorization networking issues can be complex, but a systematic approach helps isolate the root cause quickly. Start by inventorying your environment, then verify connectivity, DNS, service routing, and authorization decisions in order. Use safe, scoped configuration changes and always test in a non-production environment first. The diagnostics and recovery steps provided here give you concrete commands and expected outputs to resolve common problems. By following the operations checklist, avoiding common pitfalls, and assigning clear ownership, you can reduce downtime and prevent recurrence. Continue to build on this foundation by enabling audit logging and monitoring network policies to catch issues before they impact users.