E-NO
Kubernetes Authorization networking 9 Min Read

Kubernetes Authorization and Networking Troubleshooting: A Practical Guide with Commands

calendar_today Published: 2026-11-18
update Last Updated: 2026-11-18
analytics SEO Efficiency: 100%
Technical guide illustration for Kubernetes Authorization and Networking Troubleshooting: A Practical Guide with Commands.

Intro

Kubernetes authorization and networking are deeply intertwined. Even with correct RBAC rules, a request can fail if the network path between components is broken. Common issues include DNS resolution failures, blocked ports, misconfigured routing, or firewall rules that silently drop traffic. This guide walks through a structured troubleshooting approach with practical command examples, expected outputs, and safe diagnosis steps. You will learn how to verify connectivity, pinpoint authorization-related network failures, and restore service without guesswork.

By the end, you will have a repeatable checklist to troubleshoot Kubernetes authorization networking issues in your own clusters. We cover version inventory, safe configuration, verification, failure modes, and recovery, with concrete commands and tables for quick reference.

Version and Environment Inventory

Before troubleshooting, establish the exact environment. Run these commands on a control plane node or via kubectl with cluster-admin access. Record the Kubernetes server version, networking plugin, DNS service IP, and API server endpoint.

# Kubernetes version
kubectl version --short
# Expected output (example):
# Client Version: v1.28.2
# Server Version: v1.28.2

# API server endpoint
kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}'
# Expected output:
# https://192.168.1.100:6443

# DNS service IP (usually the cluster IP of kube-dns or CoreDNS)
kubectl get svc -n kube-system -l k8s-app=kube-dns
# Expected output (example):
# NAME          TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE
# kube-dns      ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP   10d

# Networking plugin pods (example for Calico)
kubectl get pods -n kube-system -l k8s-app=calico-node
# Expected output:
# NAME                READY   STATUS    RESTARTS   AGE
# calico-node-xxxxx   1/1     Running   0          10d

Topology assumptions: A standard cluster with one control plane node and multiple worker nodes, using CoreDNS for cluster DNS, and a CNI plugin such as Calico, Flannel, or Cilium. Firewalls may exist between nodes and between clients and the API server.

Prerequisites for troubleshooting:

  • kubectl configured with appropriate permissions (at least read access to resources and ability to run exec in kube-system).
  • Network utilities: curl, dig or nslookup, nc or telnet, traceroute, ip, iptables (on nodes).
  • Access to node shells (via SSH or kubectl debug node).

Document the following in a table to keep track:

ComponentVersion / AddressNotes
Kubernetes API serverhttps://192.168.1.100:6443Verify with kubectl cluster-info
CoreDNS service IP10.96.0.10ClusterIP of kube-dns service
CNI pluginCalico v3.26.1Check with kubectl get pods -n kube-system
Node CIDR range10.244.0.0/16From CNI config

Quick check 1 of 2

When multiple authorization modules are configured in the API server, how does Kubernetes handle a request if one module authorizes it and another module denies it (as described in the ABAC/RBAC/Webhook description)?

The passage states that if more than one authorization module is configured, Kubernetes checks each module, and if any module authorizes the request, then the request can proceed.

Safe Configuration Path

When changes are needed, start with the smallest, reversible configuration update. For authorization networking, common adjustments include:

  • Updating RBAC roles to allow DNS lookups (if using custom service accounts).
  • Adjusting API server flags for authorization modes or network settings.
  • Modifying CNI network policies that may block required ports.
  • Adding firewall rules for node-to-node and pod-to-API-server traffic.

Example: Adding a missing RBAC permission for DNS resolution

A pod's service account may lack permission to create EndpointSlices or Services, causing DNS resolution to fail intermittently. To check current permissions:

kubectl auth can-i list endpointslices --as=system:serviceaccount:default:my-app
# Expected output if denied:
# no

If the service account needs this, create a Role and RoleBinding scoped to the necessary namespace:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: app-dns-reader
rules:
- apiGroups: ["discovery.k8s.io"]
  resources: ["endpointslices"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: default
  name: app-dns-reader-binding
subjects:
- kind: ServiceAccount
  name: my-app
  namespace: default
roleRef:
  kind: Role
  name: app-dns-reader
  apiGroup: rbac.authorization.k8s.io

Apply with:

kubectl apply -f rbac-dns.yaml
# Expected output:
# role.rbac.authorization.k8s.io/app-dns-reader created
# rolebinding.rbac.authorization.k8s.io/app-dns-reader-binding created

Verify permission now:

kubectl auth can-i list endpointslices --as=system:serviceaccount:default:my-app
# Expected output:
# yes

If the issue is network policy blocking DNS traffic, define a policy scoped to the necessary pods:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: default
spec:
  podSelector:
    matchLabels:
      app: my-app
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector: {}
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53

Apply and verify with kubectl get networkpolicy.

Safe practice: Always test changes in a non-production namespace or a test cluster first. Use kubectl diff -f <file> to preview changes before applying.

Verification and Diagnostics

This section provides step-by-step commands to verify each layer of the authorization networking stack. Start from the client side and move toward the API server.

1. API Server Connectivity

From a node or workstation, test TCP connectivity to the API server:

nc -zv 192.168.1.100 6443
# Expected output:
# Connection to 192.168.1.100 6443 port [tcp/*] succeeded!

If it fails, check firewall rules between the source and the API server. Also verify the API server is listening:

# On control plane node
ss -tlnp | grep 6443
# Expected output:
# LISTEN 0 4096 *:6443 *:* users:(("kube-apiserver",pid=1234,fd=7))

2. DNS Resolution

Check DNS resolution from a pod. Create a temporary pod if needed:

kubectl run -it --rm dnsutils --image=infoblox/dnstools:latest --restart=Never -- /bin/sh
# Inside the pod:
nslookup kubernetes.default.svc.cluster.local
# Expected output (example):
# Server: 10.96.0.10
# Address: 10.96.0.10#53
#
# Name: kubernetes.default.svc.cluster.local
# Address: 10.96.0.1

If the address is not the expected service IP or the lookup times out, investigate CoreDNS. Check CoreDNS pods and logs:

kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=20

3. Service Routing

Verify that a Service routes to its endpoints. For a service named my-svc in namespace default:

kubectl get svc my-svc -n default -o wide
# Expected output:
# NAME      TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE   SELECTOR
# my-svc    ClusterIP   10.96.100.10   <none>        80/TCP    1h    app=my-app

kubectl get endpoints my-svc -n default
# Expected output:
# NAME      ENDPOINTS         AGE
# my-svc    10.244.1.2:8080   1h

If endpoints are empty, the service selector may not match any pods, or the pods are not ready. Check the pods:

kubectl get pods -n default -l app=my-app

4. Authorization Decision Verification

Use kubectl auth can-i to test specific permissions without making actual requests:

kubectl auth can-i create deployments --as=system:serviceaccount:default:my-app -n default
# Expected output:
# yes (or no)

For a deeper check, use kubectl auth reconcile to see differences, but be cautious as it can modify RBAC.

To see the actual RBAC rules for a user or service account, use kubectl get rolebindings and inspect:

kubectl get rolebinding -n default -o yaml | grep -A 10 subjects

5. Network Policy Verification

Check if any NetworkPolicies are blocking traffic. List policies in the namespace:

kubectl get networkpolicies -n default

For a specific pod, verify which policies apply:

kubectl describe networkpolicy -n default <policy-name>

Also check if the CNI plugin enforces network policies (e.g., Calico uses calicoctl for troubleshooting). For Calico:

calicoctl get networkpolicy -n default

6. Logs and Audit

Enable API server audit logging to capture authorization decisions. In the API server manifest, add audit policy and log backend. Then check logs for forbidden or unauthorized entries:

grep -i forbidden /var/log/kubernetes/audit.log | tail -5

If using kubeadm, audit logs may be in /var/log/kubernetes/audit/.

Failure Modes and Recovery

Common failure modes and their recovery steps:

1. Missing RBAC Permissions

Symptom: API calls return 403 Forbidden despite correct network connectivity.

Diagnosis:

kubectl auth can-i --list --as=system:serviceaccount:default:my-app -n default

Recovery: Apply the missing Role or ClusterRole with appropriate rules. Document changes in a version-controlled YAML file.

2. DNS Resolution Failure

Symptom: Pods cannot resolve internal service names (nslookup returns "server can't find ..." or times out).

Diagnosis: Check CoreDNS pod status and logs:

kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50
# Look for errors like "plugin/errors: 2 ... SERVFAIL"

Also test DNS from a pod using the cluster IP directly:

kubectl run -it --rm dnsutils --image=infoblox/dnstools:latest -- nslookup kubernetes.default 10.96.0.10

Recovery: If CoreDNS is down, restart the deployment. If configuration is corrupt, restore from backup. Check CoreDNS ConfigMap for correctness:

kubectl get configmap coredns -n kube-system -o yaml

3. Network Policy Blocking

Symptom: Pod A cannot reach Pod B or a service, even though DNS resolves and endpoints exist.

Diagnosis: Describe network policies in the namespace and check for matching selectors that may unintentionally apply.

Recovery: Temporarily delete or edit the network policy to allow traffic, after confirming the policy is the cause. For example:

kubectl delete networkpolicy deny-all -n default

If deletion restores connectivity, recreate the policy with correct rules.

4. API Server Endpoint Misconfiguration

Symptom: kubectl commands hang or return connection refused from nodes or pods.

Diagnosis: Check the API server service and endpoints:

kubectl get svc kubernetes -n default
kubectl get endpoints kubernetes -n default

Recovery: Ensure the API server pod is running and the service selector matches. If the endpoint IP is wrong, restart kube-apiserver.

5. Firewall Dropping Traffic

Symptom: Ports open locally but connections time out between nodes.

Diagnosis: Use telnet or nc from one node to another on required ports (e.g., 6443 for API server, 10250 for kubelet, 8472 for Flannel VXLAN).

Recovery: Add firewall rules to allow the necessary ports based on your CNI and Kubernetes requirements. Document the changes in infrastructure-as-code.

Quick check 2 of 2

According to the passage on authorization verdicts, what HTTP status code does the API server return when the overall authorization verdict is deny?

The passage states that an overall deny verdict means the API server rejects the request and responds with an HTTP 403 (Forbidden) status.

Common Pitfalls and How to Avoid Them

Here are common mistakes people make when troubleshooting Kubernetes authorization networking, along with why they happen and how to avoid or recover from them.

A 401 Unauthorized error is often misread as an RBAC problem, but it actually means authentication failed (e.g., expired token, invalid certificate). Use kubectl auth can-i to test authorization only after confirming authentication works. Check the API server logs for authentication errors.

  1. Ignoring the distinction between authentication and authorization

Running kubectl auth can-i as the cluster admin always returns yes and hides the real issue. Always use --as or --as-group to impersonate the actual user or service account. For example, kubectl auth can-i list pods --as=system:serviceaccount:default:my-app.

  1. Testing from the wrong context

Many authorization checks involve service accounts that need to resolve service names. If CoreDNS is failing, authentication and authorization may appear broken because the API server cannot validate tokens or the client cannot reach it. Always test DNS early in the troubleshooting process.

  1. Overlooking DNS as a dependency

Some CNI plugins require explicit enabling of network policy enforcement. If policies are defined but not enforced, they will not cause issues; conversely, if you think policies are not enforced but they are, traffic may be blocked unexpectedly. Verify the CNI plugin's network policy support and check its logs.

  1. Assuming network policies are not enabled

In a panic, deleting network policies or modifying RBAC rules globally can cause more harm. Always test changes in a staging environment, use kubectl diff to preview, and back up current configurations before applying.

  1. Making broad, irreversible changes

Audit logs provide the exact reason for authorization decisions. Without them, you may guess wrong. Enable audit logging if not already on, and review logs for the specific request that failed.

  1. Not checking API server audit logs

The kubelet has its own authorization mode (e.g., AlwaysAllow, Webhook). If the API server uses RBAC but the kubelet uses AlwaysAllow, it may accept unauthorized requests. Check kubelet configuration with ps aux | grep kubelet or systemctl status kubelet for flags.

  1. Ignoring the kubelet's authorization mode

Pod-to-pod, pod-to-service, and external-to-service traffic may traverse different network paths (e.g., iptables, IPVS, eBPF). A connectivity issue may only affect one path. Test each path separately to isolate the problem.

  1. Assuming all traffic uses the same network path

Operations Checklist

Use this checklist when troubleshooting Kubernetes authorization networking. Each item includes a concrete command or action, and we recommend assigning an owner and a review frequency for recurring issues.

  • [ ] Confirm Kubernetes and CNI versions (kubectl version, CNI pod logs). Owner: platform engineer (e.g., Priya Shah). Review: before every troubleshooting session.
  • [ ] Verify API server reachability (nc -zv <api-server-ip> 6443). Owner: network admin. Review: as needed.
  • [ ] Test DNS resolution from a pod (nslookup kubernetes.default). Owner: developer on-call. Review: during pod connectivity issues.
  • [ ] Check CoreDNS pods and logs (kubectl logs -n kube-system -l k8s-app=kube-dns). Owner: cluster operator. Review: when DNS fails.
  • [ ] Verify service endpoints are non-empty (kubectl get endpoints <svc>). Owner: application owner. Review: when service routing is suspect.
  • [ ] Test authorization as the affected user (kubectl auth can-i --as=...). Owner: security engineer. Review: when access is denied.
  • [ ] List and inspect NetworkPolicies in the namespace (kubectl get networkpolicies). Owner: network admin. Review: when traffic is blocked.
  • [ ] Review API server audit logs for forbidden requests. Owner: security engineer. Review: monthly or after incidents.
  • [ ] Check node firewall rules for required ports (6443, 10250, CNI ports). Owner: infrastructure engineer. Review: quarterly or after network changes.
  • [ ] Document findings and any configuration changes made. Owner: incident commander. Review: after every incident.

Repeat these steps in a structured manner, moving from network connectivity to authorization decisions. Keep a log of commands executed and their outputs for post-incident review.

Conclusion

Kubernetes authorization networking issues can be complex, but a systematic approach helps isolate the root cause quickly. Start by inventorying your environment, then verify connectivity, DNS, service routing, and authorization decisions in order. Use safe, scoped configuration changes and always test in a non-production environment first. The diagnostics and recovery steps provided here give you concrete commands and expected outputs to resolve common problems. By following the operations checklist, avoiding common pitfalls, and assigning clear ownership, you can reduce downtime and prevent recurrence. Continue to build on this foundation by enabling audit logging and monitoring network policies to catch issues before they impact users.

Related Research

Article Quality Score

Reader usefulness 100%
  • check_circle Reader-ready guide
  • check_circle Practical examples included
  • check_circle Clean SEO article URL