## Intro When a pod using a Kubernetes Service Account cannot reach an internal service, the root cause is rarely the application code. More often, it is a networking misconfiguration involving the service account's permissions, DNS resolution, or network policies. This guide walks through a systematic troubleshooting approach for Kubernetes Service Account networking issues, using practical examples and commands you can run in your own cluster. We focus on the network path from a pod with a specific service account to a Kubernetes Service. We cover how to verify that the service account exists, that the pod is using it correctly, that DNS resolves the service name, that network policies allow traffic, and that the service endpoints are reachable. By the end, you will have a repeatable process to diagnose and fix common networking problems tied to service accounts. ## Version and Environment Inventory Before troubleshooting, document your environment. This ensures that you are testing against the correct cluster and that you understand the components involved. ### 1. Record Kubernetes and Client Versions ```bash kubectl version --short ``` Expected output (example): ```text Client Version: v1.24.3 Server Version: v1.24.3 ``` If your client and server versions differ significantly, some commands or API resources might behave differently. Note the versions for reference. ### 2. Confirm Cluster Context and Namespace Ensure you are operating in the correct cluster and namespace. ```bash kubectl config current-context kubectl get namespaces ``` Check the namespace where your pod and service live. For this guide, we assume a namespace called `app-ns`. ### 3. Identify the Pod and Its Service Account List pods with their service accounts: ```bash kubectl get pods -n app-ns -o custom-columns=POD:.metadata.name,SERVICE_ACCOUNT:.spec.serviceAccountName ``` Expected output (example): ```text POD SERVICE_ACCOUNT web-5b7c9d6f8-abcde web-sa api-7f8d9c6b5-fghij api-sa ``` Note the service account name for the pod you are troubleshooting. In this example, we will use `web-sa`. ### 4. Check Service Account Existence and Details Verify the service account exists and note its secrets (if any) and associated roles. ```bash kubectl get serviceaccount web-sa -n app-ns -o yaml ``` Look for `secrets` and `imagePullSecrets` fields. If the service account has no secrets listed, that is normal in Kubernetes 1.24+ where token secrets are not automatically created. The pod can still authenticate using projected tokens. ### 5. Verify Network Topology Understand how pods communicate. Common CNI plugins (Calico, Flannel, Cilium) have different default behaviors. Check for NetworkPolicy resources in the namespace: ```bash kubectl get networkpolicies -n app-ns ``` If any exist, they may be blocking traffic. We will examine them later. ### Prerequisites for Troubleshooting You need `kubectl` access to the cluster and, ideally, permission to exec into pods. You will also need a basic understanding of Kubernetes Services, Endpoints, and DNS. ## Safe Configuration Path To demonstrate troubleshooting, we need a working setup. We will create a minimal example: a service account, a deployment using that service account, and a Service that the pod should reach. ### 1. Create a Service Account Define a service account in `sa.yaml`: ```yaml apiVersion: v1 kind: ServiceAccount metadata: name: web-sa namespace: app-ns ``` Apply it: ```bash kubectl apply -f sa.yaml ``` Expected output: ```text serviceaccount/web-sa created ``` ### 2. Create a Deployment Using the Service Account Create `deploy.yaml`: ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: web namespace: app-ns spec: replicas: 1 selector: matchLabels: app: web template: metadata: labels: app: web spec: serviceAccountName: web-sa containers: - name: web image: nginx:alpine ports: - containerPort: 80 ``` Apply: ```bash kubectl apply -f deploy.yaml ``` Verify the pod is running: ```bash kubectl get pods -n app-ns -l app=web ``` Expected output: ```text NAME READY STATUS RESTARTS AGE web-5b7c9d6f8-abcde 1/1 Running 0 30s ``` ### 3. Create a Service Define `service.yaml` for an internal service that the web pod should access. We use a simple ClusterIP service: ```yaml apiVersion: v1 kind: Service metadata: name: api namespace: app-ns spec: selector: app: api ports: - port: 80 targetPort: 8080 ``` Apply it: ```bash kubectl apply -f service.yaml ``` Verify: ```bash kubectl get services -n app-ns ``` Expected output: ```text NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE api ClusterIP 10.96.100.50 80/TCP 10s ``` ### 4. Create a Backend Pod for the Service For the service to have endpoints, create a simple backend deployment: ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: api namespace: app-ns spec: replicas: 1 selector: matchLabels: app: api template: metadata: labels: app: api spec: containers: - name: api image: hashicorp/http-echo:latest args: - "-text=API is up" - "-listen=:8080" ports: - containerPort: 8080 ``` Apply and wait for it to be ready. This setup gives us a controlled environment to test connectivity. The web pod uses service account `web-sa`. We have not yet configured any network policies, so by default all traffic is allowed. ## Verification and Diagnostics Now we systematically verify each layer of the networking path. ### 1. Confirm Pod Uses Correct Service Account Check the pod spec: ```bash kubectl get pod -n app-ns -l app=web -o jsonpath='{.items[0].spec.serviceAccountName}' ``` Expected output: ```text web-sa ``` If empty, the pod uses the `default` service account in the namespace. ### 2. Check Service Account Token Mounted in Pod Kubernetes projects a service account token into the pod at `/var/run/secrets/kubernetes.io/serviceaccount`. Verify it exists: ```bash kubectl exec -n app-ns -- ls /var/run/secrets/kubernetes.io/serviceaccount ``` Expected output: ```text ca.crt namespace token ``` If these files are missing, the pod may not be able to authenticate to the API server, which can affect certain networking features (though not direct service-to-service traffic). ### 3. DNS Resolution Inside Pod From the web pod, attempt to resolve the service's DNS name. The full DNS name for a service in the same namespace is `..svc.cluster.local`. For cross-namespace, it's `..svc.cluster.local` as well, but the short name `` works within the same namespace. ```bash kubectl exec -n app-ns -- nslookup api ``` Expected output (example): ```text Server: 10.96.0.10 Address: 10.96.0.10#53 Name: api.app-ns.svc.cluster.local Address: 10.96.100.50 ``` If `nslookup` is not available, use `getent hosts api` or `cat /etc/resolv.conf` to check DNS configuration. Check `resolv.conf`: ```bash kubectl exec -n app-ns -- cat /etc/resolv.conf ``` Expected output (example): ```text nameserver 10.96.0.10 search app-ns.svc.cluster.local svc.cluster.local cluster.local options ndots:5 ``` The nameserver should be the cluster DNS service IP (usually `kube-dns`). If DNS fails, the service IP may not be returned. ### 4. Test Connectivity to Service IP and Port Use `curl` or `wget` from the pod to access the service. Since our backend echoes a message, we can expect a response. ```bash kubectl exec -n app-ns -- curl -s http://api ``` Expected output: ```text API is up ``` If `curl` is not available, use `nc -zv api 80` or `telnet api 80` to test TCP connectivity. If this fails, check the service endpoints: ```bash kubectl get endpoints -n app-ns api ``` Expected output (example): ```text NAME ENDPOINTS AGE api 10.244.1.5:8080 5m ``` If endpoints are empty, the service selector does not match any pods. Verify the backend pod labels: ```bash kubectl get pods -n app-ns -l app=api --show-labels ``` ### 5. Check Network Policies If NetworkPolicies exist, they may block traffic. List them: ```bash kubectl get networkpolicies -n app-ns ``` If present, inspect them. For example, a policy might deny all ingress to pods with label `app=api` unless from specific sources. You can test by temporarily creating an allow-all policy (only in a test environment): ```yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-all namespace: app-ns spec: podSelector: {} ingress: - {} egress: - {} policyTypes: - Ingress - Egress ``` Apply it, retest connectivity, then remove it. If connectivity works with the policy, you know the previous policies were the cause. ### 6. Verify Service Account Permissions (RBAC) for API Access Sometimes the pod needs to call the Kubernetes API (e.g., to list services). This requires RBAC roles bound to the service account. Check if the service account has a role binding: ```bash kubectl get rolebindings,clusterrolebindings -n app-ns -o json | jq '.items[] | select(.subjects[]?.name=="web-sa")' ``` If no bindings exist, the pod cannot perform API calls. To test API access, run: ```bash kubectl exec -n app-ns -- curl -s -k -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" https://kubernetes.default.svc/api/v1/namespaces/app-ns/services ``` If you get a 403 Forbidden, the service account lacks RBAC permissions. Note that service-to-service traffic does not require API access, but some applications rely on it. ### 7. Inspect kube-proxy and Service Rules On the node (if you have access), check iptables rules for the service IP: ```bash sudo iptables-save | grep 10.96.100.50 ``` This shows the DNAT rules that route traffic to the pod IP. If rules are missing, kube-proxy may be malfunctioning. ## Failure Modes and Recovery Common failure modes and how to recover from them. ### Failure 1: Service Account Name Typo or Missing If the pod spec references a non-existent service account, the pod will fail to start with an error like: ```text Error creating: pods "web-..." is forbidden: error looking up service account app-ns/web-sa: serviceaccount "web-sa" not found ``` **Recovery:** Create the service account or correct the name in the deployment. Apply the change and the pod will restart automatically. ### Failure 2: DNS Resolution Failure Symptoms: `nslookup` returns `server can't find api: NXDOMAIN` or times out. Possible causes: - CoreDNS pods are not running: check `kubectl get pods -n kube-system -l k8s-app=kube-dns`. - CoreDNS configuration issues: check logs `kubectl logs -n kube-system -l k8s-app=kube-dns`. - Pod's `resolv.conf` is incorrect due to custom `dnsPolicy` or `dnsConfig`. **Recovery:** Ensure CoreDNS is healthy. If the pod's DNS settings are wrong, adjust the pod spec. For quick testing, you can temporarily use the service's ClusterIP directly: `curl http://10.96.100.50`. ### Failure 3: Network Policy Blocking Traffic If NetworkPolicies deny traffic, pods cannot connect. Verify by examining policies and their selectors. Often a policy is intended to allow only specific namespaces or pods. **Recovery:** Modify the NetworkPolicy to allow traffic from the web pod's labels or namespace. For example, to allow traffic from pods with label `app=web`: ```yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-web-to-api namespace: app-ns spec: podSelector: matchLabels: app: api ingress: - from: - podSelector: matchLabels: app: web ports: - port: 8080 ``` Apply and retest. ### Failure 4: Service Endpoints Empty If `kubectl get endpoints` shows no endpoints, the service selector does not match any running pods. Check that the backend pods have the correct labels and are in the Ready state. **Recovery:** Fix the labels or ensure the backend pods are running. For example, if the backend pods have label `app=api-v2` but the service selects `app=api`, update the service selector. ### Failure 5: RBAC Restrictions on API Calls If the pod needs to access the Kubernetes API and gets 403 errors, create appropriate Role and RoleBinding. Example Role to allow reading services: ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: namespace: app-ns name: service-reader rules: - apiGroups: [""] resources: ["services"] verbs: ["get", "list"] ``` RoleBinding: ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: namespace: app-ns name: web-sa-service-reader subjects: - kind: ServiceAccount name: web-sa namespace: app-ns roleRef: kind: Role name: service-reader apiGroup: rbac.authorization.k8s.io ``` Apply and retest API access. ### Rollback Strategies Always keep a backup of original YAML files before making changes. If a change breaks connectivity, revert to the previous version using `kubectl apply -f previous.yaml`. For NetworkPolicies, you can delete the offending policy and reapply the known-good one. ### Recovery Checks After any change, run the following verification sequence: - `kubectl get pods -n app-ns` - ensure pods are Running and Ready. - `kubectl get endpoints -n app-ns api` - ensure endpoints are populated. - `kubectl exec -n app-ns -- curl -s http://api` - expect the backend response. - `kubectl exec -n app-ns -- nslookup api` - expect correct IP. ## Operations Checklist Use this checklist for repeatable troubleshooting and maintenance. | Step | Action | Command | Expected Result | |------|--------|---------|-----------------| | 1 | Verify service account exists | `kubectl get sa web-sa -n app-ns` | Service account listed | | 2 | Confirm pod uses correct SA | `kubectl get pod -n app-ns -o jsonpath='{.spec.serviceAccountName}'` | Outputs `web-sa` | | 3 | Check pod token files | `kubectl exec -n app-ns -- ls /var/run/secrets/kubernetes.io/serviceaccount` | Lists `ca.crt`, `namespace`, `token` | | 4 | Test DNS resolution | `kubectl exec -n app-ns -- nslookup api` | Returns service IP | | 5 | Test service connectivity | `kubectl exec -n app-ns -- curl -s http://api` | Returns expected response | | 6 | Check endpoints | `kubectl get endpoints api -n app-ns` | Non-empty endpoints | | 7 | Check network policies | `kubectl get networkpolicies -n app-ns` | Review any policies | | 8 | Check RBAC bindings | `kubectl get rolebindings,clusterrolebindings -n app-ns -o json \| jq '.items[] \| select(.subjects[]?.name=="web-sa")'` | Bindings if API access needed | | 9 | Review CoreDNS logs | `kubectl logs -n kube-system -l k8s-app=kube-dns` | No errors | Perform these checks whenever a networking issue arises, after any configuration change, and during routine audits. ## Common Pitfalls and How to Avoid Them Even experienced Kubernetes administrators can overlook certain details that lead to service account networking failures. Here are the most frequent mistakes and how to steer clear of them. ### 1. Assuming the Default Network Policy Allows All Traffic Many teams enable a default-deny NetworkPolicy at the namespace level and then forget to add an allow rule for their workloads. A pod that worked yesterday may suddenly lose connectivity after a policy is applied. **Why it happens:** Kubernetes NetworkPolicies are additive: if any policy selects a pod, the pod is isolated and only traffic explicitly allowed by the union of policies is permitted. **How to avoid:** Always document NetworkPolicies and their intended effect. When troubleshooting, check for policies that select the source or destination pods. Use `kubectl describe networkpolicy -n ` to see the exact rules. ### 2. Ignoring DNS Search Domains and ndots The `ndots:5` option in `/etc/resolv.conf` causes the resolver to try the full name with search domains before attempting the absolute name. This can lead to unexpected DNS lookups and delays if you frequently use external hostnames without a trailing dot. **Why it happens:** Kubernetes injects the cluster search domains into every pod by default. For a name like `api`, the resolver first tries `api.app-ns.svc.cluster.local`, then `api.svc.cluster.local`, etc., before falling back to the absolute name. **How to avoid:** Be aware of this behavior when debugging DNS issues. Use fully qualified domain names (FQDNs) ending with a dot to bypass search domains, or adjust `dnsConfig` in the pod spec if necessary. ### 3. Forgetting That Service Accounts Do Not Directly Affect Service-to-Service Networking A common misconception is that a service account is required for a pod to reach another service. In reality, service-to-service traffic within the cluster relies on DNS, kube-proxy, and network policies, not on the service account. The service account matters only when the pod needs to authenticate to the Kubernetes API or to external systems using its token. **Why it happens:** Documentation often bundles service account configuration with general pod security and networking, leading to confusion. **How to avoid:** Separate the concerns. If a pod cannot reach a service, check DNS and network policies first. Only investigate RBAC and service account tokens if the pod fails to access the Kubernetes API or external services that require authentication. ### 4. Using a Stale Endpoint Object for Debugging After fixing a service selector or backend pod labels, the Endpoints object may not update immediately. Checking `kubectl get endpoints` too soon can show empty endpoints even though the service is now correctly configured. **Why it happens:** The Endpoints controller has a reconciliation loop and may take a few seconds to update. **How to avoid:** Wait for the controller to sync, or watch the endpoints object with `kubectl get endpoints -n app-ns -w` to see the update in real time. ### 5. Overlooking the Impact of NodePort or LoadBalancer Services on Internal Communication When a service is exposed as NodePort or LoadBalancer, it still has a ClusterIP. Internal pods should use the ClusterIP or the DNS name; using the node port from a pod may fail if the node's firewall or network policy blocks that port. **Why it happens:** Developers sometimes test from a pod using the node's IP and the NodePort, which may not be reachable due to network policies or node firewall rules. **How to avoid:** Always use the service DNS name or ClusterIP from inside the cluster. Reserve NodePort and LoadBalancer for external traffic. ## Conclusion Troubleshooting Kubernetes Service Account networking requires a methodical approach. Start by verifying the service account and pod configuration, then test DNS resolution and connectivity, and finally inspect network policies and RBAC. The practical commands and examples in this guide provide a repeatable process to identify and resolve most issues. By following the operations checklist, you can quickly diagnose problems and restore service communication. Remember to keep your environment documented and to test changes in a safe, isolated namespace before rolling out to production. With these tools, you can confidently manage service account networking in Kubernetes.