Kubernetes Service Account Networking Troubleshooting with Practical Examples

Intro

When a pod using a Kubernetes Service Account cannot reach an internal service, the root cause is rarely the application code. More often, it is a networking misconfiguration involving the service account's permissions, DNS resolution, or network policies. This guide walks through a systematic troubleshooting approach for Kubernetes Service Account networking issues, using practical examples and commands you can run in your own cluster.

We focus on the network path from a pod with a specific service account to a Kubernetes Service. We cover how to verify that the service account exists, that the pod is using it correctly, that DNS resolves the service name, that network policies allow traffic, and that the service endpoints are reachable. By the end, you will have a repeatable process to diagnose and fix common networking problems tied to service accounts.

Version and Environment Inventory

Before troubleshooting, document your environment. This ensures that you are testing against the correct cluster and that you understand the components involved.

1. Record Kubernetes and Client Versions

kubectl version --short

Expected output (example):

Client Version: v1.24.3
Server Version: v1.24.3

If your client and server versions differ significantly, some commands or API resources might behave differently. Note the versions for reference.

2. Confirm Cluster Context and Namespace

Ensure you are operating in the correct cluster and namespace.

kubectl config current-context
kubectl get namespaces

Check the namespace where your pod and service live. For this guide, we assume a namespace called app-ns.

3. Identify the Pod and Its Service Account

List pods with their service accounts:

kubectl get pods -n app-ns -o custom-columns=POD:.metadata.name,SERVICE_ACCOUNT:.spec.serviceAccountName

Expected output (example):

POD                     SERVICE_ACCOUNT
web-5b7c9d6f8-abcde     web-sa
api-7f8d9c6b5-fghij     api-sa

Note the service account name for the pod you are troubleshooting. In this example, we will use web-sa.

4. Check Service Account Existence and Details

Verify the service account exists and note its secrets (if any) and associated roles.

kubectl get serviceaccount web-sa -n app-ns -o yaml

Look for secrets and imagePullSecrets fields. If the service account has no secrets listed, that is normal in Kubernetes 1.24+ where token secrets are not automatically created. The pod can still authenticate using projected tokens.

5. Verify Network Topology

Understand how pods communicate. Common CNI plugins (Calico, Flannel, Cilium) have different default behaviors. Check for NetworkPolicy resources in the namespace:

kubectl get networkpolicies -n app-ns

If any exist, they may be blocking traffic. We will examine them later.

Prerequisites for Troubleshooting

You need kubectl access to the cluster and, ideally, permission to exec into pods. You will also need a basic understanding of Kubernetes Services, Endpoints, and DNS.

Safe Configuration Path

To demonstrate troubleshooting, we need a working setup. We will create a minimal example: a service account, a deployment using that service account, and a Service that the pod should reach.

1. Create a Service Account

Define a service account in sa.yaml:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: web-sa
  namespace: app-ns

Apply it:

kubectl apply -f sa.yaml

Expected output:

serviceaccount/web-sa created

2. Create a Deployment Using the Service Account

Create deploy.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: app-ns
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      serviceAccountName: web-sa
      containers:
      - name: web
        image: nginx:alpine
        ports:
        - containerPort: 80

Apply:

kubectl apply -f deploy.yaml

Verify the pod is running:

kubectl get pods -n app-ns -l app=web

Expected output:

NAME                     READY   STATUS    RESTARTS   AGE
web-5b7c9d6f8-abcde      1/1     Running   0          30s

3. Create a Service

Define service.yaml for an internal service that the web pod should access. We use a simple ClusterIP service:

apiVersion: v1
kind: Service
metadata:
  name: api
  namespace: app-ns
spec:
  selector:
    app: api
  ports:
  - port: 80
    targetPort: 8080

Apply it:

kubectl apply -f service.yaml

Verify:

kubectl get services -n app-ns

Expected output:

NAME   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
api    ClusterIP   10.96.100.50    <none>        80/TCP    10s

4. Create a Backend Pod for the Service

For the service to have endpoints, create a simple backend deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  namespace: app-ns
spec:
  replicas: 1
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
      - name: api
        image: hashicorp/http-echo:latest
        args:
        - "-text=API is up"
        - "-listen=:8080"
        ports:
        - containerPort: 8080

Apply and wait for it to be ready.

This setup gives us a controlled environment to test connectivity. The web pod uses service account web-sa. We have not yet configured any network policies, so by default all traffic is allowed.

Verification and Diagnostics

Now we systematically verify each layer of the networking path.

1. Confirm Pod Uses Correct Service Account

Check the pod spec:

kubectl get pod -n app-ns -l app=web -o jsonpath='{.items[0].spec.serviceAccountName}'

Expected output:

web-sa

If empty, the pod uses the default service account in the namespace.

2. Check Service Account Token Mounted in Pod

Kubernetes projects a service account token into the pod at /var/run/secrets/kubernetes.io/serviceaccount. Verify it exists:

kubectl exec -n app-ns <web-pod-name> -- ls /var/run/secrets/kubernetes.io/serviceaccount

Expected output:

ca.crt  namespace  token

If these files are missing, the pod may not be able to authenticate to the API server, which can affect certain networking features (though not direct service-to-service traffic).

3. DNS Resolution Inside Pod

From the web pod, attempt to resolve the service's DNS name. The full DNS name for a service in the same namespace is <service>.<namespace>.svc.cluster.local. For cross-namespace, it's <service>.<namespace>.svc.cluster.local as well, but the short name <service> works within the same namespace.

kubectl exec -n app-ns <web-pod-name> -- nslookup api

Expected output (example):

Server:         10.96.0.10
Address:        10.96.0.10#53

Name:   api.app-ns.svc.cluster.local
Address: 10.96.100.50

If nslookup is not available, use getent hosts api or cat /etc/resolv.conf to check DNS configuration.

Check resolv.conf:

kubectl exec -n app-ns <web-pod-name> -- cat /etc/resolv.conf

Expected output (example):

nameserver 10.96.0.10
search app-ns.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

The nameserver should be the cluster DNS service IP (usually kube-dns). If DNS fails, the service IP may not be returned.

4. Test Connectivity to Service IP and Port

Use curl or wget from the pod to access the service. Since our backend echoes a message, we can expect a response.

kubectl exec -n app-ns <web-pod-name> -- curl -s http://api

Expected output:

API is up

If curl is not available, use nc -zv api 80 or telnet api 80 to test TCP connectivity.

If this fails, check the service endpoints:

kubectl get endpoints -n app-ns api

Expected output (example):

NAME   ENDPOINTS           AGE
api    10.244.1.5:8080     5m

If endpoints are empty, the service selector does not match any pods. Verify the backend pod labels:

kubectl get pods -n app-ns -l app=api --show-labels

5. Check Network Policies

If NetworkPolicies exist, they may block traffic. List them:

kubectl get networkpolicies -n app-ns

If present, inspect them. For example, a policy might deny all ingress to pods with label app=api unless from specific sources. You can test by temporarily creating an allow-all policy (only in a test environment):

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-all
  namespace: app-ns
spec:
  podSelector: {}
  ingress:
  - {}
  egress:
  - {}
  policyTypes:
  - Ingress
  - Egress

Apply it, retest connectivity, then remove it. If connectivity works with the policy, you know the previous policies were the cause.

6. Verify Service Account Permissions (RBAC) for API Access

Sometimes the pod needs to call the Kubernetes API (e.g., to list services). This requires RBAC roles bound to the service account. Check if the service account has a role binding:

kubectl get rolebindings,clusterrolebindings -n app-ns -o json | jq '.items[] | select(.subjects[]?.name=="web-sa")'

If no bindings exist, the pod cannot perform API calls. To test API access, run:

kubectl exec -n app-ns <web-pod-name> -- curl -s -k -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" https://kubernetes.default.svc/api/v1/namespaces/app-ns/services

If you get a 403 Forbidden, the service account lacks RBAC permissions. Note that service-to-service traffic does not require API access, but some applications rely on it.

7. Inspect kube-proxy and Service Rules

On the node (if you have access), check iptables rules for the service IP:

sudo iptables-save | grep 10.96.100.50

This shows the DNAT rules that route traffic to the pod IP. If rules are missing, kube-proxy may be malfunctioning.

Failure Modes and Recovery

Common failure modes and how to recover from them.

Failure 1: Service Account Name Typo or Missing

If the pod spec references a non-existent service account, the pod will fail to start with an error like:

Error creating: pods "web-..." is forbidden: error looking up service account app-ns/web-sa: serviceaccount "web-sa" not found

Recovery: Create the service account or correct the name in the deployment. Apply the change and the pod will restart automatically.

Failure 2: DNS Resolution Failure

Symptoms: nslookup returns server can't find api: NXDOMAIN or times out.

Possible causes:

• CoreDNS pods are not running: check kubectl get pods -n kube-system -l k8s-app=kube-dns.
• CoreDNS configuration issues: check logs kubectl logs -n kube-system -l k8s-app=kube-dns.
• Pod's resolv.conf is incorrect due to custom dnsPolicy or dnsConfig.

Recovery: Ensure CoreDNS is healthy. If the pod's DNS settings are wrong, adjust the pod spec. For quick testing, you can temporarily use the service's ClusterIP directly: curl http://10.96.100.50.

Failure 3: Network Policy Blocking Traffic

If NetworkPolicies deny traffic, pods cannot connect. Verify by examining policies and their selectors. Often a policy is intended to allow only specific namespaces or pods.

Recovery: Modify the NetworkPolicy to allow traffic from the web pod's labels or namespace. For example, to allow traffic from pods with label app=web:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-web-to-api
  namespace: app-ns
spec:
  podSelector:
    matchLabels:
      app: api
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: web
    ports:
    - port: 8080

Apply and retest.

Failure 4: Service Endpoints Empty

If kubectl get endpoints shows no endpoints, the service selector does not match any running pods. Check that the backend pods have the correct labels and are in the Ready state.

Recovery: Fix the labels or ensure the backend pods are running. For example, if the backend pods have label app=api-v2 but the service selects app=api, update the service selector.

Failure 5: RBAC Restrictions on API Calls

If the pod needs to access the Kubernetes API and gets 403 errors, create appropriate Role and RoleBinding.

Example Role to allow reading services:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: app-ns
  name: service-reader
rules:
- apiGroups: [""]
  resources: ["services"]
  verbs: ["get", "list"]

RoleBinding:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: app-ns
  name: web-sa-service-reader
subjects:
- kind: ServiceAccount
  name: web-sa
  namespace: app-ns
roleRef:
  kind: Role
  name: service-reader
  apiGroup: rbac.authorization.k8s.io

Apply and retest API access.

Rollback Strategies

Always keep a backup of original YAML files before making changes. If a change breaks connectivity, revert to the previous version using kubectl apply -f previous.yaml. For NetworkPolicies, you can delete the offending policy and reapply the known-good one.

Recovery Checks

After any change, run the following verification sequence:

• kubectl get pods -n app-ns - ensure pods are Running and Ready.
• kubectl get endpoints -n app-ns api - ensure endpoints are populated.
• kubectl exec -n app-ns <web-pod> -- curl -s http://api - expect the backend response.
• kubectl exec -n app-ns <web-pod> -- nslookup api - expect correct IP.

Operations Checklist

Use this checklist for repeatable troubleshooting and maintenance.

Perform these checks whenever a networking issue arises, after any configuration change, and during routine audits.

Common Pitfalls and How to Avoid Them

Even experienced Kubernetes administrators can overlook certain details that lead to service account networking failures. Here are the most frequent mistakes and how to steer clear of them.

1. Assuming the Default Network Policy Allows All Traffic

Many teams enable a default-deny NetworkPolicy at the namespace level and then forget to add an allow rule for their workloads. A pod that worked yesterday may suddenly lose connectivity after a policy is applied.

Why it happens: Kubernetes NetworkPolicies are additive: if any policy selects a pod, the pod is isolated and only traffic explicitly allowed by the union of policies is permitted.

How to avoid: Always document NetworkPolicies and their intended effect. When troubleshooting, check for policies that select the source or destination pods. Use kubectl describe networkpolicy <name> -n <namespace> to see the exact rules.

2. Ignoring DNS Search Domains and ndots

The ndots:5 option in /etc/resolv.conf causes the resolver to try the full name with search domains before attempting the absolute name. This can lead to unexpected DNS lookups and delays if you frequently use external hostnames without a trailing dot.

Why it happens: Kubernetes injects the cluster search domains into every pod by default. For a name like api, the resolver first tries api.app-ns.svc.cluster.local, then api.svc.cluster.local, etc., before falling back to the absolute name.

How to avoid: Be aware of this behavior when debugging DNS issues. Use fully qualified domain names (FQDNs) ending with a dot to bypass search domains, or adjust dnsConfig in the pod spec if necessary.

3. Forgetting That Service Accounts Do Not Directly Affect Service-to-Service Networking

A common misconception is that a service account is required for a pod to reach another service. In reality, service-to-service traffic within the cluster relies on DNS, kube-proxy, and network policies, not on the service account. The service account matters only when the pod needs to authenticate to the Kubernetes API or to external systems using its token.

Why it happens: Documentation often bundles service account configuration with general pod security and networking, leading to confusion.

How to avoid: Separate the concerns. If a pod cannot reach a service, check DNS and network policies first. Only investigate RBAC and service account tokens if the pod fails to access the Kubernetes API or external services that require authentication.

4. Using a Stale Endpoint Object for Debugging

After fixing a service selector or backend pod labels, the Endpoints object may not update immediately. Checking kubectl get endpoints too soon can show empty endpoints even though the service is now correctly configured.

Why it happens: The Endpoints controller has a reconciliation loop and may take a few seconds to update.

How to avoid: Wait for the controller to sync, or watch the endpoints object with kubectl get endpoints -n app-ns -w to see the update in real time.

5. Overlooking the Impact of NodePort or LoadBalancer Services on Internal Communication

When a service is exposed as NodePort or LoadBalancer, it still has a ClusterIP. Internal pods should use the ClusterIP or the DNS name; using the node port from a pod may fail if the node's firewall or network policy blocks that port.

Why it happens: Developers sometimes test from a pod using the node's IP and the NodePort, which may not be reachable due to network policies or node firewall rules.

How to avoid: Always use the service DNS name or ClusterIP from inside the cluster. Reserve NodePort and LoadBalancer for external traffic.

Conclusion

Troubleshooting Kubernetes Service Account networking requires a methodical approach. Start by verifying the service account and pod configuration, then test DNS resolution and connectivity, and finally inspect network policies and RBAC. The practical commands and examples in this guide provide a repeatable process to identify and resolve most issues. By following the operations checklist, you can quickly diagnose problems and restore service communication.

Remember to keep your environment documented and to test changes in a safe, isolated namespace before rolling out to production. With these tools, you can confidently manage service account networking in Kubernetes.