Kubernetes Service Account Architecture: Design, Security, and Operations with Practical Examples

Intro

Service accounts in Kubernetes are the identities that workloads use to authenticate to the Kubernetes API and other services. A pod without a service account is like a user without credentials: it can exist, but it cannot do anything meaningful inside the cluster. Yet many teams treat service accounts as an afterthought, minting cluster-admin tokens for every deployment and leaving them in plaintext inside pod specs. This article walks through the architecture of Kubernetes service accounts, how tokens are issued and mounted, how RBAC turns an identity into permissions, and how to operate them safely in production. Along the way, you will find concrete commands, manifests, failure scenarios, and verification steps that move you from guesswork to a repeatable process.

Even if you have used Kubernetes for years, the service account landscape has changed several times. Legacy long-lived tokens stored in Secrets are being replaced by bound, time-limited tokens projected directly into pods. Understanding this shift matters because old documentation, old manifests, and old habits can silently create permissions that are broader and longer-lived than you intend. By the end of this article, you should be able to inspect an existing cluster, issue a narrowly scoped identity to an application, verify that it works, and know what to do when it does not.

Version and Environment Inventory

Before changing anything, you need to know what you are running. Kubernetes service account behavior differs between versions, especially around legacy tokens and projected volumes. Run the following read-only commands and record the output. They tell you the cluster version, the API resources available, and the current state of service accounts in a namespace.

kubectl version --short
kubectl api-versions | grep authentication
kubectl get serviceaccounts -n default
kubectl get secrets -n default

In Kubernetes 1.24 and later, service accounts are no longer automatically populated with a Secret containing a long-lived token. Instead, you must explicitly create a token using kubectl create token or rely on the projected service account token that is mounted into every pod by default. If you see an empty Secrets column for a service account, that is expected and not an error. If you are on an older cluster, each service account typically has a Secret named <sa-name>-token-<random> containing a token, a CA certificate, and a namespace. Compare what you see against the version to avoid chasing a missing Secret that newer versions no longer create.

Check the admission controllers and feature gates that affect service accounts. The ServiceAccount admission plugin is responsible for attaching the projected token volume and setting the pod's serviceAccountName. The BoundServiceAccountTokenVolume feature gate has been enabled by default since 1.21 and stable since 1.22, so you normally do not need to toggle it, but custom distributions may differ.

kubectl get pod -n kube-system -l component=kube-apiserver -o jsonpath='{.items[0].spec.containers[0].command}' | tr ',' '\n' | grep -E 'service-account|token'

Keep a small, local test environment for experiments. Minikube, kind, or k3s are sufficient. Apply one manifest at a time and inspect the generated resources. For example, create a pod with a custom service account and verify the token mount:

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata:
  name: inventory-sa
  namespace: default
---
apiVersion: v1
kind: Pod
metadata:
  name: inventory-test
spec:
  serviceAccountName: inventory-sa
  containers:
  - name: app
    image: nginx:1.25
EOF

kubectl exec inventory-test -- ls /var/run/secrets/kubernetes.io/serviceaccount
# Expected output: ca.crt  namespace  token

Notice that the token file is a projected token. It is short-lived, automatically rotated, and tied to the pod's identity. The legacy Secret-based token, if it exists, is separate and may be long-lived.

Safe Configuration Path

The phrase "least privilege" gets repeated often but rarely operationalized. For service accounts, the safe configuration path is: create a dedicated service account for each workload, bind it to a Role or ClusterRole with the minimum verbs and resources it actually needs, and let the default token projection handle authentication. Do not attach the same broad ClusterRole to multiple workloads, and never use cluster-admin for an application.

Start by documenting what the workload does. Suppose you have a deployment that only needs to list pods in its own namespace. Create the identity first:

kubectl create serviceaccount pod-reader-sa -n default

Then define a Role that allows only get and list on pods in that namespace:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-reader
  namespace: default
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch"]

Bind the Role to the ServiceAccount with a RoleBinding:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: pod-reader-binding
  namespace: default
subjects:
- kind: ServiceAccount
  name: pod-reader-sa
  namespace: default
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

Apply the Role and RoleBinding, then test the permission by creating a temporary pod using the service account and attempting to list pods:

kubectl apply -f role.yaml -f rolebinding.yaml
kubectl run test-pod --rm -i --tty --image=alpine --serviceaccount=pod-reader-sa -- sh
# Inside the pod:
wget -qO- --header="Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" --ca-certificate=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt https://kubernetes.default.svc/api/v1/namespaces/default/pods

If the role is correctly scoped, you will see a JSON list of pods. If you try to list deployments, you should get a 403 Forbidden response. That negative test is just as important as the positive one because it proves the Role is not broader than intended.

For pods that need read-only cluster-wide access, use a ClusterRole and ClusterRoleBinding. For example, this ClusterRole grants read access to nodes:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: node-reader
rules:
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["get", "list", "watch"]

Bind it to a specific service account in a specific namespace:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: node-reader-binding
subjects:
- kind: ServiceAccount
  name: node-reader-sa
  namespace: monitoring
roleRef:
  kind: ClusterRole
  name: node-reader
  apiGroup: rbac.authorization.k8s.io

Never bind cluster-admin to a ServiceAccount unless you are building a cluster management controller, and even then consider more granular roles. If you find an existing binding that uses cluster-admin, replace it or at least add an annotation explaining the owner and review date.

Verification and Diagnostics

When a workload reports permission errors, the fastest diagnostic is to read the pod's events and then inspect the service account and its bindings. Use the following sequence:

kubectl describe pod <pod-name> -n <namespace>
kubectl get serviceaccount <sa-name> -n <namespace> -o yaml
kubectl get rolebindings,clusterrolebindings -n <namespace> -o wide | grep <sa-name>

The pod's events may show Failed to mount volume or token not found. If the service account does not exist, the pod will stay in a Pending state with an event like serviceaccount "missing-sa" not found.

A common issue is that the role's apiGroups does not match the resource. For core resources like pods, services, and nodes, the apiGroups should be [""] (empty string), not ["v1"] or ["core"]. For apps resources like deployments, use ["apps"]. Check the resource's API group with kubectl api-resources | grep <resource>.

If a pod previously worked and now fails, check whether its service account token has expired. Projected tokens have a default lifetime of 1 hour, but the kubelet rotates them automatically. However, if the pod is long-running and the rotation fails due to a kubelet restart or network issue, the pod may retain an expired token. Verify the token's expiry by decoding it:

kubectl exec <pod-name> -- cat /var/run/secrets/kubernetes.io/serviceaccount/token | cut -d. -f2 | base64 -d 2>/dev/null | jq .exp

Compare the numeric expiration timestamp to the current time. A healthy pod should have a token that expires later than now, usually within the next hour.

To see all permissions that a service account currently has, use kubectl auth can-i:

kubectl auth can-i list pods --as=system:serviceaccount:default:pod-reader-sa -n default
kubectl auth can-i create deployments --as=system:serviceaccount:default:pod-reader-sa -n default

The first command should return yes, the second no. This is a quick way to validate RBAC without deploying a test pod.

Failure Modes and Recovery

Missing service account. If a pod's manifest refers to a service account that does not exist, the pod will never be scheduled. Recover by creating the service account or correcting the name. Use kubectl get serviceaccounts -n <namespace> to list available accounts.

Overly broad permissions. A service account bound to cluster-admin can delete all resources. Recovery: identify the offending binding, remove it, create a minimal Role with only needed verbs and resources, and bind that instead. Run kubectl auth can-i --list --as=system:serviceaccount:<ns>:<sa> to audit all permissions.

Token theft. A leaked token can be used until it expires, which for legacy tokens may be years. Recovery: delete the legacy Secret, recreate the service account if needed, and migrate to projected tokens. You can force-rotate all tokens by deleting the pod. If you cannot stop using long-lived tokens, at least set a short expiration on the Secret.

Namespace mismatch in RoleBinding. A RoleBinding in namespace A cannot grant permissions to a service account in namespace B for namespaced resources. The binding must be in the same namespace as the Role and the service account. Recovery: verify the namespaces in the RoleBinding's subjects and roleRef.

API group errors. Symptoms include 403 Forbidden even though the role seems correct. Use kubectl explain or the API reference to confirm the correct apiGroups for the resource. Core resources use "", apps use "apps", batch uses "batch".

Expired token in long-running pod. If a pod cannot reach the API after many hours, manually check the token expiry. If expired and not rotated, restart the pod to mount a fresh token. For apps that need to detect token refresh dynamically, use a client library that watches the token file. Official Kubernetes clients and most community clients support automatic reload.

ServiceAccount admission plugin disabled. If the kubelet does not mount a token at all, verify that the ServiceAccount admission plugin is enabled in the API server. This is rare but can happen in custom clusters built with kubeadm or other tools. Recovery involves editing the API server manifest to include the plugin and restarting the API server.

Operations Checklist

Use this checklist as a starting point for any new workload that needs Kubernetes API access. Adjust to your organization's change management process, but keep the spirit: verify before you modify, and modify the smallest thing possible.

1. Identify the workload's required Kubernetes API actions. Which resources and verbs does it actually need? Write them down. Example: get, list pods in namespace 'payments'.
2. Create a dedicated ServiceAccount. kubectl create serviceaccount payments-reader -n payments. Do not use the default service account.
3. Define a Role or ClusterRole with exactly those permissions. Use verbs: ["get", "list"] and resources: ["pods"]. Avoid wildcards unless absolutely necessary, and document why.
4. Create a RoleBinding or ClusterRoleBinding that connects the ServiceAccount to the Role. Verify the namespaces match.
5. Test the permissions before deploying to production. Run kubectl auth can-i with the service account as shown earlier. Confirm both allowed and denied actions.
6. Deploy the application with serviceAccountName set in the pod spec. Ensure no legacy Secret token is mounted unless explicitly needed.
7. Monitor pod logs and events for authentication or authorization errors. Set up an alert for 403 Forbidden in your log aggregation.
8. Review service account permissions quarterly. Use kubectl get rolebindings,clusterrolebindings --all-namespaces -o json and script a report that flags bindings to high-risk ClusterRoles like cluster-admin, edit, and admin.
9. Rotate any long-lived tokens at least every 90 days. Better yet, eliminate them and rely on projected tokens.
10. Document the service account's purpose, owner, and review date in an annotation on the ServiceAccount. For example:

metadata:
  name: payments-reader
  namespace: payments
  annotations:
    purpose: "Read-only access to pods for the reconciliation controller"
    owner: "Priya Shah, Platform Team Lead"
    review-date: "2026-03-31"

Assign a single accountable owner for each major decision in this checklist. For example, the Platform Team Lead owns approving any ClusterRoleBinding, the Security Architect owns the quarterly permission review, and the Application Developer owns ensuring their deployment uses the correct serviceAccountName. Revisit the roles and permissions quarterly, or immediately upon any significant change to the application's functionality.

Common Pitfalls

Even experienced engineers fall into the same traps. Here are the most frequent mistakes and how to avoid them.

Pitfall 1: Giving every app cluster-admin. Teams under time pressure often start with cluster-admin and never narrow it down. Why it happens: the app initially fails with permission errors, and someone grants the broadest role to make it work quickly. How to avoid: invest an hour in determining the exact API calls the app makes. Use kubectl auth can-i --list to simulate and iteratively build a minimal Role.

Pitfall 2: Not understanding legacy vs. projected tokens. Older tutorials and manifests often reference secrets and serviceAccountToken. If you are on Kubernetes 1.24+, those might not work as expected. How to avoid: check your cluster version first, and prefer the default projected token. If you must use a legacy token, explicitly create a Secret of type kubernetes.io/service-account-token and set automountServiceAccountToken: false where appropriate.

Pitfall 3: RoleBinding in the wrong namespace. A namespace mismatch silently fails. Suppose you create a Role in namespace staging and bind it to a service account in production without a ClusterRoleBinding. The production pod will get 403 errors. How to avoid: always specify namespace in both the RoleBinding subjects and roleRef, and verify with kubectl auth can-i.

Pitfall 4: Misusing apiGroups. For core resources like pods and services, using apiGroups: ["v1"] instead of apiGroups: [""] results in no permissions. How to avoid: run kubectl api-resources | grep <resource> to see the correct API group. Core group is blank, not v1.

Pitfall 5: Overlooking the default service account. Every namespace has a default service account with no permissions, and pods use it if you do not specify one. If a pod makes API calls, it will get 403 Forbidden. How to avoid: explicitly set serviceAccountName for pods that need API access, and never add permissions to the default service account, because then all pods without a custom account inherit them.

Pitfall 6: Leaking tokens in logs or config. A common mistake is printing the token in a debug command or committing it to a config map. How to avoid: use kubectl create token for temporary access, and never store tokens in plaintext. Treat a service account token like a bearer credential: if it leaks, revoke it by deleting the pod or the Secret.

Conclusion

Service accounts are a foundational part of Kubernetes security, but they are often overlooked until something breaks or is breached. By understanding the architecture of tokens, RBAC bindings, and pod identity projection, you can design workloads that have exactly the permissions they need and no more. The commands and manifests in this article give you a practical toolkit for auditing existing service accounts, creating new one with least privilege, diagnosing access issues, and recovering from common failures.

Start with one low-risk verification: pick a running pod in your cluster, inspect its service account, run kubectl auth can-i --list, and compare the output with what the application actually does. Chances are you will find at least one surprise. Then document the service account, assign an owner, and set a review date. This small habit builds the reliability and safety that production Kubernetes demands.

A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. Service accounts are not just a configuration detail; they are the identity foundation of your cluster's security model.