Kubernetes Service Accounts Upgrade and Migration: An Admin's Practical Guide

Intro

Kubernetes Service Accounts are the identities that pods use to authenticate to the Kubernetes API and to external services. As clusters grow and security requirements mature, administrators often face upgrades such as moving from long-lived static tokens to projected, time-bound tokens, or migrating workloads to new Service Accounts with cleaner RBAC bindings. These tasks can disrupt applications if done without a clear plan.

This guide walks through the entire lifecycle: from initial version and environment inventory to safe configuration changes, verification, failure recovery, and an operational checklist. It is written for developers, DevOps consultants, and technical startup teams who need practical, command-focused procedures. While the examples focus on Service Accounts, the concepts also touch on RBAC and Secrets because they are tightly coupled with identity and access.

The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached. Each section includes concrete commands and expected outputs so you can follow along with confidence.

Version and Environment Inventory

Before touching any Service Account, know exactly what you have. A thorough inventory includes the Kubernetes version, API server flags, existing Service Accounts, their token types, and RBAC bindings. This prevents surprises like making a change that is incompatible with an older API version.

Start by checking the cluster version:

kubectl version --short

Expected output (example):

Client Version: v1.28.2
Server Version: v1.27.6

If your cluster is v1.24 or later, projected Service Account tokens (the BoundServiceAccountTokenVolume feature) are stable and should be your default. For earlier versions, you may still be relying on legacy static tokens. Knowing this difference is critical for an upgrade.

List existing Service Accounts in all namespaces:

kubectl get serviceaccounts --all-namespaces

Look for accounts that have been around a long time, especially in kube-system or custom namespaces. Then inspect a specific Service Account to see its secrets and associated tokens:

kubectl get serviceaccount my-app-sa -n my-namespace -o yaml

In the output, note the secrets field. If it lists multiple secrets, those are likely legacy token secrets. In newer clusters, this field is often empty because tokens are projected and not stored as secrets.

To determine whether the Service Account can issue short-lived tokens via the TokenRequest API, run:

kubectl create token my-app-sa -n my-namespace

This command requests a fresh, time-bound token and prints it to stdout. You can decode the JWT (e.g., using jq -R 'split(".") | .[1] | @base64d | fromjson') to inspect the exp claim and confirm it expires. Legacy static tokens are not created this way; they are stored as Secrets and never expire. If you find any legacy token Secrets in the ServiceAccount's secrets field, plan a migration to short-lived tokens.

Finally, review RBAC bindings for the Service Account:

kubectl get rolebindings,clusterrolebindings --all-namespaces -o json | jq '.items[] | select(.subjects[]?.name == "my-app-sa")'

This shows every role and cluster role assigned to the account. Document the permissions, because you will need to reproduce them securely after migration.

A quick environment inventory checklist:

• Kubernetes server version: kubectl version --short
• API server token settings: check kube-apiserver pod arguments for --service-account-issuer, --service-account-key-file, and --service-account-signing-key-file
• List all Service Accounts and their namespaces
• Identify legacy token secrets
• Map RBAC bindings for each Service Account in use

Safe Configuration Path

Once you understand your current state, you can plan a safe upgrade. The safest path is to create new projected token Service Accounts alongside the old ones, switch workloads gradually, and remove legacy tokens only after verification. This limits the blast radius and gives you a clear rollback point.

Step 1: Enable projected tokens (if not already enabled)

For clusters v1.21+, the feature is on by default. Verify by checking the kube-apiserver pod's spec:

kubectl get pod -n kube-system -l component=kube-apiserver -o yaml | grep -E 'service-account-(issuer|key-file|signing-key-file)'

You should see values for --service-account-issuer, --service-account-key-file, and --service-account-signing-key-file. If they are missing, you need to add them to the API server configuration and restart the API server. For managed clusters (EKS, GKE, AKS), this is handled by the provider.

Step 2: Create a new Service Account

Suppose your legacy Service Account is named legacy-app-sa. Create a new one with a clear name:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: new-app-sa
  namespace: my-namespace

Apply it:

kubectl apply -f new-sa.yaml

At this point, the new Service Account has no extra secrets and no RBAC bindings. It is harmless.

Step 3: Assign the same RBAC permissions

If legacy-app-sa had a RoleBinding to a Role named app-role, create an equivalent binding for new-app-sa:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: new-app-sa-binding
  namespace: my-namespace
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: app-role
subjects:
- kind: ServiceAccount
  name: new-app-sa
  namespace: my-namespace

Apply and verify:

kubectl apply -f new-binding.yaml
kubectl auth can-i list pods --as=system:serviceaccount:my-namespace:new-app-sa -n my-namespace

The last command should return yes if the role grants the permission.

Step 4: Switch a test workload

Modify a deployment to use the new Service Account:

spec:
  template:
    spec:
      serviceAccountName: new-app-sa
      automountServiceAccountToken: true

Apply the change and watch the rollout:

kubectl apply -f deployment.yaml
kubectl rollout status deployment/my-app -n my-namespace

Expected output:

Waiting for deployment "my-app" rollout to finish: 1 old replicas are pending termination...
deployment "my-app" successfully rolled out

Now inspect the pod to confirm it has a projected token:

kubectl get pod -n my-namespace -l app=my-app -o jsonpath='{.spec.volumes[?(@.projected.sources[0].serviceAccountToken)].projected.sources[0].serviceAccountToken.expirationSeconds}'

If the value is 3600 (or any number), the token is projected and will expire after that many seconds.

Step 5: Remove legacy tokens

After all workloads are migrated, remove the legacy token secrets from the old Service Account. First, delete the secret references from the old Service Account (or just delete the old Service Account if no longer needed):

kubectl delete serviceaccount legacy-app-sa -n my-namespace

But before deleting, ensure no pods are using it:

kubectl get pods --all-namespaces -o json | jq '.items[] | select(.spec.serviceAccountName == "legacy-app-sa")'

If empty, safe to delete. If not, update those workloads first.

A note on rollback: To roll back a single deployment, simply change serviceAccountName back to legacy-app-sa and apply again. If you have not yet deleted the old Service Account, this works immediately. That is why we migrate gradually instead of deleting everything at once.

Verification and Diagnostics

After making changes, always verify that the new Service Account works as expected and that no lingering issues exist. Diagnostics should cover token presence, token expiry, API access, and application behavior.

Check projected token availability

Inside a pod using the new Service Account, the token is available at /var/run/secrets/kubernetes.io/serviceaccount/token. Exec into the pod and view its expiration:

kubectl exec -it my-app-pod -n my-namespace -- cat /var/run/secrets/kubernetes.io/serviceaccount/token

This prints the JWT. Decode it (using a tool like jq or an online decoder) to see the exp claim. Alternatively, use kubectl to describe the pod and look for the mounted volume:

kubectl describe pod my-app-pod -n my-namespace

Look for a projected volume with a serviceAccountToken source. The Expiration field should be set.

Validate API access

Use kubectl auth can-i to test permissions:

kubectl auth can-i get pods --as=system:serviceaccount:my-namespace:new-app-sa -n my-namespace

If the output is yes, the Service Account has the necessary permission. If no, review the RBAC binding.

Monitor application logs

Check if your application logs any authentication errors:

kubectl logs my-app-pod -n my-namespace --tail=50

Look for messages like forbidden, token expired, or unauthorized. If the token expires and the app does not reload it, you will see failures after the expiration period (default 1 hour). To test, you can temporarily set a very short expiration in the pod spec (e.g., expirationSeconds: 60) and observe the app's behavior. If it fails, add logic to reload the token.

Diagnose token issues

If a pod fails to start with the new Service Account, check events:

kubectl get events -n my-namespace --sort-by=.lastTimestamp

Look for messages like MountVolume.SetUp failed for volume "kube-api-access-xyz" : failed to fetch token: serviceaccount "new-app-sa" not found. This would indicate a typo in the Service Account name.

Common issue: The Service Account exists but the pod cannot mount the token because automountServiceAccountToken is set to false at the Service Account level. Check:

kubectl get serviceaccount new-app-sa -n my-namespace -o yaml | grep automountServiceAccountToken

If it says false, either set it to true or explicitly set automountServiceAccountToken: true in the pod spec.

Failure Modes and Recovery

Even with careful planning, things can go wrong. Here are common failure modes during Service Account upgrades and how to recover from them.

Failure 1: Application breaks after switching Service Account

Symptoms: Pods restart with errors, application logs show forbidden or unable to authenticate.

Cause: RBAC permissions not correctly assigned to the new Service Account, or the application is trying to access a resource that was allowed before but is now denied.

Recovery:

1. Check permissions: kubectl auth can-i --list --as=system:serviceaccount:my-namespace:new-app-sa -n my-namespace
2. Compare with permissions of old Service Account: kubectl auth can-i --list --as=system:serviceaccount:my-namespace:legacy-app-sa -n my-namespace
3. Fix missing RoleBindings and apply.
4. If immediate rollback is needed, change deployment back to legacy-app-sa. This should take effect within a rollout.

Failure 2: Service Account token expired in long-running pod

Symptoms: After about 1 hour (default token lifetime), application starts getting 401 errors when calling the API.

Cause: The application caches the token and does not reload it. Projected tokens expire; legacy tokens did not.

Recovery:

• Update the application to watch the token file and reload on change, or use a client library that handles rotation automatically (e.g., official Kubernetes clients).
• As a temporary measure, you can extend the token expiration by setting expirationSeconds in the pod spec to a larger value (up to 604800 seconds, 7 days). But fix the root cause.
• Rollback: switch back to legacy Service Account if available, but note that this is not a long-term solution.

Failure 3: Service Account not found

Symptoms: Pod fails to start with MountVolume.SetUp failed.

Cause: Typo in serviceAccountName or the Service Account was deleted prematurely.

Recovery:

• Correct the name or recreate the Service Account and its bindings.
• If the old Service Account was deleted and you need to restore it from a backup, use kubectl apply from your saved manifests or recreate manually.

Failure 4: Legacy token still works but you cannot delete it

Symptoms: You try to delete a legacy token secret, but it immediately gets recreated by the Service Account controller.

Cause: The Service Account still references the secret in its secrets field. The controller re-adds it.

Recovery:

• Remove the secret reference from the Service Account first:

  kubectl patch serviceaccount my-sa -n my-namespace -p '{"secrets": []}'

• Then delete the secret:

  kubectl delete secret my-sa-token-abcde -n my-namespace

Failure 5: Managed cluster does not allow changing Service Account issuer

Symptoms: In managed Kubernetes (EKS, GKE, AKS), trying to customize the issuer results in error.

Cause: The cloud provider manages the API server and does not expose those flags.

Recovery:

• Use the managed cluster's default issuer and token configuration. You can still migrate to projected tokens if supported.
• For specialized needs, consider using a separate service account token via a ServiceAccount token secret (still legacy) or use an external identity provider with workload identity federation.

General recovery steps:

1. Keep a backup of all Service Accounts and RBAC bindings: kubectl get serviceaccounts,roles,rolebindings,clusterroles,clusterrolebindings --all-namespaces -o yaml > backup.yaml
2. Restore from backup: kubectl apply -f backup.yaml
3. Test rollback procedure in a staging environment before production.

Operations Checklist

Use this checklist to ensure a smooth Service Account upgrade and migration. Assign a single owner for each major decision and revisit periodically.

• Inventory: Owner: Platform Engineer (e.g., Priya Shah). Frequency: Before any change and quarterly. Verify Kubernetes version, list Service Accounts, find legacy tokens, map RBAC.
• Enable Projected Tokens: Owner: Cluster Admin (e.g., Marcus Lee). Frequency: Once during upgrade. Confirm API server flags, if missing coordinate with provider.
• Create New Service Accounts: Owner: Application Team Lead (e.g., Ana Rodriguez). Frequency: Per migration. For each legacy SA, create a new SA with a clear naming convention.
• Assign RBAC: Owner: Security Engineer (e.g., Tom Chen). Frequency: Per migration. Replicate permissions using least privilege. Review with kubectl auth can-i --list.
• Test Migration: Owner: QA Lead (e.g., Sarah Kim). Frequency: Per migration. Switch a canary deployment to the new SA, monitor logs, validate token expiration.
• Full Rollout: Owner: Release Manager (e.g., David Nguyen). Frequency: Per application. Update all deployments, perform rolling updates, monitor for failures.
• Remove Legacy Tokens: Owner: Security Engineer (Tom Chen). Frequency: After all workloads migrated and verified. Remove secret references, delete secrets, then delete old SAs if unused.
• Post-Migration Review: Owner: Platform Architect (e.g., Emily Johnson). Frequency: Within one week after each major migration. Check for token errors, review RBAC creep, document lessons learned.
• Quarterly Access Audit: Owner: Security Engineer (Tom Chen). Frequency: Quarterly. Review all Service Account permissions, remove unused SAs, ensure token policies are enforced.

Common Pitfalls and How to Avoid Them

Here are pitfalls that many teams encounter during Service Account migrations, along with explanations and prevention strategies.

Pitfall 1: Deleting legacy Service Accounts too early

Why it happens: Teams want to clean up quickly after migration. Consequences: Any missed pod referencing the old SA will fail to start. How to avoid: Before deleting, search for all pods using the SA:

kubectl get pods --all-namespaces -o json | jq '.items[] | select(.spec.serviceAccountName == "legacy-app-sa")'

If any appear, migrate them first. Keep the old SA for at least one full release cycle.

Pitfall 2: Not updating token refresh logic in applications

Why it happens: Developers assume tokens are static because they always were. Consequences: Applications break after token expiry, often in production at odd hours. How to avoid: Test with a short token expiration (e.g., 60 seconds) in staging. Ensure the client library reloads the token. Document this requirement for all apps.

Pitfall 3: Overly permissive RBAC on new Service Accounts

Why it happens: Copying old roles that had accumulated excessive permissions. Consequences: Security risk; the new SA may have more access than needed. How to avoid: Start with least privilege. Use kubectl auth can-i --list to see actual effective permissions and trim as needed. Involve security review.

Pitfall 4: Ignoring Service Account secrets in CI/CD pipelines

Why it happens: Pipelines use hardcoded token secrets that are not part of the pod spec. Consequences: Secret rotation breaks CI/CD. How to avoid: Inventory all external consumers of Service Account tokens, including CI systems, monitoring tools, and external scripts. Migrate them to use projected tokens via Kubernetes API or use workload identity where possible.

Pitfall 5: Not monitoring token expiration in long-running pods

Why it happens: Tokens expire silently; teams only notice when things fail. Consequences: Intermittent failures that are hard to debug. How to avoid: Set up monitoring alerts for pods with projected tokens that are nearing expiration. Use a controller or DaemonSet to check token age. Or extend expiration to 7 days as a stopgap.

Conclusion

Upgrading and migrating Kubernetes Service Accounts is a multi-step process that requires inventory, planning, gradual rollout, and verification. By following the operational safety principles of observing before changing, limiting blast radius, securing secrets, and verifying each change, you can avoid common failures.

As a next step, pick one legacy Service Account in your cluster, run the inventory commands from Section 1, and create a migration plan using the Safe Configuration Path. Assign an owner and set a review date. Then execute the migration on a canary deployment, verify token expiry and permissions, and use the Operations Checklist to track progress.

A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. With this guide, you have the commands and procedures to perform Service Account upgrades confidently.