## Intro

Kubernetes Service Accounts are the identities that pods use to authenticate to the Kubernetes API and to external services. As clusters grow and security requirements mature, administrators often face upgrades such as moving from long-lived static tokens to projected, time-bound tokens, or migrating workloads to new Service Accounts with cleaner RBAC bindings. These tasks can disrupt applications if done without a clear plan.

This guide walks through the entire lifecycle: from initial version and environment inventory to safe configuration changes, verification, failure recovery, and an operational checklist. It is written for developers, DevOps consultants, and technical startup teams who need practical, command-focused procedures. While the examples focus on Service Accounts, the concepts also touch on RBAC and Secrets because they are tightly coupled with identity and access.

The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached. Each section includes concrete commands and expected outputs so you can follow along with confidence.

## Version and Environment Inventory

Before touching any Service Account, know exactly what you have. A thorough inventory includes the Kubernetes version, API server flags, existing Service Accounts, their token types, and RBAC bindings. This prevents surprises like making a change that is incompatible with an older API version.

Start by checking the cluster version:

```bash
kubectl version --short
```

Expected output (example):
```
Client Version: v1.28.2
Server Version: v1.27.6
```

If your cluster is v1.24 or later, projected Service Account tokens (the `BoundServiceAccountTokenVolume` feature) are stable and should be your default. For earlier versions, you may still be relying on legacy static tokens. Knowing this difference is critical for an upgrade.

List existing Service Accounts in all namespaces:

```bash
kubectl get serviceaccounts --all-namespaces
```

Look for accounts that have been around a long time, especially in `kube-system` or custom namespaces. Then inspect a specific Service Account to see its secrets and associated tokens:

```bash
kubectl get serviceaccount my-app-sa -n my-namespace -o yaml
```

In the output, note the `secrets` field. If it lists multiple secrets, those are likely legacy token secrets. In newer clusters, this field is often empty because tokens are projected and not stored as secrets.

To determine whether the Service Account can issue short-lived tokens via the TokenRequest API, run:

```bash
kubectl create token my-app-sa -n my-namespace
```

This command requests a fresh, time-bound token and prints it to stdout. You can decode the JWT (e.g., using `jq -R 'split(".") | .[1] | @base64d | fromjson'`) to inspect the `exp` claim and confirm it expires. Legacy static tokens are not created this way; they are stored as Secrets and never expire. If you find any legacy token Secrets in the ServiceAccount's `secrets` field, plan a migration to short-lived tokens.

Finally, review RBAC bindings for the Service Account:

```bash
kubectl get rolebindings,clusterrolebindings --all-namespaces -o json | jq '.items[] | select(.subjects[]?.name == "my-app-sa")'
```

This shows every role and cluster role assigned to the account. Document the permissions, because you will need to reproduce them securely after migration.

A quick environment inventory checklist:
- Kubernetes server version: `kubectl version --short`
- API server token settings: check `kube-apiserver` pod arguments for `--service-account-issuer`, `--service-account-key-file`, and `--service-account-signing-key-file`
- List all Service Accounts and their namespaces
- Identify legacy token secrets
- Map RBAC bindings for each Service Account in use

## Safe Configuration Path

Once you understand your current state, you can plan a safe upgrade. The safest path is to create new projected token Service Accounts alongside the old ones, switch workloads gradually, and remove legacy tokens only after verification. This limits the blast radius and gives you a clear rollback point.

### Step 1: Enable projected tokens (if not already enabled)

For clusters v1.21+, the feature is on by default. Verify by checking the `kube-apiserver` pod's spec:

```bash
kubectl get pod -n kube-system -l component=kube-apiserver -o yaml | grep -E 'service-account-(issuer|key-file|signing-key-file)'
```

You should see values for `--service-account-issuer`, `--service-account-key-file`, and `--service-account-signing-key-file`. If they are missing, you need to add them to the API server configuration and restart the API server. For managed clusters (EKS, GKE, AKS), this is handled by the provider.

### Step 2: Create a new Service Account

Suppose your legacy Service Account is named `legacy-app-sa`. Create a new one with a clear name:

```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: new-app-sa
  namespace: my-namespace
```

Apply it:

```bash
kubectl apply -f new-sa.yaml
```

At this point, the new Service Account has no extra secrets and no RBAC bindings. It is harmless.

### Step 3: Assign the same RBAC permissions

If `legacy-app-sa` had a RoleBinding to a Role named `app-role`, create an equivalent binding for `new-app-sa`:

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: new-app-sa-binding
  namespace: my-namespace
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: app-role
subjects:
- kind: ServiceAccount
  name: new-app-sa
  namespace: my-namespace
```

Apply and verify:

```bash
kubectl apply -f new-binding.yaml
kubectl auth can-i list pods --as=system:serviceaccount:my-namespace:new-app-sa -n my-namespace
```

The last command should return `yes` if the role grants the permission.

### Step 4: Switch a test workload

Modify a deployment to use the new Service Account:

```yaml
spec:
  template:
    spec:
      serviceAccountName: new-app-sa
      automountServiceAccountToken: true
```

Apply the change and watch the rollout:

```bash
kubectl apply -f deployment.yaml
kubectl rollout status deployment/my-app -n my-namespace
```

Expected output:
```
Waiting for deployment "my-app" rollout to finish: 1 old replicas are pending termination...
deployment "my-app" successfully rolled out
```

Now inspect the pod to confirm it has a projected token:

```bash
kubectl get pod -n my-namespace -l app=my-app -o jsonpath='{.spec.volumes[?(@.projected.sources[0].serviceAccountToken)].projected.sources[0].serviceAccountToken.expirationSeconds}'
```

If the value is `3600` (or any number), the token is projected and will expire after that many seconds.

### Step 5: Remove legacy tokens

After all workloads are migrated, remove the legacy token secrets from the old Service Account. First, delete the secret references from the old Service Account (or just delete the old Service Account if no longer needed):

```bash
kubectl delete serviceaccount legacy-app-sa -n my-namespace
```

But before deleting, ensure no pods are using it:

```bash
kubectl get pods --all-namespaces -o json | jq '.items[] | select(.spec.serviceAccountName == "legacy-app-sa")'
```

If empty, safe to delete. If not, update those workloads first.

A note on rollback: To roll back a single deployment, simply change `serviceAccountName` back to `legacy-app-sa` and apply again. If you have not yet deleted the old Service Account, this works immediately. That is why we migrate gradually instead of deleting everything at once.

## Verification and Diagnostics

After making changes, always verify that the new Service Account works as expected and that no lingering issues exist. Diagnostics should cover token presence, token expiry, API access, and application behavior.

### Check projected token availability

Inside a pod using the new Service Account, the token is available at `/var/run/secrets/kubernetes.io/serviceaccount/token`. Exec into the pod and view its expiration:

```bash
kubectl exec -it my-app-pod -n my-namespace -- cat /var/run/secrets/kubernetes.io/serviceaccount/token
```

This prints the JWT. Decode it (using a tool like `jq` or an online decoder) to see the `exp` claim. Alternatively, use `kubectl` to describe the pod and look for the mounted volume:

```bash
kubectl describe pod my-app-pod -n my-namespace
```

Look for a `projected` volume with a `serviceAccountToken` source. The `Expiration` field should be set.

### Validate API access

Use `kubectl auth can-i` to test permissions:

```bash
kubectl auth can-i get pods --as=system:serviceaccount:my-namespace:new-app-sa -n my-namespace
```

If the output is `yes`, the Service Account has the necessary permission. If `no`, review the RBAC binding.

### Monitor application logs

Check if your application logs any authentication errors:

```bash
kubectl logs my-app-pod -n my-namespace --tail=50
```

Look for messages like `forbidden`, `token expired`, or `unauthorized`. If the token expires and the app does not reload it, you will see failures after the expiration period (default 1 hour). To test, you can temporarily set a very short expiration in the pod spec (e.g., `expirationSeconds: 60`) and observe the app's behavior. If it fails, add logic to reload the token.

### Diagnose token issues

If a pod fails to start with the new Service Account, check events:

```bash
kubectl get events -n my-namespace --sort-by=.lastTimestamp
```

Look for messages like `MountVolume.SetUp failed for volume "kube-api-access-xyz" : failed to fetch token: serviceaccount "new-app-sa" not found`. This would indicate a typo in the Service Account name.

Common issue: The Service Account exists but the pod cannot mount the token because `automountServiceAccountToken` is set to `false` at the Service Account level. Check:

```bash
kubectl get serviceaccount new-app-sa -n my-namespace -o yaml | grep automountServiceAccountToken
```

If it says `false`, either set it to `true` or explicitly set `automountServiceAccountToken: true` in the pod spec.

## Failure Modes and Recovery

Even with careful planning, things can go wrong. Here are common failure modes during Service Account upgrades and how to recover from them.

### Failure 1: Application breaks after switching Service Account

**Symptoms:** Pods restart with errors, application logs show `forbidden` or `unable to authenticate`.

**Cause:** RBAC permissions not correctly assigned to the new Service Account, or the application is trying to access a resource that was allowed before but is now denied.

**Recovery:**
1. Check permissions: `kubectl auth can-i --list --as=system:serviceaccount:my-namespace:new-app-sa -n my-namespace`
2. Compare with permissions of old Service Account: `kubectl auth can-i --list --as=system:serviceaccount:my-namespace:legacy-app-sa -n my-namespace`
3. Fix missing RoleBindings and apply.
4. If immediate rollback is needed, change deployment back to `legacy-app-sa`. This should take effect within a rollout.

### Failure 2: Service Account token expired in long-running pod

**Symptoms:** After about 1 hour (default token lifetime), application starts getting 401 errors when calling the API.

**Cause:** The application caches the token and does not reload it. Projected tokens expire; legacy tokens did not.

**Recovery:**
- Update the application to watch the token file and reload on change, or use a client library that handles rotation automatically (e.g., official Kubernetes clients).
- As a temporary measure, you can extend the token expiration by setting `expirationSeconds` in the pod spec to a larger value (up to 604800 seconds, 7 days). But fix the root cause.
- Rollback: switch back to legacy Service Account if available, but note that this is not a long-term solution.

### Failure 3: Service Account not found

**Symptoms:** Pod fails to start with `MountVolume.SetUp failed`.

**Cause:** Typo in `serviceAccountName` or the Service Account was deleted prematurely.

**Recovery:**
- Correct the name or recreate the Service Account and its bindings.
- If the old Service Account was deleted and you need to restore it from a backup, use `kubectl apply` from your saved manifests or recreate manually.

### Failure 4: Legacy token still works but you cannot delete it

**Symptoms:** You try to delete a legacy token secret, but it immediately gets recreated by the Service Account controller.

**Cause:** The Service Account still references the secret in its `secrets` field. The controller re-adds it.

**Recovery:**
- Remove the secret reference from the Service Account first:
  ```bash
  kubectl patch serviceaccount my-sa -n my-namespace -p '{"secrets": []}'
  ```
- Then delete the secret:
  ```bash
  kubectl delete secret my-sa-token-abcde -n my-namespace
  ```

### Failure 5: Managed cluster does not allow changing Service Account issuer

**Symptoms:** In managed Kubernetes (EKS, GKE, AKS), trying to customize the issuer results in error.

**Cause:** The cloud provider manages the API server and does not expose those flags.

**Recovery:**
- Use the managed cluster's default issuer and token configuration. You can still migrate to projected tokens if supported.
- For specialized needs, consider using a separate service account token via a ServiceAccount token secret (still legacy) or use an external identity provider with workload identity federation.

General recovery steps:
1. Keep a backup of all Service Accounts and RBAC bindings: `kubectl get serviceaccounts,roles,rolebindings,clusterroles,clusterrolebindings --all-namespaces -o yaml > backup.yaml`
2. Restore from backup: `kubectl apply -f backup.yaml`
3. Test rollback procedure in a staging environment before production.

## Operations Checklist

Use this checklist to ensure a smooth Service Account upgrade and migration. Assign a single owner for each major decision and revisit periodically.

- **Inventory:** Owner: Platform Engineer (e.g., Priya Shah). Frequency: Before any change and quarterly. Verify Kubernetes version, list Service Accounts, find legacy tokens, map RBAC.
- **Enable Projected Tokens:** Owner: Cluster Admin (e.g., Marcus Lee). Frequency: Once during upgrade. Confirm API server flags, if missing coordinate with provider.
- **Create New Service Accounts:** Owner: Application Team Lead (e.g., Ana Rodriguez). Frequency: Per migration. For each legacy SA, create a new SA with a clear naming convention.
- **Assign RBAC:** Owner: Security Engineer (e.g., Tom Chen). Frequency: Per migration. Replicate permissions using least privilege. Review with `kubectl auth can-i --list`.
- **Test Migration:** Owner: QA Lead (e.g., Sarah Kim). Frequency: Per migration. Switch a canary deployment to the new SA, monitor logs, validate token expiration.
- **Full Rollout:** Owner: Release Manager (e.g., David Nguyen). Frequency: Per application. Update all deployments, perform rolling updates, monitor for failures.
- **Remove Legacy Tokens:** Owner: Security Engineer (Tom Chen). Frequency: After all workloads migrated and verified. Remove secret references, delete secrets, then delete old SAs if unused.
- **Post-Migration Review:** Owner: Platform Architect (e.g., Emily Johnson). Frequency: Within one week after each major migration. Check for token errors, review RBAC creep, document lessons learned.
- **Quarterly Access Audit:** Owner: Security Engineer (Tom Chen). Frequency: Quarterly. Review all Service Account permissions, remove unused SAs, ensure token policies are enforced.

## Common Pitfalls and How to Avoid Them

Here are pitfalls that many teams encounter during Service Account migrations, along with explanations and prevention strategies.

### Pitfall 1: Deleting legacy Service Accounts too early

**Why it happens:** Teams want to clean up quickly after migration.
**Consequences:** Any missed pod referencing the old SA will fail to start.
**How to avoid:** Before deleting, search for all pods using the SA:
```bash
kubectl get pods --all-namespaces -o json | jq '.items[] | select(.spec.serviceAccountName == "legacy-app-sa")'
```
If any appear, migrate them first. Keep the old SA for at least one full release cycle.

### Pitfall 2: Not updating token refresh logic in applications

**Why it happens:** Developers assume tokens are static because they always were.
**Consequences:** Applications break after token expiry, often in production at odd hours.
**How to avoid:** Test with a short token expiration (e.g., 60 seconds) in staging. Ensure the client library reloads the token. Document this requirement for all apps.

### Pitfall 3: Overly permissive RBAC on new Service Accounts

**Why it happens:** Copying old roles that had accumulated excessive permissions.
**Consequences:** Security risk; the new SA may have more access than needed.
**How to avoid:** Start with least privilege. Use `kubectl auth can-i --list` to see actual effective permissions and trim as needed. Involve security review.

### Pitfall 4: Ignoring Service Account secrets in CI/CD pipelines

**Why it happens:** Pipelines use hardcoded token secrets that are not part of the pod spec.
**Consequences:** Secret rotation breaks CI/CD.
**How to avoid:** Inventory all external consumers of Service Account tokens, including CI systems, monitoring tools, and external scripts. Migrate them to use projected tokens via Kubernetes API or use workload identity where possible.

### Pitfall 5: Not monitoring token expiration in long-running pods

**Why it happens:** Tokens expire silently; teams only notice when things fail.
**Consequences:** Intermittent failures that are hard to debug.
**How to avoid:** Set up monitoring alerts for pods with projected tokens that are nearing expiration. Use a controller or DaemonSet to check token age. Or extend expiration to 7 days as a stopgap.

## Conclusion

Upgrading and migrating Kubernetes Service Accounts is a multi-step process that requires inventory, planning, gradual rollout, and verification. By following the operational safety principles of observing before changing, limiting blast radius, securing secrets, and verifying each change, you can avoid common failures.

As a next step, pick one legacy Service Account in your cluster, run the inventory commands from Section 1, and create a migration plan using the Safe Configuration Path. Assign an owner and set a review date. Then execute the migration on a canary deployment, verify token expiry and permissions, and use the Operations Checklist to track progress.

A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. With this guide, you have the commands and procedures to perform Service Account upgrades confidently.