Kubernetes Service Accounts Admin Production Operations Checklist with Practical Examples

Intro

Service accounts in Kubernetes are the identities that Pods use to interact with the Kubernetes API and other cluster services. When a service account is misconfigured, workloads can fail with authorization errors, tokens can expire, or overly broad permissions can become a security incident. This article provides a production operations checklist for administering Kubernetes service accounts, with practical examples that move from an observed problem to a verified result.

This guide is written for developers, DevOps consultants, and technical startup teams who manage Kubernetes clusters in production. It connects service account operations, checklists, best practices, and maintenance to concrete commands, expected output, failure signals, and recovery decisions. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached.

Throughout this article, we use Kubernetes v1.28 as the reference version. However, the concepts and commands apply to most recent releases. Always check the Kubernetes release notes for your specific version for breaking changes.

Version and Environment Inventory

Before touching any service account, you must know your environment. Begin with read-only observations to capture the current state, versions, and topology. This reduces the risk of applying changes that are incompatible with older APIs or features.

Start by confirming the cluster version and API availability:

kubectl version --short
# Example output:
# Client Version: v1.28.2
# Server Version: v1.28.4

Check that the service account API is available and list existing service accounts in the target namespace:

kubectl api-resources | grep serviceaccounts
# Expected output:
# serviceaccounts                   sa           v1                   true         ServiceAccount

kubectl get serviceaccounts -n production
# Example output:
# NAME               SECRETS   AGE
# default            0         30d
# app-sa             1         3d
# monitoring-sa      1         5d

Note the SECRETS column: in Kubernetes v1.24+, the LegacyServiceAccountTokenNoAutoGeneration feature gate is enabled, so service accounts no longer automatically get a long-lived secret token. If a service account shows 0 secrets, it may rely on projected tokens or manual token creation. This is important for troubleshooting.

Capture the exact Pods using each service account:

kubectl get pods -n production -o json | jq -r '.items[] | "\(.metadata.name) -> \(.spec.serviceAccountName // "default")"'
# Example output:
# web-frontend-6d4b8c9f7-abcde -> app-sa
# web-frontend-6d4b8c9f7-fghij -> app-sa
# monitoring-agent-2f8d7c6b5-klmno -> monitoring-sa
# backend-worker-7c9b8d6e5-pqrst -> default

This inventory helps you understand which workloads depend on which identity. Before making changes, confirm that any replacement or modification will not break these associations.

Also record any external dependencies, such as cloud IAM roles or external secret stores that integrate with the service account. For example, if using AWS IAM Roles for Service Accounts (IRSA), list the IAM role ARNs mapped to each service account annotation:

kubectl get serviceaccount app-sa -n production -o yaml | grep -A1 annotations
# Example output:
#   annotations:
#     eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/prod-app-role

Documenting these associations is essential for recovery and audit.

Safe Configuration Path

When modifying service accounts, always follow a safe configuration path: use read-only checks, create or patch with explicit manifests, verify the resulting state, and have a rollback plan. Never edit service accounts in place without recording the original state.

Creating a Service Account with Least Privilege

Define a service account that includes only the necessary annotations and is not bound to any token secrets by default:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: app-sa
  namespace: production
  # Annotations for cloud IAM integration if needed
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/prod-app-role
automountServiceAccountToken: true  # set to false if the Pod does not need API access

Apply it:

kubectl apply -f sa-app.yaml

Verify the service account exists and has the correct fields:

kubectl get serviceaccount app-sa -n production -o yaml
# Check that the annotations and automountServiceAccountToken are as expected.

Binding to Roles with RBAC

Service accounts gain permissions through RoleBindings or ClusterRoleBindings. Use the least privilege principle: grant only what the workload needs.

First, inspect existing RBAC bindings for the service account:

kubectl get rolebindings,clusterrolebindings -n production -o json | jq -r '.items[] | select(.subjects[]?.name == "app-sa") | "\(.kind): \(.metadata.name) -> \(.roleRef.name)"'
# Example output:
# RoleBinding: app-sa-view -> view

Create a Role with specific permissions, for example read access to Pods:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-reader
  namespace: production
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch"]

Create a RoleBinding to link the service account to the role:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: app-sa-pod-reader
  namespace: production
subjects:
- kind: ServiceAccount
  name: app-sa
  namespace: production
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

Apply and verify:

kubectl apply -f role.yaml -f rolebinding.yaml
kubectl auth can-i list pods --as=system:serviceaccount:production:app-sa -n production
# Expected: yes
kubectl auth can-i delete pods --as=system:serviceaccount:production:app-sa -n production
# Expected: no

This confirms the permissions are correctly scoped.

Managing Token Mounts

By default, a Pod mounts a service account token at /var/run/secrets/kubernetes.io/serviceaccount. If a Pod does not need to call the Kubernetes API, disable this to reduce attack surface:

apiVersion: v1
kind: Pod
metadata:
  name: static-web
spec:
  serviceAccountName: default
  automountServiceAccountToken: false

Verify that no token is mounted:

kubectl exec -it static-web -n production -- ls /var/run/secrets/kubernetes.io/serviceaccount
# Expected: No such file or directory

Verification and Diagnostics

After any change, you must verify that the service account behaves as expected and that workloads can authenticate and authorize correctly.

Check Service Account Token Projection

Modern clusters use projected service account tokens. Inspect the token expiration and audience:

kubectl create token app-sa -n production --duration=1h --audience=api
# Example output:
# eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9... (token string)

This command generates a token you can test with. Note: this token is time-limited and not persisted. For automated workflows, use TokenRequest API or client libraries.

Test API Access from a Pod

Create a debugging Pod using the service account and attempt an authorized action:

kubectl run test-pod --image=alpine --restart=Never --rm -it -n production --overrides='
{
  "spec": {
    "serviceAccountName": "app-sa"
  }
}' -- sh

Inside the Pod, check the mounted token and call the API:

# Inside the pod
TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
APISERVER=https://kubernetes.default.svc
curl -sk -H "Authorization: Bearer $TOKEN" $APISERVER/api/v1/namespaces/production/pods

# Expected: JSON list of pods if the RBAC allows. If forbidden, you'll see a 403 error.

Exit and delete the test Pod automatically when done.

Audit Logs for Service Account Actions

Enable Kubernetes audit logging to track service account usage. For managed clusters, this may be available via the provider's console. For self-managed, configure audit-policy.yaml to log service account token creations and other actions:

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: Metadata
  users: ["system:serviceaccount:production:app-sa"]

Then review logs for anomalies:

grep "app-sa" /var/log/kubernetes/audit.log | tail -20

Validate ServiceAccountName in Deployments

Ensure that Deployments reference existing service accounts. A missing service account causes Pods to fail scheduling or runtime. Check with:

kubectl get deployment web-frontend -n production -o jsonpath='{.spec.template.spec.serviceAccountName}'
# Expected: app-sa

If empty, it defaults to default. If the name does not exist, Pods will not start and will log an error like:

Error creating: pods "web-frontend-..." is forbidden: error looking up service account production/non-existent-sa: serviceaccount "non-existent-sa" not found

Failure Modes and Recovery

Service account issues can cause various failures. Here are common failure modes and how to recover.

Failure: Pod Fails with "serviceaccount not found"

Symptom: Pod stays in Pending or ContainerCreating state, events show:

Warning  FailedCreate  3s (x10 over 1m)  replicaset-controller  Error creating: pods "web-frontend-..." is forbidden: error looking up service account production/app-sa: serviceaccount "app-sa" not found

Diagnosis: The referenced service account does not exist in the namespace.

Recovery: Create the missing service account (or correct the reference).

kubectl create serviceaccount app-sa -n production
# Verify
kubectl get serviceaccount app-sa -n production

If the service account was deleted accidentally, recover from backup or reapply the manifest.

Failure: Forbidden API Access (403)

Symptom: Application logs show forbidden: User "system:serviceaccount:production:app-sa" cannot list resource "pods"...

Diagnosis: RBAC permissions are insufficient.

Recovery: Review the Role/RoleBinding. Add necessary permissions with least privilege. Use kubectl auth can-i to test:

kubectl auth can-i list pods --as=system:serviceaccount:production:app-sa -n production
# If no, inspect rolebindings
kubectl get rolebinding -n production -o yaml | grep -A5 app-sa

Apply the missing role binding and retest.

Failure: Token Expiration or Invalid Token

Symptom: Application receives 401 Unauthorized from API server, logs show token validation errors.

Diagnosis: The mounted token may be expired (for projected tokens with short lifetimes) or the service account token secret may be missing.

Recovery:

• For projected tokens, ensure the Pod restarts to get a new token, or adjust the TokenRequest if using a custom setup.
• For legacy secret tokens, recreate the secret and update the service account if needed. However, note that legacy tokens are deprecated. Prefer projected tokens.
• Check the token's expiration using kubectl create token to test a fresh token and compare.

kubectl create token app-sa -n production --duration=1h
# Use this token in a test to see if it works.

Failure: Service Account Automated Using Cloud IAM (e.g., AWS IRSA)

Symptom: Pod fails to assume IAM role, logs show WebIdentityErr: failed to retrieve credentials or similar.

Diagnosis: The service account annotation may be incorrect, or the OIDC provider trust is misconfigured.

Recovery:

• Verify the annotation: kubectl get serviceaccount app-sa -n production -o yaml | grep role-arn
• Check the IAM role trust policy to include the cluster's OIDC provider and the service account principal.
• Confirm the OIDC provider is set up on the cluster: kubectl describe cluster usually shows it.

For detailed steps, refer to cloud provider documentation.

Operations Checklist

Use the following checklist as a reference for service account administration in production. Each item includes a command or verification step.

• [ ] Inventory service accounts and bindings monthly

  kubectl get serviceaccounts --all-namespaces
  kubectl get rolebindings,clusterrolebindings --all-namespaces -o json | jq '.items[] | select(.subjects[]?.kind == "ServiceAccount") | {namespace: .metadata.namespace, name: .metadata.name, role: .roleRef.name}'

Look for unused accounts or overly broad bindings.

For each service account, check automountServiceAccountToken and secrets fields:

• [ ] Review token creation settings

  kubectl get serviceaccount -o json | jq '.items[] | {name: .metadata.name, automount: .automountServiceAccountToken, secrets: [.secrets[].name]}'

Ensure only necessary accounts have token mounted.

Before applying to production, run:

• [ ] Test RBAC permissions in a staging environment

  kubectl auth can-i --list --as=system:serviceaccount:namespace:sa

Verify that the permissions match the application requirements.

If you use long-lived secrets, rotate them and update deployments:

• [ ] Rotate manually created tokens regularly

  kubectl create token mysa --duration=... # or create a new secret and patch the service account

Prefer short-lived projected tokens or external identity providers.

Set up alerts for:

Use tools like Prometheus and Alertmanager with custom queries.

• [ ] Monitor service account-related events and metrics
• Failed pod creations due to missing service accounts (reason=FailedCreate with message containing serviceaccount ... not found)
• Increases in 401/403 responses from the API server with service account names.

For each failure mode in this article, ensure you have a runbook and that team members have practiced recovery in a non-production environment.

• [ ] Document and test recovery procedures

Store service account definitions and RBAC roles in a Git repository. Use GitOps or CI/CD to apply changes, ensuring review and auditability.

• [ ] Keep service account YAML manifests in version control

Run a pre-deployment check:

• [ ] Validate service account references in workload manifests

  # Example: iterate over deployments and check serviceaccount existence
  for ns in $(kubectl get ns -o jsonpath='{.items[*].metadata.name}'); do
    for deploy in $(kubectl get deploy -n $ns -o jsonpath='{.items[*].metadata.name}'); do
      sa=$(kubectl get deploy $deploy -n $ns -o jsonpath='{.spec.template.spec.serviceAccountName}')
      if ! kubectl get sa $sa -n $ns > /dev/null 2>&1; then
        echo "Missing SA $sa for deploy $deploy in $ns"
      fi
    done
  done

Integrate these checks into your CI pipeline and regular operations review.

Conclusion

Administering Kubernetes Service Accounts in production requires a disciplined approach: observe state, change minimally, verify outcomes, and prepare for recovery. The checklist provided in this article gives you a practical foundation for managing service accounts safely and effectively.

Start by applying one low-risk verification from this guide: record the current state of a service account, run the documented check, compare the result with the expected signal, and review dependencies such as RBAC, Secrets, and cloud IAM integration.

A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. Integrate these practices into your team's routine to keep your Kubernetes clusters secure and stable.