Kubernetes Service Account Production Operations Checklist with Practical Examples

Intro

Service Accounts are the identity mechanism for workloads running inside Kubernetes. They authenticate pods to the Kubernetes API and to external systems, and their configuration directly affects security posture, auditability, and operational reliability. Yet many teams treat Service Accounts as a one-time setup item and only revisit them during incidents.

This article provides a production operations checklist for Kubernetes Service Accounts, with practical examples that connect each step to commands, expected output, failure signals, and recovery decisions. It is written for developers, DevOps consultants, and technical startup teams who manage Kubernetes clusters in production and need a repeatable, low-risk workflow.

The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached. Each section follows the same structure: a read-only observation, the smallest justified change, and a verification command. All examples are version-scoped and avoid exposing sensitive values.

Version and Environment Inventory

Before touching any Service Account configuration, document the installed Kubernetes version, the deployment topology, and the exact component under inspection. This prevents commands from being run against the wrong cluster or an unsupported API version.

Start with a read-only inventory.

kubectl version
kubectl get nodes -o wide
kubectl get serviceaccounts --all-namespaces

Expected output includes the client and server versions, node status and roles, and a list of Service Accounts in each namespace. Note the API group for Service Accounts: v1 in Kubernetes 1.24 and later, v1beta1 in older versions. If you use kubectl describe serviceaccount <name>, the output shows the associated secrets only up to Kubernetes 1.23; from 1.24 onward, Service Account tokens are managed separately via the TokenRequest API.

A common mistake here is to assume all namespaces have the same Service Account defaults. Verify the default Service Account in each namespace and its automount behavior.

kubectl get serviceaccount default -n <namespace> -o yaml

Look for automountServiceAccountToken: false. If true, every pod in that namespace receives a token even if it does not need one, increasing the attack surface.

When planning changes, limit the blast radius. Apply one manifest at a time and inspect the generated resources. For example, to disable automounting in a specific namespace, create a patch that targets only the default Service Account in that namespace.

kubectl patch serviceaccount default -n <namespace> -p '{"automountServiceAccountToken": false}'

Verify with:

kubectl get serviceaccount default -n <namespace> -o jsonpath='{.automountServiceAccountToken}'

Expected output is false. If it returns true or no value, the patch did not apply; check your RBAC permissions and the namespace name.

Safe Configuration Path

The safe configuration path for Service Accounts involves three layers: the Service Account itself, the RBAC policy that grants it permissions, and the workload that uses it. Each layer has its own observation and modification steps.

Layer 1: Service Account Definition

Observe the current definition.

kubectl get serviceaccount <sa-name> -n <namespace> -o yaml

Look for annotations and labels that may affect behavior. For example, the eks.amazonaws.com/role-arn annotation is used on AWS EKS to associate an IAM role with the Service Account. If you are not on EKS, this annotation is irrelevant.

A minimal change example: add a label for ownership tracking.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: payment-sa
  namespace: payments
  labels:
    team: checkout
    app: payment-processor

Apply it and verify.

kubectl apply -f sa.yaml
kubectl get serviceaccount payment-sa -n payments --show-labels

Expected output includes team=checkout,app=payment-processor.

Layer 2: RBAC Policy

RBAC ties the Service Account to permissions via Roles and RoleBindings (namespaced) or ClusterRoles and ClusterRoleBindings (cluster-wide). Observe the current bindings.

kubectl get rolebinding,clusterrolebinding --all-namespaces | grep <sa-name>

Look for bindings that grant more permissions than needed. For example, a binding to the cluster-admin ClusterRole for a Service Account used by a web application is a serious misconfiguration.

Change example: create a least-privilege Role for a Service Account that only needs to read ConfigMaps in its namespace.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: config-reader
  namespace: payments
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: payment-sa-config-reader
  namespace: payments
subjects:
- kind: ServiceAccount
  name: payment-sa
  namespace: payments
roleRef:
  kind: Role
  name: config-reader
  apiGroup: rbac.authorization.k8s.io

Apply and verify.

kubectl apply -f rbac.yaml
kubectl auth can-i get configmaps --as=system:serviceaccount:payments:payment-sa -n payments

Expected output is yes. To check a denied action:

kubectl auth can-i create deployments --as=system:serviceaccount:payments:payment-sa -n payments

Expected output is no.

Layer 3: Workload Usage

Observe which pods use the Service Account.

kubectl get pods -n payments -o json | jq '.items[] | select(.spec.serviceAccountName == "payment-sa") | .metadata.name'

Change example: update a Deployment to use the Service Account.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-processor
  namespace: payments
spec:
  template:
    spec:
      serviceAccountName: payment-sa
      automountServiceAccountToken: true

Apply and verify.

kubectl apply -f deployment.yaml
kubectl rollout status deployment/payment-processor -n payments

Expected output: deployment "payment-processor" successfully rolled out.

Verification and Diagnostics

Verification for Service Accounts goes beyond checking existence. You need to confirm that the intended identity is working, that tokens have the right claims, and that the audit trail is clean.

Token Inspection

To inspect a Service Account token without exposing it, use the TokenRequest API.

kubectl create token payment-sa -n payments --duration=1h

This prints a JWT. Decode the payload (do not share it) to inspect claims.

echo <token> | cut -d. -f2 | base64 -d 2>/dev/null | jq .

Expected output includes fields like sub: system:serviceaccount:payments:payment-sa, namespace: payments, and expiration timestamps.

Pod Identity Diagnostics

If an application reports authentication errors, check the projected token mounted in the pod.

kubectl exec -it <pod-name> -n payments -- cat /var/run/secrets/kubernetes.io/serviceaccount/token

Then decode it as above. If the token is missing, check the pod spec for automountServiceAccountToken: false or a missing serviceAccountName.

Audit Log Correlation

If audit logging is enabled, search for requests made by the Service Account.

grep 'system:serviceaccount:payments:payment-sa' /var/log/kubernetes/audit.log | tail -n 20

Look for unexpected API calls or permission denials. Common failure signal: repeated 403 Forbidden errors from the same Service Account indicate a missing Role or RoleBinding.

Failure Modes and Recovery

Service Accounts fail in predictable ways. Here are the most common production issues, their symptoms, and recovery steps.

Failure 1: Missing Service Account

Symptom: Pod fails to start with Error creating: pods "x" is forbidden: error looking up service account ns/sa: serviceaccount "sa" not found.

Recovery: Create the Service Account or correct the serviceAccountName in the pod spec.

kubectl create serviceaccount <sa-name> -n <namespace>

Then restart the pod.

Failure 2: Automount Misconfiguration

Symptom: Application cannot authenticate because no token is mounted.

Diagnosis:

kubectl exec <pod-name> -n <namespace> -- ls /var/run/secrets/kubernetes.io/serviceaccount/

If the directory is empty or missing, automount is disabled somewhere. Check the pod spec and the Service Account.

Recovery: Set automountServiceAccountToken: true in the pod spec or Service Account, then recreate the pod.

Failure 3: Overprivileged Service Account Compromised

Symptom: Unusual API activity from a Service Account, such as attempting to list secrets across namespaces.

Immediate response: Revoke the token by deleting the Service Account secret (Kubernetes <1.24) or restarting the pod to invalidate the projected token if token rotation is set, then replace RBAC with least privilege.

Recovery steps:

1. Identify the Service Account: kubectl get pods -n <namespace> -o json | jq -r '.items[] | select(.spec.serviceAccountName == "<sa>") | .metadata.name'
2. Delete the pod to force a new token.
3. Review and tighten RBAC.
4. Audit logs for the time window of compromise.

Failure 4: Token Expiration

Symptom: Application logs show Token has expired.

Diagnosis: Check the token's expiration in the JWT payload. If using static secrets, they may not expire; if using projected tokens, they expire based on expirationSeconds.

Recovery: For static secrets, delete and recreate the secret. For projected tokens, restart the pod to get a new token. Consider enabling the LegacyServiceAccountTokenNoAutoGeneration feature gate to prevent static token auto-generation and force projected tokens.

Operations Checklist

Here is a concise operations checklist for maintaining Service Accounts in production. Each item includes a responsible owner and a review cadence.

Each item's owner is a single accountable person, not a group. They are responsible for running the check, documenting results, and initiating follow-ups. The review cadence can be adjusted based on team size and risk profile, but it should be explicit and tracked in a calendar or ticketing system.

Common Pitfalls and How to Avoid Them

Pitfall 1: Using the Default Service Account for Workloads

The default Service Account in each namespace typically has no explicit permissions but may have automount enabled, giving every pod a token. If a pod is compromised, the attacker can use that token to interact with the API (even if limited).

Avoidance: Set automountServiceAccountToken: false on the default Service Account, and create dedicated Service Accounts for each workload.

Recovery: If you discover pods using the default Service Account, migrate them to a dedicated one and patch the deployment, then restart the pods.

Pitfall 2: Granting Cluster-Wide Permissions to Namespaced Workloads

Using ClusterRoleBindings for a Service Account that only needs access within one namespace is a common over-granting mistake. For example, binding a web app's Service Account to view ClusterRole exposes resources across all namespaces.

Avoidance: Use RoleBinding with a namespaced Role whenever possible. Reserve ClusterRoleBindings for cluster-level controllers like ingress controllers or monitoring agents.

Pitfall 3: Hardcoding Service Account Tokens in Application Config

Applications sometimes read the Service Account token from a ConfigMap or environment variable, bypassing the mounted token. This creates a secret sprawl problem and breaks token rotation.

Avoidance: Always use the projected token mounted at /var/run/secrets/kubernetes.io/serviceaccount/token. If the application needs a long-lived token for external use, use a separate mechanism like a dedicated Secret or an external identity provider.

Pitfall 4: Ignoring Token Expiry in Long-Running Jobs

A long-running batch job using a static Service Account token may fail after the token expires (if the cluster enforces expiration).

Avoidance: For long-running jobs, use a dedicated Service Account with a projected token that has a long expirationSeconds (up to the maximum allowed) or configure the job to handle token refresh.

Conclusion

A production operations checklist for Kubernetes Service Accounts turns identity management from a one-time setup task into a continuously verified process. By following the observe-before-change principle, limiting the blast radius, and using concrete verification commands, teams can avoid common security and reliability failures.

Start with a low-risk verification: inventory your Service Accounts, check automount settings, and review one RBAC binding. Document the current state, run the check, compare the result with the expected signal, and assign an owner with a review date. Then expand the checklist to cover all namespaces and include audit log review.

A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. Service Accounts are a small part of the Kubernetes control plane, but their configuration has an outsized impact on cluster security and workload reliability.