## Intro Kubernetes Service Account CI/CD automation turns manual identity management into a repeatable pipeline, but a pipeline that applies YAML without verification is just a script. This guide gives developers, DevOps consultants, and startup platform teams a practical path from baseline observation to verified change for service accounts, their roles, bindings, and token handling. We focus on the daily operational loop: inspect the current state, make the smallest justified change, verify the outcome, and know how to recover. You will find concrete kubectl commands, version-scoped checks, and example manifests for each stage. The examples assume a working Kubernetes cluster (v1.24 or later) and a namespace named payments unless stated otherwise. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document recovery before an incident forces the decision. ## Version and Environment Inventory Before touching service accounts, record the cluster's control plane and API versions. Many CI/CD failures start because the local kubectl client, the cluster API, or a manifest's apiVersion disagree. Begin with read-only commands and capture the output with timestamps. ### Check the Cluster and Client Version Run kubectl version --short (or kubectl version on newer clients) to see the client and server versions. For example: $ kubectl version --short Client Version: v1.28.2 Server Version: v1.28.1 If your pipeline targets a cluster with an older API version, the manifest will fail. Service accounts, roles, and role bindings are stable v1 objects. But token request APIs differ: the TokenRequest API is available since v1.20, and the legacy Secret -based service account tokens are no longer auto-created since v1.24. Use kubectl api-versions | grep authentication to confirm token-related APIs. ### Inspect Existing Service Accounts and Dependencies List service accounts in the target namespace: kubectl get serviceaccounts -n payments Expected output if only the default account exists: NAME SECRETS AGE default 0 45d If a CI/CD pipeline expects a specific service account, confirm it exists with kubectl get serviceaccount ci-deployer -n payments -o yaml . The YAML will show its annotations, image pull secrets, and any auto-generated tokens. Record related RBAC objects: roles ( kubectl get roles -n payments ), role bindings ( kubectl get rolebindings -n payments ), and secrets if you still mount legacy tokens ( kubectl get secrets -n payments | grep ci-deployer ). ### Example Baseline Snapshot Store a snapshot of the current identity configuration before any change. This is your recovery reference: mkdir -p /tmp/sa-baseline && cd /tmp/sa-baseline kubectl get serviceaccount ci-deployer -n payments -o yaml > sa-before.yaml kubectl get role ci-deployer-role -n payments -o yaml > role-before.yaml kubectl get rolebinding ci-deployer-binding -n payments -o yaml > binding-before.yaml Use timestamps in filenames or a git commit to freeze the baseline. If the change fails, kubectl apply -f sa-before.yaml restores the previous service account. The recovery path is clear before you change anything.
Quick check 1 of 2 What is a recommended way to validate Kubernetes service account credentials in your own code? - TokenReview API - OIDC discovery - Manual JWT parsing - Using kubectl auth can-i
The passage states that the Kubernetes project recommends using the TokenReview API for validating service account credentials, as it invalidates tokens bound to deleted API objects.
## Safe Configuration Path Service accounts are harmless without roles and bindings, but a misconfigured binding can grant unintended access to a pod. The safe path is to create or update one object at a time, verify its effect, and only then proceed. This section covers the recommended sequence for a typical CI/CD service account. ### Step 1: Define the Service Account with Minimal Metadata Create a service account for a deployment pipeline that will deploy applications in the payments namespace. Apply this YAML: # sa-ci-deployer.yaml apiVersion: v1 kind: ServiceAccount metadata: name: ci-deployer namespace: payments labels: app.kubernetes.io/name: payments app.kubernetes.io/component: deployer Apply and check the result: kubectl apply -f sa-ci-deployer.yaml kubectl get serviceaccount ci-deployer -n payments -o wide Expected output shows no secrets (since v1.24): NAME SECRETS AGE ci-deployer 0 2m Do not set automountServiceAccountToken: false unless you are absolutely certain the workload never needs to authenticate to the Kubernetes API. For CI/CD jobs, the token is usually required, so leave it true (the default). ### Step 2: Create a Role with Least Privilege Define a role that allows the CI/CD system to manage only deployments, services, and config maps in payments . This is a scoped role, not cluster-wide. Apply this YAML: # role-ci-deployer.yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: ci-deployer-role namespace: payments rules: - apiGroups: ["apps"] resources: ["deployments"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] - apiGroups: [""] resources: ["services", "configmaps"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] Apply and inspect the role: kubectl apply -f role-ci-deployer.yaml kubectl describe role ci-deployer-role -n payments Expected output includes each rule with resources and verbs. If you need to allow the pipeline to scale deployments, add "scale" to the verbs for deployments/scale under apps . ### Step 3: Bind the Role to the Service Account A role is inert until bound. Use a RoleBinding to connect the service account to the role in the same namespace: # rolebinding-ci-deployer.yaml apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: ci-deployer-binding namespace: payments subjects: - kind: ServiceAccount name: ci-deployer namespace: payments roleRef: kind: Role name: ci-deployer-role apiGroup: rbac.authorization.k8s.io Apply the binding: kubectl apply -f rolebinding-ci-deployer.yaml kubectl get rolebinding ci-deployer-binding -n payments -o wide Verify the subject is correct: kubectl describe rolebinding ci-deployer-binding -n payments Look for a line similar to: Subjects: Kind Name Namespace ---- ---- --------- ServiceAccount ci-deployer payments If the subject shows a different namespace or kind, the binding will not grant permissions. Fix it before moving on. ## Verification and Diagnostics After applying the role and binding, you must prove that the service account actually has the intended permissions and that a pod using it can perform the allowed actions. Do not assume success from YAML apply output. ### Verify Permissions with kubectl auth can-i Use kubectl auth can-i with --as to simulate the service account. For example, check if the CI/CD account can create deployments in payments : kubectl auth can-i create deployments --as=system:serviceaccount:payments:ci-deployer -n payments Expected result if the role is correct: yes Check a prohibited action, such as deleting a secret: kubectl auth can-i delete secrets --as=system:serviceaccount:payments:ci-deployer -n payments Expected result: no If you get yes for an unintended action, review the role rules and any cluster roles that might be bound to this service account. ### Verify a Pod Can Use the Service Account Token Create a temporary pod that uses the service account and attempts a Kubernetes API call. Save the following as pod-test-sa.yaml : apiVersion: v1 kind: Pod metadata: name: sa-token-test namespace: payments spec: serviceAccountName: ci-deployer containers: - name: kubectl image: bitnami/kubectl:latest command: ["sleep", "3600"] restartPolicy: Never Run the pod, exec into it, and inspect the mounted token: kubectl apply -f pod-test-sa.yaml kubectl wait --for=condition=Ready pod/sa-token-test -n payments --timeout=60s kubectl exec -it sa-token-test -n payments -- sh # inside the container cat /var/run/secrets/kubernetes.io/serviceaccount/token The token should be a long JWT string. Attempt an API call: kubectl exec -it sa-token-test -n payments -- sh -c 'kubectl auth can-i list deployments -n payments --token=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)' Expected output: yes . A no may indicate that the service account's token is not being projected or the role lacks the permission. ### Debugging Common Issues - Pod cannot list deployments but kubectl auth can-i says yes : The pod's service account may not be correctly specified in the pod spec. Check kubectl get pod sa-token-test -n payments -o jsonpath='{.spec.serviceAccountName}' to confirm it matches. - Token expired or empty : Since v1.24, service accounts do not create long-lived secrets by default. The projected token is short-lived and automatically rotated. For CI/CD, use a pod spec that requests a token with a longer expiration via automountServiceAccountToken or use the TokenRequest API. Alternatively, create a secret of type kubernetes.io/service-account-token manually, but prefer the projected token. - RBAC denials in cluster logs : Enable audit logging temporarily to see why a request was denied. For an EKS cluster, you can use CloudTrail. For a self-managed cluster, check kube-apiserver audit logs. A typical denied request will show the user as system:serviceaccount:payments:ci-deployer and the missing verb/resource.
Quick check 2 of 2 Which of the following is a suggested use case for service accounts according to the reference? - Granting cross-namespace access to resources - Storing database credentials for applications - Encrypting network traffic between pods - Providing DNS resolution for services
The passage lists granting cross-namespace access, such as allowing a Pod in namespace 'example' to read Lease objects in 'kube-node-lease', as a use case for service accounts.
## Failure Modes and Recovery Even with correct YAML, CI/CD automation can fail due to namespace absence, binding misconfiguration, or token issues. This section outlines common failure modes, how to detect them, and exact recovery steps. ### Failure 1: Namespace Does Not Exist If the pipeline applies a manifest with namespace: payments but the namespace was never created, kubectl returns an error like: Error from server (NotFound): namespaces "payments" not found Detection : The CI/CD job fails immediately on apply. In a GitOps system like Argo CD, the sync status shows Unknown or Missing for the namespace. Recovery : Create the namespace and reapply the manifests: kubectl create namespace payments kubectl apply -f sa-ci-deployer.yaml -f role-ci-deployer.yaml -f rolebinding-ci-deployer.yaml Prevent this by adding a namespace creation step at the start of the pipeline or using a cluster-scoped manifest that includes the namespace. ### Failure 2: Role Binding Points to a Different Service Account A common mistake is binding a role to the wrong subject. For example, the binding subject might have name: default instead of ci-deployer . The pod will start without errors but any API call requiring permissions will fail with 403 Forbidden . Detection : Run kubectl auth can-i for the intended service account (as shown in Verification and Diagnostics) and check the binding's subject with kubectl get rolebinding ci-deployer-binding -n payments -o yaml . Look for the subjects array. Recovery : Correct the subject in the RoleBinding YAML and reapply: subjects: - kind: ServiceAccount name: ci-deployer # corrected from default namespace: payments Then reapply: kubectl apply -f rolebinding-ci-deployer.yaml Verify again with kubectl auth can-i . ### Failure 3: Legacy Secret Token Missing After Cluster Upgrade If you upgraded from Kubernetes v1.23 to v1.24 or later, existing service accounts may lose their auto-created secrets. A pipeline that references a secret by name for a service account token will fail to find it. Detection : The pipeline logs show an error like Secret "ci-deployer-token-abcde" not found . kubectl get serviceaccount ci-deployer -n payments -o yaml no longer lists the secret under secrets . Recovery : Use the projected token instead. If you absolutely need a long-lived token, create a secret manually and link it to the service account: # sa-token-secret.yaml apiVersion: v1 kind: Secret metadata: name: ci-deployer-token namespace: payments annotations: kubernetes.io/service-account.name: ci-deployer type: kubernetes.io/service-account-token Then patch the service account to include the secret: kubectl apply -f sa-token-secret.yaml kubectl patch serviceaccount ci-deployer -n payments -p '{"secrets": [{"name": "ci-deployer-token"}]}' However, this approach has security risks and is discouraged. Prefer updating the CI/CD system to authenticate via the projected token or using a short-lived token obtained from the TokenRequest API. ### Rollback Strategy Because you saved the baseline YAML files at the start, rollback is a simple kubectl apply of the previous state. For example, to revert a role change: kubectl apply -f /tmp/sa-baseline/role-before.yaml If the change was applied via a pipeline, use the pipeline's rollback feature or a Git revert commit. Always re-run the verification step after rollback to confirm the previous permissions are restored. ## Operations Checklist Use this checklist before, during, and after any service account change in CI/CD. It is designed to be pasted into a runbook or issue template and filled with concrete values. ### Pre-Change Checklist kubectl get serviceaccount ci-deployer -n payments -o yaml > /tmp/sa-baseline/sa-before.yaml kubectl get role ci-deployer-role -n payments -o yaml > /tmp/sa-baseline/role-before.yaml kubectl get rolebinding ci-deployer-binding -n payments -o yaml > /tmp/sa-baseline/binding-before.yaml - [ ] Confirm cluster version with kubectl version --short and note any API differences. - [ ] Verify target namespace exists: kubectl get namespace payments . - [ ] List current service accounts and RBAC objects in the namespace: kubectl get sa,role,rolebinding -n payments . - [ ] Record baseline YAML for the specific objects you will change: - [ ] Confirm the change is scoped to a single namespace or object set. - [ ] Note the recovery procedure: restore baseline YAML. ### Change Execution Checklist - [ ] Apply the service account manifest: kubectl apply -f sa-ci-deployer.yaml - [ ] Verify the service account is created: kubectl get serviceaccount ci-deployer -n payments - [ ] Apply the role manifest: kubectl apply -f role-ci-deployer.yaml - [ ] Verify the role rules: kubectl describe role ci-deployer-role -n payments - [ ] Apply the role binding: kubectl apply -f rolebinding-ci-deployer.yaml - [ ] Verify the binding subject: kubectl get rolebinding ci-deployer-binding -n payments -o yaml - [ ] Test permissions with kubectl auth can-i for at least one allowed and one denied action. - [ ] (Optional) Run a temporary pod using the service account to test token-based access. ### Post-Change Verification Checklist - [ ] Confirm kubectl auth can-i results match expectations for all key actions. - [ ] Check that no unintended cluster-scoped permissions were granted ( kubectl get clusterrolebinding -o yaml | grep ci-deployer should return nothing). - [ ] Update documentation with the new service account name, permissions, and usage example. - [ ] Tag the change in git for auditability. - [ ] If the change is part of a CI/CD pipeline, run the pipeline's own verification step (e.g., a smoke test deployment using the service account). ### Example Filled Checklist for a Deployment Pipeline Upgrade Here is an example of a completed checklist for a specific change: creating a new service account ci-deployer for the payments backend deployment pipeline.
ItemCheckOutput or Note
Cluster versionkubectl version --shortv1.28.1
Namespace existskubectl get namespace paymentsActive
Baseline savedgit commit 4f2a1cFiles in /tmp/sa-baseline
SA appliedkubectl apply -f sa-ci-deployer.yamlserviceaccount/ci-deployer created
Role appliedkubectl apply -f role-ci-deployer.yamlrole.rbac.authorization.k8s.io/ci-deployer-role created
Binding appliedkubectl apply -f rolebinding-ci-deployer.yamlrolebinding.rbac.authorization.k8s.io/ci-deployer-binding created
Allowed actionkubectl auth can-i create deployments --as=... -n paymentsyes
Denied actionkubectl auth can-i delete secrets --as=... -n paymentsno
Pipeline smoke testRun deployment jobDeployment successful
## Conclusion Kubernetes Service Account CI/CD automation is valuable only when each step is version-scoped, observable, and reversible. Copying a command without checking prerequisites and expected output is not an operations procedure; it is a gamble. Start with one low-risk verification: choose a service account, record its current state with the baseline commands, run the documented checks, compare the result with the expected signal, and review dependencies such as roles, role bindings, and secrets. A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. Use the checklist in this guide as a living document: adjust it as your cluster evolves and your CI/CD pipeline matures. Your next step is to test this process in a development namespace. Create a test service account, bind a minimal role, run the verification commands, and intentionally break a binding to practice recovery. Once the loop feels natural, apply the same rigor to production changes.