Kubernetes Service Accounts Admin: Capacity Planning with Practical Examples

Intro

Kubernetes Service Accounts are the identity layer for workloads running inside a cluster. Every pod is assigned a service account, and that identity controls which Kubernetes API operations the pod can perform. In production clusters, the number of service accounts and their associated tokens, secrets, and RBAC bindings can grow quickly, leading to performance degradation, security risks, and operational pain.

This article provides a practical guide to capacity planning for Kubernetes Service Accounts from an administrator's perspective. It is written for DevOps engineers, SREs, and technical startup teams who need to understand how service accounts consume cluster resources, set appropriate limits, and plan for scale.

We will cover:

• How service accounts consume cluster resources and why capacity planning matters
• Concrete methods to count and measure service accounts and related objects
• How to set and enforce quotas and limits
• How to monitor and verify the state of service accounts
• Common failure modes and recovery procedures
• A ready-to-use operations checklist

The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached.

How Service Accounts Impact Cluster Capacity

Service accounts themselves are lightweight objects, but they are tightly coupled to several other Kubernetes resources that can consume significant cluster capacity:

• ServiceAccount objects: Stored in etcd, each service account consumes memory and storage.
• Secrets (legacy token secrets): Before Kubernetes 1.24, each service account automatically generated a token secret. These secrets are stored in etcd and can consume substantial space.
• Tokens (projected volume tokens): Since Kubernetes 1.24, service account tokens are time-bound and issued via the TokenRequest API. They are not stored as secrets, but the kube-apiserver must handle token issuance and validation, adding CPU and memory load.
• RoleBindings and ClusterRoleBindings: Each binding references a service account. The RBAC authorizer must evaluate these bindings for every API request, so a large number of bindings can slow down API requests.
• Pods: Every pod mounts a service account token (unless disabled). The kubelet must project that token into the pod, and the pod's identity is used for admission control and authorization.

Capacity planning for service accounts therefore involves not just counting ServiceAccount objects, but also estimating the growth of related secrets, tokens, and bindings, and understanding their impact on etcd and the API server.

Resource Consumption Examples

Let us quantify the impact with a realistic example.

Assume a cluster with:

• 10,000 service accounts
• Each service account has 1 legacy token secret (if using Kubernetes < 1.24) or 1 projected token (if using >= 1.24)
• Each service account is bound to at least one RoleBinding or ClusterRoleBinding

etcd storage:

• A ServiceAccount object is typically 1-2 KB in size.
• A Secret object (legacy token) is about 2-5 KB (includes token data and metadata).
• So for 10,000 service accounts with legacy secrets, total etcd storage for these objects is about 30-70 MB. That may seem small, but etcd performance degrades as the database grows, and frequent writes (e.g., secret rotations) can amplify I/O.

API server memory:

• The API server caches most objects in memory for efficient reads. Each object in the cache consumes memory.
• A ServiceAccount object may use about 500 bytes in cache.
• A Secret object may use about 1 KB in cache.
• So 10,000 service accounts + 10,000 secrets could add 15 MB to the API server's memory. This is manageable, but if you have 100,000 or 1,000,000, it becomes significant.

RBAC authorization cost:

• The RBAC authorizer builds a map of bindings to roles. The number of bindings directly affects the time to authorize a request.
• With many bindings, the authorizer may need to scan a large list to find applicable rules. This can add milliseconds to each API request.
• In a busy cluster, a high number of bindings can become a bottleneck.

Token issuance rate:

• When using projected tokens, each pod requests a token from the API server at startup. The API server must issue and sign these tokens.
• The token signing operation is CPU-intensive. If you create many pods simultaneously, the token issuance rate can spike CPU usage.

Thus, capacity planning must account for all these dimensions.

Observing Current Service Account Usage

Before making any changes, you must measure the current state. Use read-only commands to gather data.

Count Service Accounts

kubectl get serviceaccounts --all-namespaces | wc -l

This gives a rough count. To get a more precise number per namespace:

kubectl get serviceaccounts --all-namespaces -o json | jq '.items | length'

Note: The default namespace contains the default service account automatically created for each namespace.

Count Legacy Token Secrets

If you are on Kubernetes < 1.24, each service account automatically gets a token secret. Count them with:

kubectl get secrets --all-namespaces --field-selector type=kubernetes.io/service-account-token | wc -l

On newer clusters, this may return zero because projected tokens are used instead.

Count RoleBindings and ClusterRoleBindings

kubectl get rolebindings --all-namespaces | wc -l
kubectl get clusterrolebindings --all-namespaces | wc -l

Check etcd Database Size

If you have access to etcd, check its database size and endpoint status:

ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  endpoint status --write-out=table

Look at the DB SIZE column. The default etcd quota is 2 GiB (configurable via --quota-backend-bytes). If you are near the quota, writes will fail, including service account creation.

Check API Server Metrics

If you have metrics-server or Prometheus, query relevant metrics:

• apiserver_request_duration_seconds (especially for RBAC-heavy operations)
• etcd_db_total_size_in_bytes
• apiserver_storage_objects (by resource type)

For example, to see the number of service accounts stored:

apiserver_storage_objects{resource="serviceaccounts"}

This gives a direct count from the API server's perspective.

Setting Limits and Quotas

To prevent unbounded growth, you should set resource quotas and limits on namespaces. This is one of the most effective capacity planning measures.

Namespace-Level Quotas

You can create a ResourceQuota that limits the number of service accounts and secrets in a namespace.

Example quota.yaml:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: service-account-quota
  namespace: my-app
spec:
  hard:
    serviceaccounts: "100"
    secrets: "200"

Apply it:

kubectl apply -f quota.yaml

Then try to create more service accounts than allowed. You will see an error like:

Error from server (Forbidden): error when creating "sa.yaml": serviceaccounts "test-sa" is forbidden: exceeded quota: service-account-quota, requested: serviceaccounts=1, used: serviceaccounts=100, limited: serviceaccounts=100

Cluster-Level Limits

There is no built-in cluster-wide quota for service accounts, but you can use admission controllers like OPA Gatekeeper or Kyverno to enforce policies.

For example, with Kyverno, you can create a cluster policy that limits the total number of service accounts:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: limit-serviceaccounts
spec:
  validationFailureAction: enforce
  background: false
  rules:
  - name: limit-serviceaccount-count
    match:
      resources:
        kinds:
        - ServiceAccount
    validate:
      message: "Too many service accounts in namespace"
      pattern:
        metadata:
          labels:
            app.kubernetes.io/name: "?*"

However, this is a simple example; a more robust policy would use a context to count existing service accounts and deny if above a threshold. Kyverno has built-in support for such checks using deny rules with conditions.

Token Lifetimes and Rotation

When using projected tokens, you can set a shorter token lifetime to limit exposure and reduce the window for token theft. The default lifetime is 1 hour, but it can be configured via the --service-account-max-token-expiration flag on the kube-apiserver. The per-pod token lifetime can be set in the pod spec:

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  serviceAccountName: my-sa
  containers:
  - name: app
    image: nginx
    volumeMounts:
    - mountPath: /var/run/secrets/tokens
      name: my-token
  volumes:
  - name: my-token
    projected:
      sources:
      - serviceAccountToken:
          path: my-token
          expirationSeconds: 3600

Setting expirationSeconds to a lower value (e.g., 600 seconds) forces token rotation more frequently, which can increase API server load but reduces the risk of a stolen token being useful.

Disable Automounting Tokens

Not all pods need to talk to the Kubernetes API. You can disable automounting of service account tokens at the service account or pod level to reduce token issuance and mount overhead.

Service account level:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-sa
automountServiceAccountToken: false

Pod level:

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  serviceAccountName: my-sa
  automountServiceAccountToken: false
  containers:
  - name: app
    image: nginx

This reduces the number of tokens requested and mounted, relieving some pressure on the API server and kubelet.

Monitoring and Verification

After applying limits, you must verify that they are effective and that the cluster remains healthy.

Check Quota Usage

kubectl describe resourcequota service-account-quota -n my-app

Output shows used vs hard limits.

Monitor API Server Latency

Watch the apiserver_request_duration_seconds metric for RBAC operations. If you see an increase after adding many bindings, you may need to scale the API server or reduce bindings.

Test Token Issuance

You can simulate pod creation and measure token issuance time. For example, create a test pod and measure the time from creation to ready:

time kubectl run test-pod --image=nginx --restart=Never

A slow token issuance may indicate API server overload.

Verify etcd Health

Regularly check etcd health:

ETCDCTL_API=3 etcdctl endpoint health

If etcd becomes read-only due to quota, all mutations will fail, including service account creation.

Failure Modes and Recovery

Failure: Exceeded Service Account Quota

Symptom: kubectl create serviceaccount fails with error: exceeded quota.

Recovery:

1. Check current usage: kubectl get serviceaccounts -n my-app | wc -l
2. Identify obsolete service accounts and delete them:

   kubectl delete serviceaccount obsolete-sa -n my-app

1. Or temporarily increase quota if needed, but reassess after cleanup.

Failure: etcd Quota Exceeded

Symptom: API server returns etcdserver: mvcc: database space exceeded errors, and writes fail.

Recovery:

1. Immediately run etcd compaction and defragmentation:

   ETCDCTL_API=3 etcdctl compact <current-revision>
   ETCDCTL_API=3 etcdctl defrag

1. Identify large objects and clean up (e.g., old secrets).
2. Increase etcd quota if necessary, but first understand what caused the growth.

Failure: API Server Overloaded by Token Requests

Symptom: High CPU on API server, increased latency, pods stuck in ContainerCreating.

Recovery:

1. Reduce token issuance by disabling automount where possible.
2. Add more API server replicas or increase CPU/memory.
3. Use token request caching if available.

Failure: RBAC Authorization Delays

Symptom: Slow API responses even when etcd and CPU are fine; authorization decisions take long.

Recovery:

1. Reduce the number of role bindings by consolidating roles.
2. Use aggregation rules to simplify cluster roles.
3. Enable RBAC authorization caching if supported (e.g., in some Kubernetes distributions).

Failure: Service Account Token Expired

Symptom: Pods report authentication errors when calling the API: Unauthorized.

Recovery:

1. Check pod's token expiration: kubectl exec my-pod -- cat /var/run/secrets/kubernetes.io/serviceaccount/token | cut -d'.' -f2 | base64 -d | jq .exp
2. If expired, delete the pod to force recreation with a fresh token.
3. For long-running pods, use workload identity mechanisms (e.g., AWS IAM roles for service accounts) to avoid token lifetime issues.

Operations Checklist

Use this checklist regularly to ensure service account capacity is under control.

Daily Checks

• [ ] Monitor apiserver_storage_objects{resource="serviceaccounts"} for unexpected spikes.
• [ ] Check API server error rate for quota exceeded errors.
• [ ] Verify etcd health: etcdctl endpoint health.

Weekly Checks

• [ ] Review namespace quotas: kubectl get resourcequota --all-namespaces.
• [ ] Identify namespaces with high service account counts: kubectl get serviceaccounts --all-namespaces -o json | jq -r '.items[] | .metadata.namespace' | sort | uniq -c | sort -nr | head.
• [ ] Look for unused service accounts (not referenced by any pod): cross-reference with pod specs.

Monthly Checks

• [ ] Analyze etcd database size trend and forecast growth.
• [ ] Review token expiry settings and rotation policies.
• [ ] Audit RBAC bindings for over-permissive or stale entries.

Capacity Planning Adjustments

• [ ] If service account count is approaching quota limits, plan to increase quota or optimize.
• [ ] If etcd size is above 70% of quota, plan compaction/defrag or increase quota.
• [ ] If API server latency is degrading, consider scaling or reducing bindings.

Scaling Strategies

When your cluster grows, here are strategies to scale service account management.

Automate Service Account Lifecycle

Use tools like Terraform, GitOps (ArgoCD/Flux), or custom controllers to create and delete service accounts as part of application deployment. This prevents orphaned service accounts.

Use Namespace Segmentation

Create separate namespaces for different teams or applications, each with its own quota. This limits the blast radius of any single application.

Adopt Workload Identity

Where possible, use cloud provider IAM roles (e.g., IAM Roles for Service Accounts on EKS, Workload Identity on GKE, Azure AD Pod Identity) to reduce reliance on Kubernetes service account tokens. This can eliminate token management overhead and improve security.

Regularly Clean Up Stale Objects

Schedule a cron job to delete unused service accounts and secrets. For example, use Kubernetes' built-in TTL controller for finished resources, or write a custom script.

Conclusion

Capacity planning for Kubernetes Service Accounts is not just about counting objects; it requires understanding the interplay between service accounts, secrets, tokens, and RBAC, and their impact on etcd and the API server. By observing current usage, setting appropriate quotas, monitoring key metrics, and having recovery procedures in place, you can keep your cluster healthy as it scales.

The operational principles emphasized throughout this article are: observe before changing, limit blast radius, verify results, and document recovery paths. By following the practical examples and checklist, you can build a robust service account management strategy that supports both security and performance.

As a next step, start by measuring your current service account usage with the commands provided, then set namespace quotas and monitor the impact. Over time, iterate on your capacity plan as your cluster evolves.