## Introduction Kubernetes Secrets let you store and manage sensitive information such as passwords, tokens, and keys. However, Secrets are not infinite. Each Secret has a size limit, and storing too many large Secrets can strain etcd, the API server, and node memory. Capacity planning for Secrets means deciding how many Secrets you need, how large they can be, and how to avoid hitting cluster limits. Many teams treat Secrets as free-form storage and only discover problems when the API server rejects a create request or etcd latency spikes. This guide walks through a structured approach: inventory your environment, understand the real limits, design a safe configuration, verify with practical commands, plan for failure, and maintain an operational checklist. Every step includes concrete examples and commands you can run to validate your own cluster. By the end, you will be able to size your Secrets, set appropriate limits, avoid common pitfalls, and recover from capacity-related incidents with confidence. ## Version and Environment Inventory Capacity planning starts with knowing exactly what you are running. Different Kubernetes versions have different default limits and behaviors for Secrets. This section shows how to gather the necessary information without making any changes. ### Check Kubernetes Version and Distribution Run the following commands to identify your cluster version and API server details: ```bash kubectl version --short kubectl cluster-info ``` Example output on a managed cluster: ``` Client Version: v1.28.2 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.27.6-gke.500 ``` Knowing the exact version matters because the maximum Secret size has changed over time. As of Kubernetes 1.21, the limit is 1 MiB per Secret. Older clusters may allow up to 2 MiB for Secrets created before the limit was enforced, but new Secrets are capped at 1 MiB. Always verify your cluster's version and any custom admission policies that might impose stricter limits. ### Identify Secret Usage Patterns Next, get a snapshot of current Secrets in each namespace. Use these read-only commands: ```bash kubectl get secrets --all-namespaces kubectl get secrets -n your-namespace -o wide ``` To find the largest Secrets, you can inspect their sizes. This command lists Secrets in a namespace sorted by their data size: ```bash kubectl get secrets -n your-namespace -o json | jq -r '.items[] | [.metadata.name, (.data | to_entries | map(.value | @base64d | length) | add)] | @tsv' | sort -k2 -n -r ``` This requires `jq` and assumes the Secrets store data as base64-encoded values. The output shows the Secret name and total plaintext size in bytes. For example: ``` my-large-secret 1048576 api-token 256 ``` If any Secret approaches 1 MiB, it is a candidate for splitting or externalizing. ### Assess etcd and API Server Health Secrets are stored in etcd, so monitoring etcd is critical. Check etcd metrics if you have access (often via Prometheus or metrics endpoints). Key metrics include etcd database size and request latency. On a self-managed cluster, you can run: ```bash kubectl -n kube-system exec -it etcd-master-node -- etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key endpoint status --write-out=table ``` Look for `DB SIZE` and `RAFT INDEX`. A large DB size (e.g., over 2 GiB) may indicate too many large Secrets or insufficient compaction. Also, watch the API server request duration for Secret writes; slow responses can signal etcd pressure. ### Understand Cluster Storage Backend Managed Kubernetes services (EKS, GKE, AKS) handle etcd for you, but they still have quotas and limits. For example, GKE limits etcd storage to a certain size per cluster and enforces a maximum number of objects. Check your provider documentation. On self-managed clusters, ensure etcd has sufficient disk space and performance. The etcd backend quota defaults to 2 GiB, but this can be increased. If you exceed the quota, etcd stops accepting writes, which can take down your cluster control plane. ## Practical Limits and Sizing for Secrets This section details the concrete limits and how to plan for them. ### Maximum Secret Size As mentioned, the maximum size for a Secret is 1 MiB (1,048,576 bytes) for the combined size of its data fields. This includes base64-encoded values. For example, a Secret with two keys, one 500 KiB and one 524 KiB, would total 1,024 KiB and be accepted. But if the total exceeds 1 MiB, the API server rejects it with an error like: ``` The Secret "too-large" is invalid: data: Too long: must have at most 1048576 bytes ``` You can test this by attempting to create a Secret from a large file: ```bash # Create a 1.1 MiB file head -c 1153433 /dev/urandom > large-file.txt kubectl create secret generic too-large --from-file=large-file.txt ``` You should see an error similar to the above. ### Number of Secrets per Namespace and Cluster There is no hard limit on the number of Secrets, but practical limits exist. Every Secret adds objects to etcd, increases memory usage on the API server, and may affect list/watch performance. As a rule of thumb, keep the total number of objects (including Secrets, ConfigMaps, Pods, etc.) in a namespace below 10,000 for optimal performance. For the entire cluster, staying under 100,000 objects is advisable on typical setups. Exceeding these can lead to slower API responses and etcd strain. To count Secrets in a namespace: ```bash kubectl get secrets -n your-namespace --no-headers | wc -l ``` ### etcd Storage Size Each Secret consumes etcd space. Since etcd stores a copy of all cluster state, including Secrets, a large number of big Secrets can quickly fill the etcd database. Monitor the etcd database size using the command shown earlier. As a starting point, plan for Secret data to be a small fraction of total etcd usage—ideally less than 20-30%. If Secrets are pushing etcd beyond its quota, consider using external secret stores like Vault or cloud provider secret managers. ### Node Memory and Secret Mounts Secrets are mounted into Pods as tmpfs volumes or environment variables. Each mounted Secret consumes memory on the node. If a Pod mounts many large Secrets, the node can run out of memory. A common guideline is to keep individual Secrets under 100 KiB unless absolutely necessary and to mount only the Secrets a Pod needs. To see mounted Secrets for a Pod: ```bash kubectl get pod -o json | jq '.spec.volumes[]?.secret.secretName' ``` ### Sizing Worked Example Suppose you have a microservices application with 50 services, each requiring 3 configuration Secrets (DB credentials, API keys, TLS certs). Average Secret size is 10 KiB, but some TLS certs are 200 KiB. Total Secrets = 50 × 3 = 150. Estimated total size = (140 × 10 KiB) + (10 × 200 KiB) = 1.4 MiB + 2 MiB = 3.4 MiB in etcd. This is well within etcd limits, but if you have thousands of such services, you would need to scale accordingly. Plan for future growth: if you expect to double services in 6 months, design for 7 MiB of Secret data. ## Safe Configuration Path When making changes to Secret capacity, follow a safe path: start small, test, and verify. ### Use Namespace Resource Quotas One of the best ways to control Secret capacity is to define resource quotas at the namespace level. You can limit the total number of Secrets and the total storage consumed by Secrets in a namespace. Example `ResourceQuota` manifest: ```yaml apiVersion: v1 kind: ResourceQuota metadata: name: secret-quota namespace: your-namespace spec: hard: secrets: "100" requests.storage: "10Mi" ``` This limits the namespace to 100 Secrets and 10 MiB of total storage requests for PVCs; however, Secret storage is not directly counted by requests.storage. Instead, use a custom resource or rely on the count limit and individual Secret size limits. There is no built-in quota for total Secret data size, but you can enforce individual size limits with an admission controller like OPA Gatekeeper or Kyverno. Apply the quota and verify: ```bash kubectl apply -f secret-quota.yaml kubectl get resourcequota secret-quota -n your-namespace -o yaml ``` When a user tries to create more Secrets than allowed, they get a `Forbidden` error. ### Implement Admission Policies for Secret Size To cap Secret size, define a Kyverno policy if you use Kyverno. Example policy that rejects Secrets larger than 500 KiB: ```yaml apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: limit-secret-size spec: validationFailureAction: Enforce background: false rules: - name: check-secret-size match: any: - resources: kinds: - Secret validate: message: "Secret size exceeds 500 KiB" pattern: metadata: annotations: kyverno.io/secret-size: "*" deny: conditions: any: - key: "{{ request.object.data.values(@) | [].length(@) | sum(@) }}" operator: GreaterThan value: 524288 ``` This policy uses Kyverno's expression language to compute the sum of data value lengths. Adjust the threshold as needed. For clusters without Kyverno, you can use OPA Gatekeeper with a Rego policy that calculates total size. ### Encrypt Secrets at Rest Ensure etcd encryption is enabled for Secrets. This protects them if etcd data is compromised. On self-managed clusters, configure encryption at rest in the API server. On managed clusters, enable the provider's encryption option. For example, on GKE, you can enable Application-layer Secret Encryption. For EKS, you can enable envelope encryption with KMS. This does not affect capacity, but is a security best practice. ### Use External Secret Stores for Large Data For very large or frequently changing secrets, consider external stores like HashiCorp Vault, AWS Secrets Manager, or Google Secret Manager. Integrate with Kubernetes using tools like External Secrets Operator or Secrets Store CSI Driver. This keeps etcd small and allows centralized rotation. For example, the External Secrets Operator can sync a secret from AWS Secrets Manager to a Kubernetes Secret, but you can also configure Pods to mount directly from the CSI driver without creating a Kubernetes Secret at all, bypassing the etcd storage entirely. ## Verification and Diagnostics After you set limits or make changes, verify that everything works as expected. ### Validate Secret Creation Limits Try creating a Secret that is under the limit and one that is over, and observe the results. For a manual check: ```bash # Create a small secret kubectl create secret generic small-secret --from-literal=key=value # Create a large secret that should fail (if policy set) echo $(head -c 600000 /dev/urandom | base64) > big.txt kubectl create secret generic big-secret --from-file=big.txt ``` Expected output for the failing secret should be a validation error from the API server or the admission webhook, not a silent acceptance. ### Monitor etcd and API Server Metrics Use `kubectl get --raw /metrics` to scrape API server metrics and look for Secret-related request latencies. For etcd, if you have direct access, check the etcd metrics endpoint. For example, on a kubeadm cluster, run: ```bash curl -s https://localhost:2379/metrics -k --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key | grep etcd_debugging_mvcc_db_total_size_in_bytes ``` The output shows the current database size in bytes. Track this over time to see growth due to Secrets. ### Check Pod Startup with Mounted Secrets If a Secret is too large to mount, the Pod may fail to start with an error like `secret is too large` or the kubelet may log warnings. Simulate by creating a large Secret and mounting it into a test Pod (within your limits). Observe the Pod status: ```bash kubectl get pods kubectl describe pod ``` Look for events indicating volume mount failures. Some Kubernetes versions limit the size of a single secret volume to 1 MiB, and if the Secret is larger, the mount fails. ### Use Debug Containers For diagnosing Secret content size, you can enter a running Pod with a debug container (if supported) and inspect the mounted files. For example: ```bash kubectl debug -it --image=busybox --target= ``` Then inside the debug container, run `ls -lh /etc/secret-volume` to check the size of files. This is read-only and does not modify the Secret. ## Failure Modes and Recovery Capacity issues can cause various failures. This section covers common failure modes and how to recover. ### Secret Creation Rejected Due to Size **Symptoms:** `kubectl create secret` returns an error `Too long: must have at most 1048576 bytes`. Application deployments that depend on that Secret fail. **Recovery:** 1. Identify the large data and split it into multiple Secrets if it logically can be split (e.g., one Secret for cert, one for key). 2. If the data cannot be split, store it externally and reference via a smaller Secret (e.g., a Secret containing only a pointer or a small token to retrieve the data). 3. Compress the data before storing in a Secret (if the application can decompress). 4. Recreate the Secret with the new approach. Example splitting a Secret: ```bash # Original: one Secret with two large keys # Instead, create two Secrets: kubectl create secret generic app-cert --from-file=tls.crt kubectl create secret generic app-key --from-file=tls.key ``` Update the Pod spec to mount both Secrets. ### etcd Database Quota Exceeded **Symptoms:** API server returns `etcdserver: mvcc: database space exceeded` or similar. The entire cluster may become read-only. **Recovery:** 1. Immediately reduce etcd storage by deleting unnecessary objects, especially large Secrets and ConfigMaps. Identify large Secrets using the earlier command. 2. Compact etcd history (if you have access) using `etcdctl compact`. 3. Defragment etcd using `etcdctl defrag`. 4. Increase the etcd quota by adjusting the `--quota-backend-bytes` flag on etcd and restarting etcd (this is a planned maintenance step). 5. In the long term, move large Secrets to external storage. For managed clusters, contact the provider support and avoid hitting the quota by monitoring. ### Node Memory Pressure Due to Mounted Secrets **Symptoms:** Pods get evicted with `The node was low on resource: memory.` Or `kubectl describe node` shows `MemoryPressure` condition as `True`. **Recovery:** 1. Identify which Pods mount large Secrets and reduce their memory footprint by not mounting unnecessary Secrets. 2. Use projected volumes to mount multiple Secrets in a single volume; that may not reduce memory but improves efficiency. 3. Consider using environment variables for small Secrets only, but note that env vars also consume memory and cannot hold very large values. 4. If possible, increase node memory or spread Pods across nodes. To see memory usage per Pod, use `kubectl top pods` (requires metrics server). ### Secret Deleted Accidentally **Symptoms:** Pods fail to start with `MountVolume.SetUp failed for volume "secret" : secret "my-secret" not found`. **Recovery:** - If you have a backup or the Secret is managed by External Secrets Operator, it will be recreated automatically. - Otherwise, restore from your secret management system or rotate the credentials and recreate the Secret. - To prevent accidental deletion, enable Kubernetes RBAC to restrict delete permissions on Secrets, and consider using tools like Velero for backup of Secrets (though note that backing up Secrets in plaintext is risky). ### Key Takeaways for Failure Recovery Always have a rollback plan. For Secret changes, keep a copy of the previous Secret data in a secure location (e.g., encrypted in your secret manager) so you can revert quickly. Test recovery procedures periodically. ## Operations Checklist This checklist provides a repeatable process for managing Secret capacity. Assign an owner for each item and revisit at the specified frequency. ### Daily Checks - [ ] **Owner: Platform Engineer** - Check etcd database size and alert if over 70% of quota. Use monitoring dashboards. - [ ] **Owner: Kubernetes Admin** - Review API server error logs for Secret-related validation errors (search for "too long" or "secret size"). ### Weekly Checks - [ ] **Owner: DevOps Lead** - Review the number of Secrets per namespace. Run `kubectl get secrets --all-namespaces | awk '{print $1}' | sort | uniq -c | sort -nr`. Look for namespaces exceeding 500 Secrets. - [ ] **Owner: Security Engineer** - Verify that encryption at rest is enabled and key rotation works if applicable. ### Monthly Checks - [ ] **Owner: Platform Architect** - Audit Secret sizes using the `jq` command from earlier. Identify Secrets over 100 KiB and plan to split or externalize. - [ ] **Owner: Team Leads** - Review resource quotas and admission policies; adjust limits based on actual usage growth. ### Quarterly Checks - [ ] **Owner: Platform Architect** - Conduct a capacity planning exercise: project Secret growth based on historical trends and planned features. Update etcd quota and cluster sizing. - [ ] **Owner: Security Engineer** - Test recovery from Secret deletion and etcd failure scenarios in a staging environment. ### Annual Checks - [ ] **Owner: Director of Engineering** - Evaluate alternative secret management solutions (e.g., External Secrets Operator, Vault) and decide whether to migrate away from direct etcd storage. Each checklist item should have a clear owner, not a group, to ensure accountability. Revisit the checklist frequency itself every quarter to adjust. ## Common Pitfalls and How to Avoid Them ### 1. Treating Secrets as Unlimited Storage **Why it happens:** Teams assume that since etcd is a database, it can store anything. They put large files, certificates, or even binary data into Secrets. **How to avoid:** Set explicit policies limiting Secret size to 1 MiB or less. Use external storage for large data and reference it from a small Secret. Educate developers about the 1 MiB limit. **Recovery:** If you have large Secrets, audit and split them or externalize them. Use the `jq` command to find offenders. ### 2. Ignoring Namespace Quotas **Why it happens:** Teams don't set quotas because they don't anticipate growth. A single microservice team can create hundreds of Secrets accidentally. **How to avoid:** Enforce ResourceQuotas for `secrets` count in every namespace. Use LimitRange to set default sizes if needed (though LimitRange does not apply to Secret sizes). **Recovery:** Apply quotas retroactively; existing over-quota Secrets will remain, but new ones will be blocked. Clean up outdated Secrets. ### 3. Not Monitoring etcd Growth **Why it happens:** etcd metrics are often not exposed or monitored, especially in small clusters. **How to avoid:** Set up Prometheus monitoring for etcd and alert on database size. For managed clusters, enable provider monitoring. **Recovery:** If etcd grows too large, compact and defrag, then delete unneeded Secrets. ### 4. Mounting Secrets Unnecessarily **Why it happens:** Developers copy Pod specs and include all available Secrets for convenience, leading to memory waste and potential leaks. **How to avoid:** Use RBAC to limit which Secrets a Pod can mount by namespace and service account. Review Pod specs in CI/CD to ensure only required Secrets are referenced. **Recovery:** Find Pods with excessive mounts using `kubectl get pods -o json | jq` and update deployments to remove unneeded mounts. ### 5. Storing Secrets in Plaintext in Git **Why it happens:** Teams want version control for Secrets and commit them to repos, violating security. **How to avoid:** Use sealed-secrets, external secrets with encrypted references, or a secret manager with GitOps integration. Never store raw Secrets in Git. **Recovery:** Rotate all exposed Secrets immediately, then move to a secure method. ### 6. Forgetting about Base64 Encoding Overhead **Why it happens:** Developers think the Secret size is the plaintext size, but base64 encoding adds ~33% overhead. A 800 KiB file becomes ~1.06 MiB after encoding, exceeding the limit. **How to avoid:** Always calculate the base64-encoded size when designing Secrets. The limit is on the encoded data in the Secret object, not the plaintext. **Recovery:** If you hit the limit, compress the plaintext before base64 encoding if the application can handle it, or reduce the data size. ## Conclusion Kubernetes Secret capacity planning is not a one-time task but an ongoing operational discipline. By understanding the hard limits, monitoring actual usage, and implementing guardrails, you can prevent outages and keep your cluster healthy. Start by auditing your current Secret landscape using the commands in this guide. Set up resource quotas and admission policies. Monitor etcd and API server metrics. Then, establish a routine review process with clear ownership. Finally, test failure scenarios to ensure you can recover quickly. A thoughtful capacity plan for Secrets will improve both security and reliability, ensuring that your applications have the secrets they need without overwhelming your cluster. Remember: small, managed, and monitored Secrets are the key to smooth operations. As a next step, pick one low-risk verification from this article, run it in your development cluster, record the results, and share the findings with your team.