## Intro Kubernetes clusters depend on X.509 certificates for secure communication between control plane components, nodes, and workloads. The Certificate Signing Request (CSR) API lets clients request certificates that the Kubernetes control plane must approve and sign. In large or dynamic clusters, the volume of CSRs can grow quickly. Without deliberate capacity planning, you can end up with a backlog of pending CSRs. That backlog can prevent new nodes from joining, block service account token issuance, and leave applications without the credentials they need. This guide gives you a practical, step-by-step approach to estimating CSR volume, configuring the controller manager for your workload, monitoring scaling signals, and maintaining safety margins. By the end, you will be able to plan, verify, and operate CSR capacity for your clusters with confidence. CSR processing is a critical path for cluster operations. When it works, everything boots and authenticates smoothly. When it fails, the symptoms are often confusing: nodes stuck in NotReady, pods failing to start, or mysterious authentication errors. This guide focuses on the Kubernetes CertificateSigningRequest API, the controllers that approve and sign those requests, and how to size them properly. ## Version and Environment Inventory Before you calculate capacity, document your Kubernetes environment. The CSR API and controller behavior have changed across releases. This guide assumes Kubernetes v1.22 or later, where the CertificateSigningRequest API is stable at `certificates.k8s.io/v1`. If you are on an older version, upgrade first or consult the relevant documentation because the signer names and controller flags differ. ### Key Components Involved in CSR Processing Several components participate in CSR processing. Understanding each one helps you know where to look when things go wrong. | Component | Role in CSR Processing | |---|---| | `kube-controller-manager` | Runs the CSR approver and signer controllers. It validates, approves, and signs certificates. | | `kube-apiserver` | Exposes the CSR API, handles authentication and authorization, and stores CSR objects in etcd. | | `etcd` | Persists CSR objects and their status. High CSR churn increases etcd write load. | | `kubelet` | Creates CSRs for node certificates during bootstrap and renewal. | | Service account controller | Creates CSRs for service account tokens when the legacy token controller is used. | To check your cluster version, run: ```bash kubectl version --short ``` Expected output: ```text Client Version: v1.27.3 Server Version: v1.27.3 ``` Record the following in your capacity plan: - Number of nodes and expected node turnover rate (e.g., 5 nodes replaced per day due to autoscaling). - Certificate lifetimes in use (default is often 8760h = 1 year for kubelet client certificates, but check your controller manager flags). - Expected rate of service account token CSRs if you use the legacy token controller (most modern clusters use projected tokens instead, so this may be zero). - Any custom CSR workflows, such as an external approver that signs CSRs for user certificates. This information feeds directly into your capacity estimation in the next section. ## Safe Configuration Path The controller manager has several flags that directly affect CSR throughput. Configure them deliberately after you understand your expected load. ### Core Flags for CSR Processing | Flag | Description | Default | |---|---|---| | `--cluster-signing-duration` | Lifetime of signed certificates. | 8760h (1 year) | | `--cluster-signing-cert-file` | Path to the CA certificate used for signing. | Empty (must be set) | | `--cluster-signing-key-file` | Path to the CA private key. | Empty (must be set) | | `--controllers` | List of controllers to enable, including `certificatesigningrequest-approving` and `certificatesigningrequest-signing`. | `*` (all controllers) | | `--concurrent-csr-signers` | Number of CSR signing operations that can run concurrently. | 5 | Concurrency is often the first lever to adjust. The default of 5 workers is enough for small clusters, but if you have thousands of nodes rotating certificates daily, you may need to raise it. However, more concurrency consumes more CPU and memory, so you must pair it with adequate resource requests. ### Example Controller Manager Configuration Here is a snippet from a kubeadm-managed cluster's `kube-controller-manager` pod spec. In production, these flags are usually in `/etc/kubernetes/manifests/kube-controller-manager.yaml`. ```yaml apiVersion: v1 kind: Pod metadata: name: kube-controller-manager namespace: kube-system spec: containers: - command: - kube-controller-manager - --cluster-signing-duration=8760h - --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt - --cluster-signing-key-file=/etc/kubernetes/pki/ca.key - --controllers=*,bootstrapsigner,tokencleaner - --concurrent-csr-signers=10 image: registry.k8s.io/kube-controller-manager:v1.27.3 resources: requests: cpu: 250m memory: 256Mi limits: cpu: 500m memory: 512Mi ``` This config increases concurrency to 10 and sets explicit resource requests. Start conservative and scale up based on observed latency, not just pending count. ### Estimating CSR Volume Capacity planning starts with a realistic estimate of how many CSRs your cluster will generate per day. A simple formula for node CSRs: ```text Daily node CSRs = Number of nodes / (certificate lifetime in days x renewal factor) ``` The renewal factor is the fraction of lifetime at which nodes typically renew. Kubelet renews its client certificate at 80% of lifetime by default, so the factor is 0.8. For example: - 1000 nodes - 30-day certificate lifetime - Renewal at 80% = 24 days ```text Daily CSRs = 1000 / (30 x 0.8) = 1000 / 24 ≈ 41.7 CSRs per day ``` That means roughly 42 CSRs per day just from node certificate renewals. Add user certificate requests and any service account token CSRs. User certificate requests vary widely; in a busy enterprise cluster, you might see another 10-20 per day. In a cluster using legacy service account tokens, each new pod might trigger a token CSR, but that pattern is discouraged in favor of projected tokens. Measure the actual rate using the Kubernetes API. The following command counts CSRs created in the last 24 hours (requires `jq` and `date`): ```bash kubectl get csr -o json | jq '[.items[] | select(.metadata.creationTimestamp > (now - 86400 | strftime("%Y-%m-%dT%H:%M:%SZ")))] | length' ``` Compare your estimate to the measured value. If the measured rate is significantly higher, investigate sources of CSRs you may have overlooked, such as frequent node churn or a misbehaving component. ### Setting Resource Requests and Limits Once you know the CSR rate, size the controller manager appropriately. A baseline for moderate load (about 50 CSRs per day) is 100m CPU and 128Mi memory. For higher rates or higher concurrency, scale up. The signing operation is CPU-intensive due to cryptographic operations, so watch CPU throttling. If the controller manager is consistently throttled, increase its CPU request. Use the following Prometheus query to monitor CPU throttling: ```promql rate(container_cpu_cfs_throttled_seconds_total{namespace="kube-system", pod=~"kube-controller-manager.*"}[5m]) ``` If the result is consistently above 0, the container is being throttled. Increase the CPU request or reduce concurrency. ### Gradual Scaling Approach Do not jump from default to maximum concurrency. Increase gradually while monitoring pending CSR count and signing latency. A safe approach: 1. Start with default `--concurrent-csr-signers=5`. 2. Monitor pending count and signing latency for one week. 3. If pending count grows or latency exceeds your SLO (e.g., 1 second), increase concurrency by 2. 4. Repeat until pending count stays near zero and latency is stable. The controller manager also runs other controllers, so excessive concurrency on CSR signing may starve other controllers. Use separate controller manager instances if you need to isolate heavy CSR workloads, but that is an advanced setup. ## Verification and Diagnostics After configuration, verify that CSR processing works end to end, then set up monitoring and alerting. ### End-to-End Test Create a test CSR to confirm the entire flow: creation, approval, signing, and retrieval. Use `openssl` to generate a key and CSR, then submit it via `kubectl`. ```bash # Generate a private key openssl genrsa -out test.key 2048 # Create a CSR with subject CN=test-user openssl req -new -key test.key -out test.csr -subj "/CN=test-user" # Base64 encode the CSR and create the Kubernetes CSR object cat <10 for 5 minutes. | | `csr_controller_manager_csr_sign_latency_seconds` | Latency of CSR signing operations. | Alert if p99 > 2 seconds for 10 minutes. | | `csr_controller_manager_csr_approve_latency_seconds` | Latency of CSR approval operations. | Monitor trend; alert on anomaly. | | `csr_controller_manager_csr_sign_total` | Total number of CSRs signed. | Track rate for capacity planning. | Example Prometheus alert rule: ```yaml groups: - name: csr-alerts rules: - alert: HighPendingCSRs expr: csr_controller_manager_pending_csrs > 10 for: 5m labels: severity: warning annotations: summary: "High number of pending CSRs" description: "{{ $value }} CSRs are pending for more than 5 minutes." ``` Set up a dashboard showing pending CSRs, signing latency, and CSR creation rate. Review it weekly. ## Failure Modes and Recovery Despite planning, failures happen. Know the common failure modes and how to recover quickly. ### High Pending CSR Count Cause: The controller manager cannot keep up with the incoming CSR rate. This could be due to insufficient CPU, low concurrency, or a backlog after an outage. Recovery: 1. Check the number of pending CSRs: ```bash kubectl get csr --field-selector=status.certificate="" | wc -l ``` 2. If high, increase controller manager resources and `--concurrent-csr-signers` as described earlier. 3. Check controller manager logs for errors. 4. If the backlog is due to a temporary burst (e.g., many nodes rebooted simultaneously), you may manually approve CSRs to help clear the queue: ```bash kubectl get csr -o name | xargs kubectl certificate approve ``` Use manual approval with caution; it bypasses normal authorization checks. ### Signing Failures Cause: The signing certificate or key file is missing, invalid, or has wrong permissions. This often happens after CA rotation. Recovery: 1. Check controller manager logs for signing errors. 2. Verify the files exist and are readable by the controller manager process: ```bash ls -l /etc/kubernetes/pki/ca.crt /etc/kubernetes/pki/ca.key ``` 3. If the CA was rotated, update the `--cluster-signing-cert-file` and `--cluster-signing-key-file` flags to point to the new CA. 4. Restart the controller manager. ### etcd Storage Full Cause: Old CSRs accumulate and consume etcd space. This can lead to etcd performance degradation or failure. Recovery: 1. Check etcd storage usage. 2. Delete old CSRs. The following command deletes CSRs older than 24 hours (requires `date` and `xargs`): ```bash kubectl get csr -o json | jq -r '.items[] | select(.metadata.creationTimestamp < (now - 86400 | strftime("%Y-%m-%dT%H:%M:%SZ"))) | .metadata.name' | xargs kubectl delete csr ``` 3. Implement a retention policy. A simple CronJob can delete CSRs older than N days. Example CronJob manifest: ```yaml apiVersion: batch/v1 kind: CronJob metadata: name: csr-cleanup namespace: kube-system spec: schedule: "0 2 * * *" jobTemplate: spec: template: spec: serviceAccountName: csr-cleanup containers: - name: cleanup image: bitnami/kubectl:latest command: - /bin/sh - -c - | kubectl get csr -o json | jq -r '.items[] | select(.metadata.creationTimestamp < (now - 172800 | strftime("%Y-%m-%dT%H:%M:%SZ"))) | .metadata.name' | xargs -r kubectl delete csr restartPolicy: OnFailure ``` This CronJob requires a ServiceAccount with permissions to delete CSRs. ### Rollback Procedure If a configuration change causes CSR processing to break, revert to the previous configuration and restart the controller manager. Always keep backups of the controller manager manifest and signing keys. In kubeadm clusters, the manifest is in `/etc/kubernetes/manifests/kube-controller-manager.yaml`; simply restore the previous file and the kubelet will restart the pod automatically. ## Operations Checklist Use this checklist to maintain CSR health. Assign a responsible owner for each item and review frequency. | Item | Frequency | Owner | How to Verify | |---|---|---|---| | Monitor pending CSR count | Daily | On-call SRE | Check Prometheus dashboard or run `kubectl get csr --field-selector=status.certificate=""` | | Review CSR signing latency | Weekly | Platform Engineer | Check Prometheus p99 latency; ensure under 2 seconds | | Check controller manager resource usage | Weekly | Platform Engineer | Verify CPU throttling is zero; memory within limits | | Clean up old CSRs | Monthly | Automation via CronJob | Confirm CronJob runs successfully | | Test CSR creation and approval | Quarterly | Security Engineer | Run end-to-end test manually or via CI | | Validate certificate lifetimes | Quarterly | Security Engineer | Inspect a sample of issued certificates for expiry dates | | Review scaling thresholds | As needed (after cluster growth >20%) | Capacity Planner | Recalculate CSR rate and compare to current controller capacity | For each item, the owner is responsible for completing the task and reporting anomalies. The review frequency should be adjusted based on cluster maturity; a rapidly growing cluster may need more frequent reviews. Additional operational commands to include in scripts: ```bash # Count pending CSRs kubectl get csr --field-selector=status.certificate="" | wc -l # List CSRs sorted by creation time (oldest first) kubectl get csr --sort-by=.metadata.creationTimestamp # Show CSR details with status kubectl get csr -o custom-columns=NAME:.metadata.name,AGE:.metadata.creationTimestamp,CONDITION:.status.conditions[*].type ``` Document your capacity plan and update it after significant cluster changes such as adding many nodes, changing certificate lifetimes, or enabling new CSR workflows. ## Common Pitfalls and How to Avoid Them Even experienced Kubernetes operators can trip over CSR capacity planning. Here are the most common pitfalls and how to steer clear of them. ### Pitfall 1: Ignoring Node Churn in Capacity Estimates Many teams calculate CSR volume based on the steady-state node count, forgetting that node replacements generate CSRs. For example, a cluster with 1000 nodes and 30-day certificate lifetime generates about 42 CSRs per day, but if you also replace 50 nodes daily due to autoscaling or spot instances, you add 50 more CSRs per day, more than doubling the load. Always include churn in your estimate. How to avoid: Track node creation and deletion rates over a week. Add the daily churn count to your CSR estimate. If churn is high, consider longer certificate lifetimes to reduce renewal frequency. ### Pitfall 2: Setting Concurrency Too High Without Resource Headroom Increasing `--concurrent-csr-signers` blindly can starve other controllers. The controller manager runs dozens of controllers, and CSR signing is CPU-intensive. If you set concurrency to 20 on a controller manager with only 100m CPU, you will see CPU throttling and delays across all controllers because the manager is a single process. How to avoid: Monitor CPU usage of the entire controller manager, not just CSR-specific metrics. Increase concurrency only after confirming CPU headroom. Use separate controller manager instances if you need to isolate heavy workloads. ### Pitfall 3: Not Setting Up Alerting for Pending CSRs A sudden influx of CSRs can go unnoticed until nodes fail to join. Without alerts, you may only discover the problem when users complain. For example, a network partition that prevents the controller manager from reaching the API server can cause a pending CSR backlog silently. How to avoid: Set up a Prometheus alert for `csr_controller_manager_pending_csrs > 10` for more than 5 minutes. Also alert on controller manager process restarts, because a crash can halt CSR processing. ### Pitfall 4: Failing to Clean Up Old CSRs CSR objects accumulate indefinitely if not deleted. Over months, this can consume significant etcd space and slow down API queries. In one real-world case, a cluster had over 100,000 CSRs in etcd, causing API latency spikes. How to avoid: Implement a retention policy using a CronJob as shown earlier. Delete CSRs older than 48 hours as a starting point. ### Pitfall 5: Assuming Default Certificate Lifetime Is Optimal The default `--cluster-signing-duration` is 8760h (1 year). Long lifetimes mean fewer renewals but also longer exposure if a key is compromised. For high-churn clusters, shorter lifetimes can increase CSR load unnecessarily. Balance security and capacity. How to avoid: Evaluate your security requirements and operational capacity. For example, if your nodes are ephemeral (autoscaling), a 30-day lifetime may be more appropriate. Recalculate CSR volume when changing lifetimes. ### Pitfall 6: Not Testing CSR Flow After Upgrades Kubernetes upgrades can change default signer names or controller flags. After upgrading, your existing CSR workflow may silently break. For instance, the `kubernetes.io/kube-apiserver-client` signer name might be deprecated in favor of a new one. How to avoid: Include a CSR end-to-end test in your post-upgrade checklist. Run the test CSR creation and approval steps manually or via an automated script. ## Conclusion Capacity planning for Kubernetes Certificate Signing Requests is not a one-time task. As your cluster grows and changes, you must revisit your assumptions. Start by documenting your environment and estimating realistic CSR volume. Configure the controller manager with appropriate concurrency and resources based on that estimate. Verify with an end-to-end test and set up monitoring and alerting for pending CSRs and signing latency. Be prepared for common failure modes such as high pending counts, signing failures, and etcd bloat. Follow the operations checklist with clear ownership and review frequencies. Finally, avoid the pitfalls that trip up many teams by accounting for node churn, scaling resources alongside concurrency, and cleaning up old CSRs regularly. With these practices, you can keep your cluster's certificate issuance reliable and responsive, even at scale.