Kubernetes Certificate Signing Request Capacity Planning: A Practical Guide

Intro

Kubernetes clusters depend on X.509 certificates for secure communication between control plane components, nodes, and workloads. The Certificate Signing Request (CSR) API lets clients request certificates that the Kubernetes control plane must approve and sign. In large or dynamic clusters, the volume of CSRs can grow quickly. Without deliberate capacity planning, you can end up with a backlog of pending CSRs. That backlog can prevent new nodes from joining, block service account token issuance, and leave applications without the credentials they need. This guide gives you a practical, step-by-step approach to estimating CSR volume, configuring the controller manager for your workload, monitoring scaling signals, and maintaining safety margins. By the end, you will be able to plan, verify, and operate CSR capacity for your clusters with confidence.

CSR processing is a critical path for cluster operations. When it works, everything boots and authenticates smoothly. When it fails, the symptoms are often confusing: nodes stuck in NotReady, pods failing to start, or mysterious authentication errors. This guide focuses on the Kubernetes CertificateSigningRequest API, the controllers that approve and sign those requests, and how to size them properly.

Version and Environment Inventory

Before you calculate capacity, document your Kubernetes environment. The CSR API and controller behavior have changed across releases. This guide assumes Kubernetes v1.22 or later, where the CertificateSigningRequest API is stable at certificates.k8s.io/v1. If you are on an older version, upgrade first or consult the relevant documentation because the signer names and controller flags differ.

Key Components Involved in CSR Processing

Several components participate in CSR processing. Understanding each one helps you know where to look when things go wrong.

To check your cluster version, run:

kubectl version --short

Expected output:

Client Version: v1.27.3
Server Version: v1.27.3

Record the following in your capacity plan:

• Number of nodes and expected node turnover rate (e.g., 5 nodes replaced per day due to autoscaling).
• Certificate lifetimes in use (default is often 8760h = 1 year for kubelet client certificates, but check your controller manager flags).
• Expected rate of service account token CSRs if you use the legacy token controller (most modern clusters use projected tokens instead, so this may be zero).
• Any custom CSR workflows, such as an external approver that signs CSRs for user certificates.

This information feeds directly into your capacity estimation in the next section.

Safe Configuration Path

The controller manager has several flags that directly affect CSR throughput. Configure them deliberately after you understand your expected load.

Core Flags for CSR Processing

Concurrency is often the first lever to adjust. The default of 5 workers is enough for small clusters, but if you have thousands of nodes rotating certificates daily, you may need to raise it. However, more concurrency consumes more CPU and memory, so you must pair it with adequate resource requests.

Example Controller Manager Configuration

Here is a snippet from a kubeadm-managed cluster's kube-controller-manager pod spec. In production, these flags are usually in /etc/kubernetes/manifests/kube-controller-manager.yaml.

apiVersion: v1
kind: Pod
metadata:
  name: kube-controller-manager
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-controller-manager
    - --cluster-signing-duration=8760h
    - --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
    - --cluster-signing-key-file=/etc/kubernetes/pki/ca.key
    - --controllers=*,bootstrapsigner,tokencleaner
    - --concurrent-csr-signers=10
    image: registry.k8s.io/kube-controller-manager:v1.27.3
    resources:
      requests:
        cpu: 250m
        memory: 256Mi
      limits:
        cpu: 500m
        memory: 512Mi

This config increases concurrency to 10 and sets explicit resource requests. Start conservative and scale up based on observed latency, not just pending count.

Estimating CSR Volume

Capacity planning starts with a realistic estimate of how many CSRs your cluster will generate per day. A simple formula for node CSRs:

Daily node CSRs = Number of nodes / (certificate lifetime in days x renewal factor)

The renewal factor is the fraction of lifetime at which nodes typically renew. Kubelet renews its client certificate at 80% of lifetime by default, so the factor is 0.8. For example:

• 1000 nodes
• 30-day certificate lifetime
• Renewal at 80% = 24 days

Daily CSRs = 1000 / (30 x 0.8) = 1000 / 24 ≈ 41.7 CSRs per day

That means roughly 42 CSRs per day just from node certificate renewals. Add user certificate requests and any service account token CSRs. User certificate requests vary widely; in a busy enterprise cluster, you might see another 10-20 per day. In a cluster using legacy service account tokens, each new pod might trigger a token CSR, but that pattern is discouraged in favor of projected tokens.

Measure the actual rate using the Kubernetes API. The following command counts CSRs created in the last 24 hours (requires jq and date):

kubectl get csr -o json | jq '[.items[] | select(.metadata.creationTimestamp > (now - 86400 | strftime("%Y-%m-%dT%H:%M:%SZ")))] | length'

Compare your estimate to the measured value. If the measured rate is significantly higher, investigate sources of CSRs you may have overlooked, such as frequent node churn or a misbehaving component.

Setting Resource Requests and Limits

Once you know the CSR rate, size the controller manager appropriately. A baseline for moderate load (about 50 CSRs per day) is 100m CPU and 128Mi memory. For higher rates or higher concurrency, scale up. The signing operation is CPU-intensive due to cryptographic operations, so watch CPU throttling. If the controller manager is consistently throttled, increase its CPU request.

Use the following Prometheus query to monitor CPU throttling:

rate(container_cpu_cfs_throttled_seconds_total{namespace="kube-system", pod=~"kube-controller-manager.*"}[5m])

If the result is consistently above 0, the container is being throttled. Increase the CPU request or reduce concurrency.

Gradual Scaling Approach

Do not jump from default to maximum concurrency. Increase gradually while monitoring pending CSR count and signing latency. A safe approach:

1. Start with default --concurrent-csr-signers=5.
2. Monitor pending count and signing latency for one week.
3. If pending count grows or latency exceeds your SLO (e.g., 1 second), increase concurrency by 2.
4. Repeat until pending count stays near zero and latency is stable.

The controller manager also runs other controllers, so excessive concurrency on CSR signing may starve other controllers. Use separate controller manager instances if you need to isolate heavy CSR workloads, but that is an advanced setup.

Verification and Diagnostics

After configuration, verify that CSR processing works end to end, then set up monitoring and alerting.

End-to-End Test

Create a test CSR to confirm the entire flow: creation, approval, signing, and retrieval. Use openssl to generate a key and CSR, then submit it via kubectl.

# Generate a private key
openssl genrsa -out test.key 2048

# Create a CSR with subject CN=test-user
openssl req -new -key test.key -out test.csr -subj "/CN=test-user"

# Base64 encode the CSR and create the Kubernetes CSR object
cat <<EOF | kubectl apply -f -
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: test-csr
spec:
  request: $(cat test.csr | base64 | tr -d '\n')
  signerName: kubernetes.io/kube-apiserver-client
  usages:
  - client auth
EOF

Approve the CSR and check its status:

kubectl certificate approve test-csr
kubectl get csr test-csr

Expected output:

NAME       AGE   SIGNERNAME                      REQUESTOR          CONDITION
test-csr   10s   kubernetes.io/kube-apiserver-client   user           Approved,Issued

If the CSR remains Pending, inspect the controller manager logs:

kubectl logs -n kube-system -l component=kube-controller-manager | grep -i csr

Look for error messages such as failed to sign CSR, no signer found, or certificate key mismatch. Those indicate configuration problems.

Monitoring Metrics and Alerts

The controller manager exposes metrics via the secure port (10257 by default). Key metrics for CSR capacity planning:

Example Prometheus alert rule:

groups:
- name: csr-alerts
  rules:
  - alert: HighPendingCSRs
    expr: csr_controller_manager_pending_csrs > 10
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "High number of pending CSRs"
      description: "{{ $value }} CSRs are pending for more than 5 minutes."

Set up a dashboard showing pending CSRs, signing latency, and CSR creation rate. Review it weekly.

Failure Modes and Recovery

Despite planning, failures happen. Know the common failure modes and how to recover quickly.

High Pending CSR Count

Cause: The controller manager cannot keep up with the incoming CSR rate. This could be due to insufficient CPU, low concurrency, or a backlog after an outage.

Recovery:

1. Check the number of pending CSRs:

kubectl get csr --field-selector=status.certificate="" | wc -l

1. If high, increase controller manager resources and --concurrent-csr-signers as described earlier.
2. Check controller manager logs for errors.
3. If the backlog is due to a temporary burst (e.g., many nodes rebooted simultaneously), you may manually approve CSRs to help clear the queue:

kubectl get csr -o name | xargs kubectl certificate approve

Use manual approval with caution; it bypasses normal authorization checks.

Signing Failures

Cause: The signing certificate or key file is missing, invalid, or has wrong permissions. This often happens after CA rotation.

Recovery:

1. Check controller manager logs for signing errors.
2. Verify the files exist and are readable by the controller manager process:

ls -l /etc/kubernetes/pki/ca.crt /etc/kubernetes/pki/ca.key

1. If the CA was rotated, update the --cluster-signing-cert-file and --cluster-signing-key-file flags to point to the new CA.
2. Restart the controller manager.

etcd Storage Full

Cause: Old CSRs accumulate and consume etcd space. This can lead to etcd performance degradation or failure.

Recovery:

1. Check etcd storage usage.
2. Delete old CSRs. The following command deletes CSRs older than 24 hours (requires date and xargs):

kubectl get csr -o json | jq -r '.items[] | select(.metadata.creationTimestamp < (now - 86400 | strftime("%Y-%m-%dT%H:%M:%SZ"))) | .metadata.name' | xargs kubectl delete csr

1. Implement a retention policy. A simple CronJob can delete CSRs older than N days. Example CronJob manifest:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: csr-cleanup
  namespace: kube-system
spec:
  schedule: "0 2 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: csr-cleanup
          containers:
          - name: cleanup
            image: bitnami/kubectl:latest
            command:
            - /bin/sh
            - -c
            - |
              kubectl get csr -o json | jq -r '.items[] | select(.metadata.creationTimestamp < (now - 172800 | strftime("%Y-%m-%dT%H:%M:%SZ"))) | .metadata.name' | xargs -r kubectl delete csr
          restartPolicy: OnFailure

This CronJob requires a ServiceAccount with permissions to delete CSRs.

Rollback Procedure

If a configuration change causes CSR processing to break, revert to the previous configuration and restart the controller manager. Always keep backups of the controller manager manifest and signing keys. In kubeadm clusters, the manifest is in /etc/kubernetes/manifests/kube-controller-manager.yaml; simply restore the previous file and the kubelet will restart the pod automatically.

Operations Checklist

Use this checklist to maintain CSR health. Assign a responsible owner for each item and review frequency.

For each item, the owner is responsible for completing the task and reporting anomalies. The review frequency should be adjusted based on cluster maturity; a rapidly growing cluster may need more frequent reviews.

Additional operational commands to include in scripts:

# Count pending CSRs
kubectl get csr --field-selector=status.certificate="" | wc -l

# List CSRs sorted by creation time (oldest first)
kubectl get csr --sort-by=.metadata.creationTimestamp

# Show CSR details with status
kubectl get csr -o custom-columns=NAME:.metadata.name,AGE:.metadata.creationTimestamp,CONDITION:.status.conditions[*].type

Document your capacity plan and update it after significant cluster changes such as adding many nodes, changing certificate lifetimes, or enabling new CSR workflows.

Common Pitfalls and How to Avoid Them

Even experienced Kubernetes operators can trip over CSR capacity planning. Here are the most common pitfalls and how to steer clear of them.

Pitfall 1: Ignoring Node Churn in Capacity Estimates

Many teams calculate CSR volume based on the steady-state node count, forgetting that node replacements generate CSRs. For example, a cluster with 1000 nodes and 30-day certificate lifetime generates about 42 CSRs per day, but if you also replace 50 nodes daily due to autoscaling or spot instances, you add 50 more CSRs per day, more than doubling the load. Always include churn in your estimate.

How to avoid: Track node creation and deletion rates over a week. Add the daily churn count to your CSR estimate. If churn is high, consider longer certificate lifetimes to reduce renewal frequency.

Pitfall 2: Setting Concurrency Too High Without Resource Headroom

Increasing --concurrent-csr-signers blindly can starve other controllers. The controller manager runs dozens of controllers, and CSR signing is CPU-intensive. If you set concurrency to 20 on a controller manager with only 100m CPU, you will see CPU throttling and delays across all controllers because the manager is a single process.

How to avoid: Monitor CPU usage of the entire controller manager, not just CSR-specific metrics. Increase concurrency only after confirming CPU headroom. Use separate controller manager instances if you need to isolate heavy workloads.

Pitfall 3: Not Setting Up Alerting for Pending CSRs

A sudden influx of CSRs can go unnoticed until nodes fail to join. Without alerts, you may only discover the problem when users complain. For example, a network partition that prevents the controller manager from reaching the API server can cause a pending CSR backlog silently.

How to avoid: Set up a Prometheus alert for csr_controller_manager_pending_csrs > 10 for more than 5 minutes. Also alert on controller manager process restarts, because a crash can halt CSR processing.

Pitfall 4: Failing to Clean Up Old CSRs

CSR objects accumulate indefinitely if not deleted. Over months, this can consume significant etcd space and slow down API queries. In one real-world case, a cluster had over 100,000 CSRs in etcd, causing API latency spikes.

How to avoid: Implement a retention policy using a CronJob as shown earlier. Delete CSRs older than 48 hours as a starting point.

Pitfall 5: Assuming Default Certificate Lifetime Is Optimal

The default --cluster-signing-duration is 8760h (1 year). Long lifetimes mean fewer renewals but also longer exposure if a key is compromised. For high-churn clusters, shorter lifetimes can increase CSR load unnecessarily. Balance security and capacity.

How to avoid: Evaluate your security requirements and operational capacity. For example, if your nodes are ephemeral (autoscaling), a 30-day lifetime may be more appropriate. Recalculate CSR volume when changing lifetimes.

Pitfall 6: Not Testing CSR Flow After Upgrades

Kubernetes upgrades can change default signer names or controller flags. After upgrading, your existing CSR workflow may silently break. For instance, the kubernetes.io/kube-apiserver-client signer name might be deprecated in favor of a new one.

How to avoid: Include a CSR end-to-end test in your post-upgrade checklist. Run the test CSR creation and approval steps manually or via an automated script.

Conclusion

Capacity planning for Kubernetes Certificate Signing Requests is not a one-time task. As your cluster grows and changes, you must revisit your assumptions. Start by documenting your environment and estimating realistic CSR volume. Configure the controller manager with appropriate concurrency and resources based on that estimate. Verify with an end-to-end test and set up monitoring and alerting for pending CSRs and signing latency. Be prepared for common failure modes such as high pending counts, signing failures, and etcd bloat. Follow the operations checklist with clear ownership and review frequencies. Finally, avoid the pitfalls that trip up many teams by accounting for node churn, scaling resources alongside concurrency, and cleaning up old CSRs regularly. With these practices, you can keep your cluster's certificate issuance reliable and responsive, even at scale.