Learn how to identify and fix Kubernetes Certificate Signing Request performance bottlenecks. This guide covers resource sizing, latency checks, throughput tuning, and safe optimization workflows with concrete commands and expected outputs.

## Intro

Kubernetes clusters rely on certificates for secure communication between components. The Certificate Signing Request (CSR) API is the standard way to obtain certificates signed by the cluster's certificate authority. However, as clusters grow, the CSR pipeline can become a performance bottleneck, causing delays in node bootstrapping, workload identity issuance, and service mesh certificate rotation. This guide provides practical steps to tune Kubernetes CSR performance, from baseline measurements to optimization and verification. By following these steps, you can reduce latency, increase throughput, and ensure reliable certificate issuance under load.

## Version and Environment Inventory

Before tuning, document your Kubernetes version, control plane configuration, and CSR usage patterns. The CSR API is part of the certificates.k8s.io API group, and its behavior can vary between versions. Use kubectl version to check client and server versions:

```
$ kubectl version --short
Client Version: v1.28.2
Server Version: v1.28.2
```

Note: In newer Kubernetes versions, the --short flag is deprecated; use kubectl version without it for full details.

Identify the controller manager configuration, especially the --cluster-signing-cert-file and --cluster-signing-key-file flags, which determine the CA used for signing CSRs. Also check the --cluster-signing-duration flag, which sets the default certificate duration. You can view the controller manager pod spec or the static pod manifest:

```
$ kubectl get pod -n kube-system -l component=kube-controller-manager -o yaml | grep -E 'cluster-signing|kube-api'
```

Expected output includes lines like:

```
- --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
- --cluster-signing-key-file=/etc/kubernetes/pki/ca.key
- --cluster-signing-duration=8760h
- --kube-api-qps=20
- --kube-api-burst=40
```

Assess your workload: are CSRs generated by kubelet during node registration, by cert-manager for webhook certificates, or by custom controllers? The volume and frequency of CSRs will guide tuning decisions. For example, a cluster with frequent node autoscaling will have high CSR churn, requiring efficient approval and signing.

Finally, establish baseline performance metrics: measure CSR approval and signing latency, and throughput under load. Tools like kubectl get csr and kubectl certificate approve can be used to manually test, but for systematic benchmarking, consider using a script that generates multiple CSRs and measures time to issuance. Here is a simple bash script to create a test CSR and measure the time until it is approved and issued:

```bash
#!/bin/bash
# Generate a private key and CSR
openssl req -new -newkey rsa:2048 -nodes -keyout test.key -out test.csr -subj "/CN=test-user/O=test-group"

# Base64 encode the CSR and create a Kubernetes CSR object
CSR_B64=$(cat test.csr | base64 | tr -d '\n')
cat <<EOF | kubectl apply -f -
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: test-csr-$(date +%s)
spec:
  request: $CSR_B64
  signerName: kubernetes.io/kube-apiserver-client
  usages:
  - client auth
EOF

# Record start time
START=$(date +%s%N)

# Approve the CSR
CSR_NAME=$(kubectl get csr -o jsonpath='{.items[?(@.metadata.name=="test-csr-*")].metadata.name}' | tr ' ' '\n' | tail -1)
kubectl certificate approve $CSR_NAME

# Wait for the CSR to be issued
while true; do
  CONDITION=$(kubectl get csr $CSR_NAME -o jsonpath='{.status.conditions[?(@.type=="Approved")].status}')
  if [ "$CONDITION" == "True" ]; then
    ISSUED=$(kubectl get csr $CSR_NAME -o jsonpath='{.status.certificate}')
    if [ -n "$ISSUED" ]; then
      END=$(date +%s%N)
      ELAPSED_MS=$(( (END - START) / 1000000 ))
      echo "CSR issued in ${ELAPSED_MS} ms"
      break
    fi
  fi
  sleep 0.1
done
```

Run this several times and record the average latency. For throughput, run the script in parallel for a known number of requests (e.g., 100 CSRs) and measure the total time.

## Safe Configuration Path

Start with scoped, reversible changes. The Kubernetes controller manager has several flags that affect CSR signing performance. The most critical are:

- --cluster-signing-duration : default certificate lifetime. Shorter durations reduce the window of compromise but may increase renewal frequency. For high-churn environments, a shorter duration may be acceptable.
- --kube-api-qps : the rate limit for requests from the controller manager to the API server. Increasing this can speed up CSR processing but may overload the API server.
- --kube-api-burst : the burst size for the above rate limit.

Example: To increase the API QPS for the controller manager, edit its manifest (e.g., /etc/kubernetes/manifests/kube-controller-manager.yaml ) and add or modify the flags:

```yaml
spec:
  containers:
  - command:
    - kube-controller-manager
    - --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
    - --cluster-signing-key-file=/etc/kubernetes/pki/ca.key
    - --cluster-signing-duration=8760h
    - --kube-api-qps=100
    - --kube-api-burst=200
```

After modifying, the controller manager will restart automatically. Monitor its logs for errors.

If using an external signer like cert-manager, tune its controller resources and API rate limits instead. Always make one change at a time and measure before and after.

For cert-manager, you can adjust resources in the deployment:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cert-manager
  namespace: cert-manager
spec:
  template:
    spec:
      containers:
      - name: cert-manager
        resources:
          requests:
            cpu: 100m
            memory: 64Mi
          limits:
            cpu: 500m
            memory: 256Mi
```

Additionally, cert-manager has its own rate limiting for ACME and other issuers; consult its documentation for fine-tuning.

## Verification and Diagnostics

After tuning, verify that CSRs are processed correctly and observe performance improvements. Use kubectl get csr to list pending CSRs and check their conditions:

```
$ kubectl get csr
NAME        AGE   SIGNERNAME                            REQUESTOR          CONDITION
csr-abcde   10s   kubernetes.io/kube-apiserver-client   kubelet-bootstrap   Pending
```

Approve a CSR manually:

```
$ kubectl certificate approve csr-abcde
certificatesigningrequest.certificates.k8s.io/csr-abcde approved
```

Then check its status:

```
$ kubectl get csr csr-abcde
NAME        AGE   SIGNERNAME                            REQUESTOR          CONDITION
csr-abcde   30s   kubernetes.io/kube-apiserver-client   kubelet-bootstrap   Approved,Issued
```

To measure latency, record the time from CSR creation to issuance. For automated testing, create a CSR YAML and submit it, noting the timestamp. Then approve and observe the time to condition Issued. For example, using the script above, you might get:

```
CSR issued in 150 ms
```

For throughput, generate multiple CSRs concurrently using a script and measure the rate at which they are issued. Expect improvements if you increased API QPS or reduced signing duration. Compare against baseline metrics collected earlier.

Additionally, check controller manager logs for any rate limiting or errors:

```
$ kubectl logs -n kube-system kube-controller-manager-<node> | grep -i csr
```

Look for messages like "Failed to sign CSR" or "Error updating CSR status".

## Failure Modes and Recovery

Tuning CSR performance can introduce risks. Common failure modes:

- API server overload : Increasing rate limits too much can cause the controller manager to overwhelm the API server, affecting other controllers. Monitor API server latency and error rates. If observed, revert the QPS/burst changes.
- Certificate expiry : Shortening --cluster-signing-duration may cause certificates to expire before renewal. Ensure renewal mechanisms are in place, such as kubelet automatic renewal or cert-manager. If certificates expire prematurely, manually rotate the affected certificates and adjust duration.
- Signing key mismatch : Changing the signing key can invalidate existing certificates. Never change the CA key without a migration plan. If accidentally changed, restore the original key from backup and restart the controller manager.
- CSR approval delay : If approval is manual, a backlog may develop. Consider using an auto-approver for well-known signers, such as kubelet client certificates, with proper RBAC restrictions.

Rollback: Since changes are often in controller manager flags, revert to the previous configuration to restore original behavior. Keep a backup of the manifest before editing. For cert-manager, roll back the Helm release or manifest changes.

Recovery checks: After rollback, verify that CSRs are being issued normally by creating a test CSR and checking that it reaches Issued state within the expected time. Monitor cluster health for API server errors.

## Operations Checklist

Use this checklist for regular CSR performance reviews and after any tuning changes:

- [ ] Verify Kubernetes version and controller manager flags.
- [ ] Measure baseline CSR latency and throughput.
- [ ] Identify CSR sources and volume patterns.
- [ ] Check for pending CSR backlog.
- [ ] Review API server and controller manager resource usage.
- [ ] Ensure certificate renewal mechanisms are functional.
- [ ] Test CSR issuance with a sample request.
- [ ] Monitor logs for signing errors.
- [ ] Document any tuning changes and their effects.
- [ ] Schedule periodic re-evaluation, especially after cluster upgrades or workload changes.

## Conclusion

Tuning Kubernetes CSR performance requires a methodical approach: establish a baseline, make incremental configuration changes, verify improvements, and have a rollback plan. By following the practical steps in this guide, you can reduce certificate issuance latency, increase throughput, and avoid common pitfalls. Remember to monitor continuously and adjust as your cluster evolves. With a well-tuned CSR pipeline, your Kubernetes security and operations will run more smoothly.