## Intro

Automating Kubernetes Certificate Signing Request (CSR) approval in CI/CD pipelines is a common need for teams that manage their own TLS certificates or integrate with external PKI. Without a controlled workflow, operators either approve CSRs manually (slow and error-prone) or leave them pending indefinitely, breaking application deployments.

This article provides practical guidance for developers, DevOps consultants, and technical startup teams who want to automate CSR approval safely. It covers the lifecycle of a CSR in Kubernetes, how to configure RBAC for automation, how to build a CI/CD step that approves CSRs, how to verify the issued certificate, and how to handle failures and rollbacks.

Throughout, we emphasize operational safety: observe before changing, limit the blast radius, protect sensitive values, verify results, and document recovery paths.

## Version and Environment Inventory

Before automating CSR approval, establish the exact Kubernetes version and environment details. The behavior of CertificateSigningRequest resources changed over releases, especially around signer names and approval permissions.

Key prerequisites:

- A Kubernetes cluster (v1.19 or later recommended; signer-specific approval introduced earlier but matured later)
- kubectl configured with appropriate permissions to inspect CSRs
- If using a custom signer, ensure it is installed and configured (e.g., cert-manager, kube-controller-manager's built-in signer, or an external CA)
- A service account for CI/CD automation with minimal RBAC

Start with read-only observation:

```bash
kubectl version --short
kubectl get csr
```

Expected output includes the cluster version and a list of CSRs with their conditions. For example:

```
NAME        AGE   SIGNERNAME                     REQUESTOR          CONDITION
my-app-csr  2m    kubernetes.io/kube-apiserver-client   system:serviceaccount:default:my-app   Pending
```

Note the `signerName`. Modern clusters use signer names like `kubernetes.io/kube-apiserver-client`, `kubernetes.io/kubelet-serving`, or custom ones. Automation must match the signer to the intended use; approving a CSR for the wrong signer can issue a certificate with unintended permissions.

Check component health:

```bash
kubectl get pods -n kube-system | grep -E 'kube-controller-manager|kube-apiserver'
```

If the controller manager that runs the CSR approver is unhealthy, automation may appear to run but not complete.

For local testing, use a single-node cluster (kind, minikube) and apply one CSR manifest at a time. Verify with:

```bash
kubectl get csr my-csr -o yaml
```

Before moving to production, confirm the cluster's certificate API is enabled (it is by default) and that the API server has `--cluster-signing-cert-file` and `--cluster-signing-key-file` set if using the built-in signer.

## Safe Configuration Path

Automating CSR approval requires careful RBAC configuration. The principle of least privilege is critical: a CI/CD service account should be able to approve only CSRs with specific signer names and, ideally, only for its namespace or a known naming pattern.

### Step 1: Create a dedicated ServiceAccount

```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: csr-approver
  namespace: ci-cd
```

Apply it:

```bash
kubectl apply -f sa.yaml
```

### Step 2: Define a ClusterRole with scoped approval permissions

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: csr-approver
rules:
- apiGroups: ["certificates.k8s.io"]
  resources: ["certificatesigningrequests/approval"]
  verbs: ["update"]
- apiGroups: ["certificates.k8s.io"]
  resources: ["certificatesigningrequests"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["certificates.k8s.io"]
  resources: ["certificatesigningrequests/status"]
  verbs: ["patch", "update"]
```

This allows viewing CSRs and updating the approval subresource, which is distinct from editing the CSR itself.

### Step 3: Bind the ClusterRole to the ServiceAccount with a resourceName restriction if possible

If you can predict CSR names (e.g., generated from a deployment name), use resourceNames to restrict approval to specific CSRs:

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: csr-approver-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: csr-approver
subjects:
- kind: ServiceAccount
  name: csr-approver
  namespace: ci-cd
```

Without resourceNames, the account can approve any CSR with the allowed verbs, which is broad but sometimes necessary if names are dynamic. An alternative is to use a validating webhook to enforce naming patterns.

Apply and verify:

```bash
kubectl apply -f rbac.yaml
kubectl auth can-i approve certificatesigningrequests --as=system:serviceaccount:ci-cd:csr-approver
```

Expected output: `yes` if permissions are correct.

Test the smallest change: create a test CSR, then approve it using the service account's credentials. Use `kubectl --as` to impersonate without creating a kubeconfig file:

```bash
kubectl create csr test-csr --signer kubernetes.io/kube-apiserver-client --requestor test
kubectl certificate approve test-csr --as=system:serviceaccount:ci-cd:csr-approver
```

If the approval succeeds, configuration is valid.

Keep sensitive data out of CI/CD: use short-lived tokens or workload identity instead of static kubeconfig files with long-lived credentials.

## Verification and Diagnostics

After implementing automation, verify that CSRs are approved correctly and that certificates are issued.

### Inspect a CSR before approval

```bash
kubectl get csr my-csr -o yaml
```

Example condensed output:

```yaml
spec:
  signerName: kubernetes.io/kube-apiserver-client
  usages:
  - client auth
  request: LS0tLS1CRUdJTiBDRV...
status:
  conditions: []
```

Check the `usages` and `signerName` match expectations. A client auth CSR should not be approved for server auth.

### Approve via kubectl

```bash
kubectl certificate approve my-csr
```

Expected output:

```
certificatesigningrequest.certificates.k8s.io/my-csr approved
```

### Check status

```bash
kubectl get csr my-csr
```

If approved and signed, the CSR will show `Approved,Issued` under CONDITIONS.

### Retrieve the issued certificate

```bash
kubectl get csr my-csr -o jsonpath='{.status.certificate}' | base64 -d > client.crt
```

Verify the certificate details:

```bash
openssl x509 -in client.crt -text -noout
```

Look for `Subject`, `Issuer`, `Validity`, and `X509v3 Extended Key Usage` to confirm the certificate matches the request.

### Diagnose stuck CSRs

If a CSR remains `Pending`, check events:

```bash
kubectl describe csr my-csr
```

Common causes:

- No approver with permission to approve that signer
- The approver is not running (if using a custom controller)
- The signer's certificate/key misconfiguration

For CI/CD pipelines, log the CSR name and status after approval attempt:

```bash
STATUS=$(kubectl get csr my-csr -o jsonpath='{.status.conditions[?(@.type=="Approved")].status}')
if [ "$STATUS" != "True" ]; then
  echo "CSR not approved: $STATUS"
  exit 1
fi
```

This prevents pipeline continuation with an unsigned certificate.

## Failure Modes and Recovery

Automated CSR approval can fail in several ways. Understanding each helps design recovery.

### Failure 1: CSR is approved but certificate not issued

Symptom: CSR shows `Approved` but not `Issued`.

Cause: The signing controller (kube-controller-manager's CSR signer) is not running or its signing key is missing.

Diagnosis:

```bash
kubectl get pods -n kube-system | grep kube-controller-manager
kubectl logs -n kube-system kube-controller-manager-<node> | grep -i csr
```

Recovery: Ensure the controller manager has `--cluster-signing-cert-file` and `--cluster-signing-key-file` configured and the files exist. Restart the controller if needed.

### Failure 2: Automated approver cannot approve CSR due to RBAC

Symptom: Pipeline logs show `error: You must be logged in to the server (Unauthorized)` or `forbidden`.

Cause: Missing RBAC rules or service account token expired.

Diagnosis:

```bash
kubectl auth can-i approve certificatesigningrequests --as=system:serviceaccount:ci-cd:csr-approver
```

Recovery: Review the ClusterRole and binding. Ensure the service account is in the correct namespace.

### Failure 3: Wrong signer approved

Symptom: The issued certificate cannot be used for its intended purpose (e.g., client cert used as server cert).

Cause: Automation approved a CSR with a signer name that does not match the application's needs.

Recovery: Deny or delete the CSR and generate a new one with the correct signer:

```bash
kubectl delete csr bad-csr
```

Prevent by validating signer names in the automation script before approval.

### Failure 4: Pipeline approves CSRs too broadly

Symptom: Unauthorized CSRs get approved automatically, potentially issuing credentials to attackers.

Cause: Overly permissive RBAC (no resourceNames, no signer filtering).

Recovery: Immediately revoke the binding, remove the service account token, and audit which CSRs were approved. Tighten RBAC to specific signer names and resource names.

Rollback for configuration changes: keep all YAML files in version control. If a change causes failures, revert to the previous commit and reapply.

Example rollback:

```bash
git revert <commit-hash>
kubectl apply -f rbac.yaml
```

Always verify the new state with `kubectl auth can-i` and a test CSR.

## Operations Checklist

Use this checklist before and after deploying CSR automation to production.

Before deployment:

- [ ] Cluster version supported (v1.19+ for signer-specific approval)
- [ ] ServiceAccount created in a dedicated namespace
- [ ] ClusterRole scoped to `certificatesigningrequests/approval` and necessary read verbs
- [ ] ClusterRoleBinding ties the ServiceAccount to the ClusterRole
- [ ] If possible, resourceNames restrict approval to known CSR names
- [ ] Test with `kubectl auth can-i` as the service account
- [ ] Test with a sample CSR end-to-end in a staging environment
- [ ] Pipeline script logs CSR name and status
- [ ] Rollback plan documented (revert commit, reapply RBAC)

During operation:

- [ ] Monitor for unexpectedly approved CSRs (audit logs)
- [ ] Rotate service account tokens periodically (if using static tokens) or use workload identity
- [ ] Keep signing certificates valid and renew before expiry
- [ ] Regularly review RBAC permissions for least privilege

After failure recovery:

- [ ] Identify root cause and document
- [ ] Apply corrective action to automation
- [ ] Re-run test CSR to confirm fix
- [ ] Update operations runbook

Assign an owner: for example, the platform lead (e.g., Priya Shah) owns the CSR automation configuration and reviews it monthly or after any infrastructure change.

## Common Pitfalls

### Pitfall 1: Approving CSRs manually as a habit

Why it happens: Automation not set up or not trusted, so teams fall back to manual approval.

How to avoid: Implement CI/CD approval with logging and alerts. If manual approval remains, restrict it to break-glass accounts.

### Pitfall 2: Using cluster-admin for the CI/CD service account

Why it happens: Quick and dirty setup, or lack of understanding of RBAC.

How to avoid: Use the scoped ClusterRole as shown above. Test with `kubectl auth can-i --list` to see all permissions.

### Pitfall 3: Ignoring signer names

Why it happens: Older clusters defaulted to a single signer, so teams assume no distinction.

How to avoid: Always inspect `spec.signerName` and match approval logic to the expected signer.

### Pitfall 4: Not verifying the issued certificate

Why it happens: Pipeline stops after approval, assuming issuance succeeded.

How to avoid: Check for `Issued` condition and decode the certificate to validate usages and expiry.

### Pitfall 5: Storing long-lived kubeconfig in CI/CD

Why it happens: Simplicity.

How to avoid: Use short-lived tokens, OIDC, or cloud workload identity. Rotate secrets automatically.

## Conclusion

Automating Kubernetes CSR approval in CI/CD is achievable with careful RBAC, clear verification steps, and robust failure handling. The key is to scope permissions tightly, validate signer names and usages, and ensure the signing controller is healthy.

Start with a low-risk implementation: create a test CSR, approve it via a restricted service account, and verify the certificate. Then integrate the approval step into your pipeline with appropriate logging and error handling.

By following the practices in this article, you can reduce manual toil, prevent misissued certificates, and maintain a secure and reliable certificate issuance process in your Kubernetes clusters.