Kubernetes Certificate Signing Request Architecture Explained with Practical Examples

Intro

Every Kubernetes cluster that serves TLS traffic eventually hits a certificate problem. A pod cannot prove its identity, a kubelet cannot join the cluster, or a custom controller refuses to start because its serving certificate is missing. The error usually points to a Certificate Signing Request (CSR) that is stuck in Pending, was denied, or failed to reach the right signer.

This article explains the Kubernetes Certificate Signing Request architecture from an operator's perspective. You will learn how a CSR moves from a private key to a signed certificate, which components are involved, how to approve or deny requests safely, and how to diagnose failures without guessing. All examples use kubectl and are suitable for a development or test cluster first. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of real secrets when possible, verify the result, and know how to recover if the expected state is not reached.

Kubernetes CSRs are useful for developers, DevOps consultants, and technical startup teams that manage their own clusters or build custom admission controllers. By the end of this article, you will be able to create a CSR, inspect it, approve it, and troubleshoot common failure modes with confidence.

Version and Environment Inventory

Before you touch any CSR, you need to know your Kubernetes version, the certificate signer you plan to use, and the namespace or cluster scope where the request will live. CSRs are cluster-scoped resources; they are not bound to a namespace. The API version for CSRs changed over time, so a manifest that works on Kubernetes 1.19 may not work on 1.29 without adjustment.

Start with a read-only inventory:

kubectl version --short
kubectl get csr
kubectl get nodes -o wide

The first command shows the client and server versions. The second shows any existing CSRs and their status. The third shows the kubelet certificates and node states; if a node is NotReady because its kubelet client certificate expired, that is a CSR-related problem.

Prerequisites for creating and approving CSRs:

• A running Kubernetes cluster where you have permission to create certificates.k8s.io/v1 CertificateSigningRequest objects. By default, system:node-bootstrapper can create CSRs for node client certificates, but other users may need RBAC.
• openssl or cfssl installed locally to generate a private key and a certificate request.
• A signer that the cluster trusts. Kubernetes has built-in signers: kubernetes.io/kube-apiserver-client, kubernetes.io/kube-apiserver-client-kubelet, kubernetes.io/kubelet-serving, and kubernetes.io/legacy-unknown. The signer name is critical: if you request a signer that the controller manager does not recognize, the CSR will stay Pending forever.

Observation before intervention:

kubectl get csr -o custom-columns=NAME:.metadata.name,AGE:.metadata.creationTimestamp,SIGNER:.spec.signerName,REQUESTOR:.spec.username,CONDITION:.status.conditions[0].type

This custom columns command avoids dumping full YAML when you only need key fields. Look for the CONDITION column; a healthy CSR may be Approved or Issued, but Pending means no controller has acted yet.

Example: In a cluster running Kubernetes 1.27, the following command creates and shows a CSR:

openssl genrsa -out myuser.key 2048
openssl req -new -key myuser.key -out myuser.csr -subj "/CN=myuser/O=dev-team"
cat myuser.csr | base64 | tr -d '\n' > myuser.csr.b64

Then create the CSR manifest myuser-csr.yaml:

apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: myuser
spec:
  request: <BASE64_ENCODED_CSR>
  signerName: kubernetes.io/kube-apiserver-client
  usages:
  - client auth

Replace <BASE64_ENCODED_CSR> with the content of myuser.csr.b64. Apply it:

kubectl apply -f myuser-csr.yaml
kubectl get csr myuser

Expected output:

NAME     AGE   SIGNERNAME                            REQUESTOR   REQUESTEDDURATION   CONDITION
myuser   5s    kubernetes.io/kube-apiserver-client   admin       <none>              Pending

The requestor is admin because we used cluster-admin credentials. In a real bootstrap scenario, the requestor would be system:node:node-name or system:serviceaccount:....

Keep the local test small: before creating a CSR for a production service, test the workflow with a dummy user or a single node in a development cluster. Use kubectl apply --dry-run=client -f myuser-csr.yaml to validate the manifest without persisting it.

Safe Configuration Path

The safe configuration path for CSRs means:

1. Generate a private key locally and protect it. Do not store the private key in the CSR object; the CSR only contains the public key and the requested identity.
2. Use the correct signerName and usages for your use case. For a client certificate for a user or service account, use kubernetes.io/kube-apiserver-client with usage client auth. For a kubelet client certificate, use kubernetes.io/kube-apiserver-client-kubelet. For a serving certificate for a pod or service, use kubernetes.io/kubelet-serving with usage server auth. Misconfigured signers are the most common cause of stuck CSRs.
3. Approve CSRs only after verifying the request identity and the CSR content. Use kubectl get csr <name> -o yaml and check the spec.username, spec.groups, and spec.request (the base64 decoded CSR) against your records.
4. Set an expiration duration if your cluster supports it (Kubernetes 1.22+). You can specify spec.expirationSeconds to limit the lifetime of the certificate. This is important for security; a certificate that never expires is a liability.

Safe approval workflow:

# Decode and inspect the CSR subject and public key
kubectl get csr myuser -o jsonpath='{.spec.request}' | base64 -d | openssl req -noout -text

# Approve the CSR
kubectl certificate approve myuser

# Check status
kubectl get csr myuser
kubectl get csr myuser -o jsonpath='{.status.certificate}' | base64 -d | openssl x509 -noout -text

Expected output after approval:

NAME     AGE   SIGNERNAME                            REQUESTOR   REQUESTEDDURATION   CONDITION
myuser   10s   kubernetes.io/kube-apiserver-client   admin       <none>              Approved,Issued

The Issued condition means the certificate has been signed and stored in status.certificate. You can then extract it and use it for authentication.

Deny safely: If you see an unexpected CSR, deny it with kubectl certificate deny <name>. This adds a Denied condition and prevents certificate issuance. For example, if you see a CSR from an unknown user or with a suspicious subject, deny it immediately and investigate. You can later delete the CSR with kubectl delete csr <name>.

Automated approval: For large clusters, you can write a controller that watches CSRs and approves them based on rules. The Kubernetes documentation provides an example using a ClusterRole and a ClusterRoleBinding to allow a service account to approve CSRs. Always scope the permissions to the minimum necessary: a controller that approves any CSR is dangerous.

Example YAML for approving CSRs with RBAC:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: csr-approver
rules:
- apiGroups: ["certificates.k8s.io"]
  resources: ["certificatesigningrequests"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["certificates.k8s.io"]
  resources: ["certificatesigningrequests/approval"]
  verbs: ["update"]
- apiGroups: ["certificates.k8s.io"]
  resources: ["certificatesigningrequests/status"]
  verbs: ["patch"]

This role allows the bound service account to approve CSRs but not to create or delete them. Use this pattern for bootstrap controllers.

Verification and Diagnostics

After a CSR is approved, you need to verify that the certificate works for its intended purpose. Verification depends on the use case:

• User client certificate: Configure kubectl with the signed certificate and key, and test access:

kubectl config set-credentials myuser --client-certificate=myuser.crt --client-key=myuser.key --embed-certs=true
kubectl config set-context myuser-context --cluster=my-cluster --user=myuser
kubectl --context=myuser-context get pods

If the RBAC permissions are correct, you should see the pods list. If you see Error from server (Forbidden), check the roles bound to the user and the usages in the CSR.

• Kubelet client certificate: After approving a node's CSR, the kubelet should automatically renew its client certificate and become Ready. Check with:

kubectl get nodes
kubectl describe node <node-name>

Look for KubeletReady condition. If the node remains NotReady, inspect the kubelet logs for certificate errors.

• Serving certificate for a pod: If you are using a CSR to obtain a serving certificate for a webhook or metrics endpoint, you need to mount the signed certificate into the pod and configure the server to use it. Use a Secret to store the certificate and key, and mount it as a volume.

Diagnostics for stuck CSRs:

If a CSR stays Pending for more than a few minutes, check:

1. kubectl get csr <name> -o yaml and look for status.conditions with a Pending message. The message often indicates that no signer is configured.
2. kubectl get clusterrolebinding system:node-bootstrapper to ensure the bootstrapper role exists.
3. kubectl -n kube-system logs kube-controller-manager-<control-plane-node> and search for csr or certificate. The controller manager log may show an error like no signer found for signerName.
4. If the signer is custom, ensure the controller that implements the signer is running and has the necessary RBAC.
5. Check the spec.signerName against the list of known signers. Typos are common, e.g., kubernetes.io/kube-apiserver-client-kubelet vs kubernetes.io/kube-apiserver-client-kubelet. Copy-paste the exact name from documentation.

Common error messages and their meaning:

• the CSR is Pending — No controller has approved it; either you need to manually approve or the automatic approval mechanism is broken.
• failed to find signer — The signerName is not registered; check the spelling and the controller manager configuration.
• certificate request is not allowed — The signer does not permit the requested usages or the subject is invalid.
• x509: certificate signed by unknown authority — The client does not trust the CA that signed the certificate; ensure you use the correct CA bundle.

Verify the certificate content:

kubectl get csr myuser -o jsonpath='{.status.certificate}' | base64 -d | openssl x509 -noout -subject -issuer -dates

This shows the subject, issuer, and validity period. Compare the subject with the CSR request; it should match the CN and O fields.

Failure Modes and Recovery

Even with careful configuration, CSR workflows can fail. Here are the most common failure modes and recovery steps.

Failure 1: CSR denied by mistake

If you accidentally deny a CSR, you cannot un-deny it. You must create a new CSR with the same private key and submit it again. The private key is still valid; you just need a new CSR request.

Recovery:

# Generate a new CSR from the existing private key
openssl req -new -key myuser.key -out myuser-new.csr -subj "/CN=myuser/O=dev-team"
# Encode and create a new CSR object with a different name
kubectl apply -f myuser-new-csr.yaml
kubectl certificate approve myuser-new

Failure 2: Certificate expired

If a certificate has expired, the CSR will still show Issued but the certificate in status.certificate is expired. You need to create a new CSR and approve it before the old certificate expires. For kubelet client certificates, the kubelet automatically renews them if the CSR approver is working. For user certificates, you must manually renew.

Check expiration:

kubectl get csr myuser -o jsonpath='{.status.certificate}' | base64 -d | openssl x509 -noout -enddate

Failure 3: Private key lost

If you lose the private key, the signed certificate is useless. You must generate a new key pair and submit a new CSR. For kubelet certificates, this might mean re-joining the node to the cluster. For user certificates, revoke the old certificate if possible and issue a new one.

Failure 4: CSR never issued because of missing usages

Some signers require specific usages. For example, the kubernetes.io/kube-apiserver-client signer may require client auth usage. If you omit it, the CSR might be approved but the certificate is not issued. Add the correct usages and resubmit.

Failure 5: Controller manager down

If the controller manager is not running, no CSR can be approved or signed. Check the controller manager pod status:

kubectl -n kube-system get pods -l component=kube-controller-manager
kubectl -n kube-system logs <controller-manager-pod> | grep -i csr

If the controller manager is down, restart it. On managed clusters, contact your provider. On self-managed clusters, ensure the static pod manifest is correct and the kubelet is running.

Recovery best practices:

• Always keep a copy of the private key in a secure location (e.g., a secrets manager). Do not store it in the cluster.
• Set spec.expirationSeconds to a reasonable value (e.g., 8760h for one year) to avoid indefinite certificates.
• Use monitoring and alerting to detect CSRs that have been Pending for more than a few minutes. You can write a simple script or Prometheus alert to watch kube_certificatesigningrequest_created and kube_certificatesigningrequest_condition metrics.
• Document your approval workflow and ensure that only authorized users can approve CSRs. Use RBAC to limit certificatesigningrequests/approval to a small group.

Operations Checklist

Use this checklist for every CSR-related task.

Before creating a CSR:

• [ ] Confirm the Kubernetes version and API group: kubectl api-versions | grep certificates.
• [ ] Identify the signer name for your use case. Check kubectl get clusterrolebinding and controller manager flags for existing signers.
• [ ] Generate a strong private key (RSA 2048 or ECDSA P-256) and store it securely.
• [ ] Prepare the CSR with the correct subject (CN, O) and usages.
• [ ] Validate the manifest with kubectl apply --dry-run=client.

After creating a CSR:

• [ ] Check status: kubectl get csr <name>; it should be Pending.
• [ ] Inspect the CSR request: decode and verify the subject and public key.
• [ ] Approve only if the request matches an expected identity.
• [ ] Observe the status change to Approved,Issued.
• [ ] Extract the certificate and verify its validity with openssl x509.

When renewing:

• [ ] Check the expiration of the current certificate.
• [ ] Generate a new CSR with the same or a new key.
• [ ] Submit and approve.
• [ ] Update all places that use the certificate (kubeconfig, service mesh, ingress).
• [ ] Test access before removing the old certificate.

On failure:

• [ ] Collect logs: controller manager, kubelet, API server.
• [ ] Check for RBAC issues: kubectl auth can-i create certificatesigningrequests.
• [ ] Review signer configuration and usages.
• [ ] If denied, create a new CSR and document why the denial happened.
• [ ] If the cluster is in a bad state, consider cordoning the node or scaling down the service until the certificate is fixed.

Example script for monitoring pending CSRs:

#!/bin/bash
# Alert if any CSR is pending for more than 10 minutes
PENDING=$(kubectl get csr -o json | jq '[.items[] | select(.status.conditions[]?.type=="Pending")] | length')
if [ "$PENDING" -gt 0 ]; then
  echo "Warning: $PENDING pending CSRs"
  kubectl get csr
fi

Run this script periodically via cron or a Kubernetes CronJob to catch stuck CSRs early.

Conclusion

Kubernetes Certificate Signing Request architecture is not just a feature; it is a workflow that touches security, automation, and operations. When you understand how a CSR moves from a private key to a signed certificate, you can troubleshoot failures quickly and avoid security holes.

By following the version and environment inventory, safe configuration path, verification and diagnostics, failure modes and recovery, and the operations checklist, you create a repeatable process. Each step is observable, reversible where possible, and limited in blast radius.

As a next step, choose one low-risk verification in your own cluster. Create a test CSR for a dummy user, approve it, extract the certificate, and use it to access the cluster. Then test a denial and recovery. Record your findings and update your team's runbook.

A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. With this guide, you have the commands and examples to manage Kubernetes CSRs with confidence.