Kubernetes Certificate Signing Request Troubleshooting: A Practical Guide

Intro

Kubernetes Certificate Signing Request (CSR) troubleshooting is a core skill for operators managing secure clusters. CSRs are the mechanism by which nodes, services, and users obtain signed certificates from the Kubernetes control plane. When a CSR gets stuck in Pending, is rejected, or fails to produce a valid certificate, the impact ranges from a single pod failing to authenticate to an entire node being unable to join the cluster.

This guide provides a structured, hands-on approach to diagnosing and resolving CSR issues. It covers version and environment inventory, safe configuration paths, verification steps, common failure modes, and a final operations checklist. Every recommendation includes concrete commands, expected outputs, failure signals, and recovery decisions.

The target audience is developers, DevOps engineers, and technical startup teams who need to move from observed problem to verified resolution without guesswork. The goal is operational safety: observe before changing, limit blast radius, protect sensitive data, verify results, and document recovery paths.

Version and Environment Inventory

Before touching a CSR, you must know your Kubernetes version, the CSR lifecycle implementation, and the exact state of the cluster. Start with read-only observations. Record the output and timestamp. Never modify resources until you understand the blast radius.

Step 1: Identify Kubernetes Version and CSR API

CSR functionality has evolved. In Kubernetes v1.19+, the certificates.k8s.io/v1 API is stable. In older versions, v1beta1 may still be used. The signerName field is required in v1. Run:

kubectl version --short

Expected output (example):

Client Version: v1.27.3
Kustomize Version: v4.5.7
Server Version: v1.27.3

If the server version is below v1.19, you may be using v1beta1 CSRs. Check with:

kubectl api-versions | grep certificates.k8s.io

Expected:

certificates.k8s.io/v1

If only v1beta1 appears, plan to upgrade or use the appropriate API version in your manifests.

Step 2: Check Cluster Prerequisites

CSR approval requires the controller-manager to have signers configured. Run:

kubectl get clusterrolebinding system:node -o yaml | grep -A5 roleRef

You should see a binding to system:node ClusterRole. Also check the controller-manager logs for signer errors:

kubectl logs -n kube-system kube-controller-manager-<master-node> | grep -i csr

Replace <master-node> with your master node name. Note any errors about unrecognized signer names.

Step 3: Inventory Current CSRs

List all CSRs and their states:

kubectl get csr --sort-by=.metadata.creationTimestamp

Expected output (example):

NAME        AGE   SIGNERNAME                     REQUESTOR          CONDITION
csr-abcde   30m   kubernetes.io/kube-apiserver-client   system:admin      Approved,Issued
csr-fghij   5m    kubernetes.io/kubelet-serving    system:node:node1   Pending

Check detailed status of a pending CSR:

kubectl describe csr csr-fghij

Look for events and conditions. A Pending CSR with no events often means no approver is configured.

Step 4: Verify Signer Support

Each signer has specific requirements. List available signers in the cluster:

kubectl get --raw /apis/certificates.k8s.io/v1/signers

This may not be allowed in all clusters. Alternatively, check controller-manager flags:

kubectl get pods -n kube-system kube-controller-manager-<master-node> -o yaml | grep -A2 'cluster-signing-cert-file'

If no signing certificate is configured, CSRs cannot be signed. That is a critical environment issue.

Practical Check for Version and Environment Inventory

• Start with read-only commands: kubectl version, kubectl get csr.
• Capture current state to a file: kubectl get csr -o yaml > csr-backup-$(date +%Y%m%d%H%M%S).yaml.
• Protect credentials: never include --token or --certificate-authority in logs.
• Change one scoped item at a time, e.g., apply a single CSR manifest.

Safe Configuration Path

Modifying CSR-related configuration must follow a safe path: smallest change, verify, and rollback plan. This section covers creating, approving, and signing CSRs safely.

Step 1: Create a Test CSR with a Known Key

Generate a private key and CSR for a test user:

openssl req -new -newkey rsa:2048 -nodes -keyout test-user.key -out test-user.csr -subj "/CN=test-user/O=dev-team"

Encode the CSR in base64:

cat test-user.csr | base64 | tr -d '\n'

Create a Kubernetes CSR manifest test-user-csr.yaml:

apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: test-user-csr
spec:
  request: <BASE64_ENCODED_CSR>
  signerName: kubernetes.io/kube-apiserver-client
  usages:
  - client auth

Apply it:

kubectl apply -f test-user-csr.yaml

Verify:

kubectl get csr test-user-csr

Expected: Pending condition.

Step 2: Approve the CSR (Small Change)

Only approve if you intend to grant the certificate. Use kubectl certificate approve:

kubectl certificate approve test-user-csr

Expected output:

certificatesigningrequest.certificates.k8s.io/test-user-csr approved

Check status:

kubectl get csr test-user-csr

Expected:

NAME            AGE   SIGNERNAME                     REQUESTOR          CONDITION
test-user-csr   2m    kubernetes.io/kube-apiserver-client   system:admin      Approved,Issued

Step 3: Retrieve and Verify the Certificate

Get the issued certificate:

kubectl get csr test-user-csr -o jsonpath='{.status.certificate}' | base64 --decode > test-user.crt

Inspect it:

openssl x509 -in test-user.crt -noout -text

Check that Subject CN matches the requested user and validity dates are correct.

Step 4: Deny a CSR (If Needed)

To deny:

kubectl certificate deny test-user-csr

Expected:

certificatesigningrequest.certificates.k8s.io/test-user-csr denied

Denied CSRs cannot be approved later without deleting and recreating.

Practical Check for Safe Configuration Path

• Test in a non-production namespace first if possible.
• Use descriptive CSR names to avoid confusion.
• Keep private keys secure: store test-user.key in a secret manager, not in version control.
• Verify the issued certificate chain with openssl verify.

Verification and Diagnostics

After any change, verify that the expected state is reached. This section provides commands to confirm CSR health and diagnose common issues.

Step 1: Check CSR Conditions

A healthy CSR should have Approved and Issued conditions. Use:

kubectl get csr <csr-name> -o jsonpath='{.status.conditions}'

Expected output (example):

[{"lastUpdateTime":"2023-10-01T12:00:00Z","message":"This CSR was approved by kubectl certificate approve.","reason":"KubectlApprove","status":"True","type":"Approved"},{"lastUpdateTime":"2023-10-01T12:00:01Z","message":"Certificate fetched and issued successfully","reason":"CertificateFetched","status":"True","type":"Issued"}]

If only Approved is present but no Issued, the signer may have failed. Check controller-manager logs.

Step 2: Inspect CSR Details

kubectl describe csr <csr-name>

Look for events like:

Events:
  Type    Reason   Age   From          Message
  ----    ------   ----  ----          -------
  Normal  Approved 10m   kube-controller-manager  Approved
  Warning SigningError 10m  kube-controller-manager  Failed to sign: x509: unknown signer

A SigningError indicates the CSR's signerName is not supported by the controller.

Step 3: Check Signer Configuration

Verify the controller-manager has the correct signing flags:

kubectl get pod -n kube-system kube-controller-manager-<master-node> -o yaml | grep -E 'cluster-signing'

Expected lines:

- --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
- --cluster-signing-key-file=/etc/kubernetes/pki/ca.key

If these flags are missing, the controller cannot sign CSRs.

Step 4: Diagnose Pending CSRs Without Conditions

If a CSR remains Pending with no conditions, no approver is watching it. Check approval permissions:

kubectl auth can-i approve certificatesigningrequests --as=system:admin

For automated approval, ensure a controller like cert-manager is installed and has RBAC permissions.

Practical Check for Verification and Diagnostics

• Always compare current state with expected output.
• Use kubectl describe for events; they often contain the root cause.
• Check logs of kube-controller-manager for signer errors.
• Verify certificate validity with openssl x509 -dates -noout.

Failure Modes and Recovery

CSR failures can be categorized into a few common patterns. For each, we describe the symptom, diagnosis, and recovery steps.

Failure Mode 1: CSR Stuck in Pending (No Approver)

Symptom: CSR remains Pending indefinitely, no events.

Diagnosis:

kubectl get csr <name> -o yaml | grep -A5 conditions

If no conditions, no approver has acted.

Recovery:

• If manual approval is acceptable: kubectl certificate approve <name>.
• If automated approval is needed, deploy an approver (e.g., cert-manager) and configure RBAC.
• If the CSR is obsolete, delete it: kubectl delete csr <name>.

Failure Mode 2: CSR Rejected due to Invalid Usages or Signer

Symptom: CSR is denied with message indicating unknown signer or usages.

Diagnosis:

kubectl describe csr <name>

Look for reason: KubectlDeny with message.

Recovery:

• Correct the signerName and usages in the manifest.
• Delete the old CSR and create a new one:

kubectl delete csr <name>
# edit manifest, then reapply

Failure Mode 3: Certificate Issued but Invalid or Misconfigured

Symptom: CSR shows Issued, but the certificate does not work for authentication.

Diagnosis:

kubectl get csr <name> -o jsonpath='{.status.certificate}' | base64 --decode > cert.crt
openssl x509 -in cert.crt -noout -text

Check the following:

• Subject CN matches the intended identity.
• Extended Key Usage (EKU) matches client auth or server auth as needed.
• Validity period is not expired.
• Signer CA is trusted by the consuming component.

Recovery:

• Adjust usages or signer in the CSR spec.
• Recreate CSR and approve again.

Failure Mode 4: Node CSR Not Approved, Node Cannot Join Cluster

Symptom: New node's kubelet reports Failed to authenticate due to missing certificate.

Diagnosis: Check node CSRs:

kubectl get csr | grep node-

If node CSRs are Pending, approve them manually:

kubectl certificate approve <node-csr-name>

Recovery:

• Implement automatic approval for node CSRs using NodeRestriction admission plugin and a bootstrap token approver, or use kubeadm's kubelet bootstrap flow correctly.

Practical Check for Failure Modes and Recovery

• Always back up the CSR manifest before deletion: kubectl get csr <name> -o yaml > csr-backup.yaml.
• Document the recovery steps in your runbook.
• Test recovery in a staging environment first.

Operations Checklist

Use this checklist for every CSR-related operation to ensure consistency and safety.

Additional Safety Rules

• Always use placeholders in scripts, never real secrets.
• Limit approval to least-privilege accounts.
• Verify certificate expiry periodically with a monitoring tool.
• Test signer changes on a non-production cluster first.

Conclusion

Kubernetes CSR troubleshooting is not about memorizing commands; it is about a systematic process: observe, diagnose, change minimally, verify, and recover safely. By following the steps in this guide, you can resolve most CSR issues without escalating to cluster owners or causing downtime.

Start by inventorying your environment and understanding the CSR lifecycle. Use the safe configuration path to create and approve CSRs. Verify thoroughly with the provided commands. When failures occur, recognize the patterns and apply the documented recovery steps.

Finally, use the operations checklist as your daily driver. It ensures consistency, reduces human error, and builds confidence. Remember: a reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision.

As a next step, choose one low-risk CSR verification from the checklist, record the current state, run the documented check, and compare the result with the expected signal. Review dependencies such as authentication, service accounts, and TLS certificates as needed. Your future self will thank you.