## Intro

Kubernetes certificates secure every control plane and worker node component. When certificates expire or are misconfigured, the cluster may become partially or completely unavailable. This article explains advanced certificate concepts with concrete commands, expected outputs, and recovery steps. You will learn how to inspect certificates, approve and sign Certificate Signing Requests (CSRs), configure kubelet client certificate rotation, set up mutual TLS (mTLS) for workloads, and troubleshoot common failures. The focus is on operational safety: observe before changing, limit the blast radius, and verify each step.

This guide is for developers, DevOps engineers, and technical startup teams running Kubernetes in production or preparing for it. You should have basic familiarity with kubectl and cluster architecture. All examples use Kubernetes v1.28 and OpenSSL 3.x, but the concepts apply to recent versions.

## Version and Environment Inventory

Before touching certificates, gather the exact cluster version, certificate locations, and expiry dates. This read-only inventory prevents accidental changes and gives you a baseline for recovery.

Run the following commands and record the output:

```bash
kubectl version --short
# Client Version: v1.28.2
# Server Version: v1.28.2
```

List all certificates used by the control plane on a kubeadm cluster (default paths):

```bash
sudo ls -l /etc/kubernetes/pki/
# ca.crt, ca.key, apiserver.crt, apiserver.key, etc.
```

Check the expiry of the API server certificate:

```bash
sudo openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -dates
# notBefore=Jan  1 00:00:00 2024 GMT
# notAfter=Jan  1 00:00:00 2025 GMT
```

For kubelet client certificates on a worker node, inspect the kubelet config:

```bash
sudo cat /var/lib/kubelet/config.yaml | grep clientCAFile
# clientCAFile: /etc/kubernetes/pki/ca.crt
```

The kubelet's serving certificate is often auto-generated and stored in `/var/lib/kubelet/pki/`. Check its expiry and subject:

```bash
sudo openssl x509 -in /var/lib/kubelet/pki/kubelet-server-current.pem -noout -subject -dates
# subject=O = system:nodes, CN = system:node:worker-1
# notBefore=... notAfter=...
```

Record the current certificate serial numbers and fingerprint for later comparison:

```bash
sudo openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -serial -fingerprint -sha256
```

## Safe Configuration Path

Certificate changes can lock you out of the cluster. Always work on a test cluster first, back up the existing PKI directory, and follow a minimal-change approach.

### Backing Up Certificates

On each control plane node, create a timestamped backup of `/etc/kubernetes/pki/`:

```bash
sudo tar -czf /backup/kubernetes-pki-$(date +%Y%m%d%H%M%S).tar.gz -C /etc/kubernetes pki
```

For kubelet certificates on workers, back up `/var/lib/kubelet/pki/` similarly.

### Renewing a Control Plane Certificate with kubeadm

Kubeadm provides a safe renewal command. First, check which certificates are due for renewal:

```bash
sudo kubeadm certs check-expiration
# [apiserver] Certificate will expire on 2025-01-01
# [apiserver-etcd-client] Certificate will expire on 2025-01-01
# etc.
```

Renew all certificates at once (on each control plane node):

```bash
sudo kubeadm certs renew all
```

Restart the control plane components to pick up the new certificates:

```bash
sudo systemctl restart kubelet
# Wait a few seconds, then verify the API server is responsive
kubectl get nodes
```

If you only need to renew the API server certificate:

```bash
sudo kubeadm certs renew apiserver
```

After renewal, verify the new expiry date:

```bash
sudo openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -dates
```

### Configuring Kubelet Client Certificate Rotation

Kubelet can automatically request new client certificates when they approach expiry. Enable rotation in the kubelet configuration file `/var/lib/kubelet/config.yaml`:

```yaml
rotateCertificates: true
serverTLSBootstrap: true
```

Then restart the kubelet:

```bash
sudo systemctl restart kubelet
```

To approve the pending CSR automatically, you can configure the controller manager with `--cluster-signing-duration=8760h` (1 year) and set up an approver. For manual approval, see the next section.

## Verification and Diagnostics

After any certificate change, verify that all components are healthy and that certificates are valid.

### Checking Certificate Expiry Across the Cluster

Use `kubeadm certs check-expiration` on control plane nodes, and for kubelet serving certificates, inspect them on each worker. A more automated approach is to use a tool like `kube-cert-manager` (deprecated) or a monitoring system with cert-exporter. For a quick check of all master certificates:

```bash
sudo kubeadm certs check-expiration
# Expected output: list of certificates with expiry dates
```

### Verifying API Server Connectivity

From a workstation with kubectl, run:

```bash
kubectl get --raw='/healthz?verbose'
# [ok]etcd ok
# [ok]poststarthook/start-kube-apiserver-admission-initializer ok
# ...
```

If the API server certificate is invalid, you may see `x509: certificate has expired or is not yet valid`. Check the API server logs:

```bash
sudo journalctl -u kube-apiserver -n 50 --no-pager | grep -i certificate
```

### Diagnosing CSR Approval Issues

When a node or user requests a certificate, a CertificateSigningRequest object is created. List pending CSRs:

```bash
kubectl get csr
# NAME        AGE   SIGNERNAME                     REQUESTOR          CONDITION
# node-csr-   10s   kubernetes.io/kube-apiserver-client-kubelet   system:node:worker-1   Pending
```

Inspect the CSR details:

```bash
kubectl describe csr node-csr-xxxx
# Events: ...
```

Approve the CSR after verifying the requestor and usage:

```bash
kubectl certificate approve node-csr-xxxx
```

If the CSR is denied, check the signer name and requestor. Common mistakes include missing RBAC permissions for the node bootstrap token or an incorrect signer name.

## Failure Modes and Recovery

Certificates fail silently until they expire. Here are common failure scenarios and exact recovery steps.

### Expired API Server Certificate

Symptoms: `kubectl` commands fail with `Unable to connect to the server: x509: certificate has expired`.

Recovery:

1. On a control plane node, confirm the expiry:

```bash
sudo openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -dates
```

2. Renew the certificate:

```bash
sudo kubeadm certs renew apiserver
```

3. Restart the kubelet to trigger static pod restart:

```bash
sudo systemctl restart kubelet
```

4. Verify connectivity:

```bash
kubectl get nodes
```

### Kubelet Client Certificate Not Rotated

Symptoms: Kubelet logs show `Failed to rotate client certificate` and may stop posting status.

Check rotation settings:

```bash
sudo grep -E 'rotateCertificates|serverTLSBootstrap' /var/lib/kubelet/config.yaml
```

If `rotateCertificates: false`, set it to `true`, restart kubelet, and then check for a new CSR:

```bash
kubectl get csr | grep node-csr
```

Approve the CSR manually if no auto-approver is configured:

```bash
kubectl certificate approve <csr-name>
```

Verify the kubelet has a new client certificate:

```bash
sudo openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -dates
```

### Workload mTLS Configuration Failure

If a pod fails to connect to a service with mTLS required, check the pod's mounted certificates. For example, an application expects `/etc/tls/tls.crt` and `/etc/tls/tls.key`:

```bash
kubectl exec -it <pod-name> -- ls -l /etc/tls/
# tls.crt, tls.key
kubectl exec -it <pod-name> -- openssl x509 -in /etc/tls/tls.crt -noout -subject -dates
```

If the certificate is for the wrong hostname, the connection will fail. Ensure the certificate's SANs include the service DNS name. You can check SANs with:

```bash
openssl x509 -in tls.crt -noout -ext subjectAltName
```

## Implementing mTLS for a Workload

Mutual TLS (mTLS) verifies both client and server identities. In Kubernetes, this often involves creating a CA, issuing client and server certificates, and configuring the application to present them.

### Creating a Private CA and Issuing Certificates with OpenSSL

1. Generate a CA key and self-signed certificate:

```bash
openssl genrsa -out ca.key 2048
openssl req -x509 -new -nodes -key ca.key -sha256 -days 365 -out ca.crt -subj "/CN=my-ca"
```

2. Generate a server key and certificate signing request (CSR) with SANs for the service DNS name:

```bash
openssl genrsa -out server.key 2048
openssl req -new -key server.key -out server.csr -subj "/CN=my-service.default.svc" -config <(cat <<EOF
[req]
distinguished_name=dn
[dn]
[SAN]
subjectAltName=DNS:my-service.default.svc,DNS:my-service.default.svc.cluster.local
EOF
)
```

Note: The exact syntax may vary; use a proper openssl config file for production. Sign the server certificate:

```bash
openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 365 -sha256 -extensions SAN -extfile <(echo "subjectAltName=DNS:my-service.default.svc,DNS:my-service.default.svc.cluster.local")
```

3. Generate client certificate similarly, with a CN representing the client identity (e.g., `my-client`).

### Storing Certificates in Kubernetes Secrets

Create a Secret containing the server certificate and key:

```bash
kubectl create secret tls my-service-tls --cert=server.crt --key=server.key
```

For the CA certificate that clients need to trust:

```bash
kubectl create secret generic my-ca --from-file=ca.crt
```

Mount the secret in your pod spec:

```yaml
volumes:
- name: tls
  secret:
    secretName: my-service-tls
- name: ca
  secret:
    secretName: my-ca
containers:
- name: app
  volumeMounts:
  - name: tls
    mountPath: "/etc/tls"
    readOnly: true
  - name: ca
    mountPath: "/etc/ca"
    readOnly: true
```

Configure your application to use the server certificate and to require client certificates signed by your CA.

## Certificate Signing Request (CSR) API Deep Dive

The Kubernetes CSR API allows users and workloads to request certificates signed by the cluster's CA. Understanding the flow helps troubleshoot and automate.

### CSR Lifecycle

1. A client generates a key and CSR.
2. The client submits a `CertificateSigningRequest` object to the Kubernetes API.
3. An administrator or controller approves the CSR.
4. The controller manager signs the CSR using the configured CA.
5. The signed certificate is retrieved via the CSR's `status.certificate` field.

### Inspecting a CSR Object

Get all CSRs:

```bash
kubectl get csr
```

View the CSR YAML:

```bash
kubectl get csr <name> -o yaml
```

Important fields:
- `spec.request`: base64-encoded CSR.
- `spec.signerName`: e.g., `kubernetes.io/kube-apiserver-client`.
- `spec.usages`: e.g., `client auth`.
- `status.conditions`: shows `Approved`, `Denied`, or `Failed`.

Approve or deny with:

```bash
kubectl certificate approve <name>
kubectl certificate deny <name>
```

To retrieve the signed certificate:

```bash
kubectl get csr <name> -o jsonpath='{.status.certificate}' | base64 -d > client.crt
```

### Common CSR Signers

- `kubernetes.io/kube-apiserver-client`: for user client certificates.
- `kubernetes.io/kube-apiserver-client-kubelet`: for kubelet client certificates.
- `kubernetes.io/kubelet-serving`: for kubelet serving certificates.

Ensure your cluster's controller manager has the appropriate signer flags:

```bash
--cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
--cluster-signing-key-file=/etc/kubernetes/pki/ca.key
```

## Certificate Rotation Strategies

Beyond kubeadm, you may need to plan for rotation of CA certificates. Rotating the cluster CA is disruptive and requires careful orchestration.

### Kubeadm CA Rotation (Manual)

Kubeadm does not support automatic CA rotation. To rotate the CA, you must:

1. Generate a new CA.
2. Re-issue all component certificates.
3. Distribute the new CA to all nodes.
4. Restart all components.

This is a high-risk operation. Always test on a non-production cluster and have a rollback plan. Some tools like `kubeadm certs renew` can help with component certs, but not the CA itself.

### Kubelet Serving Certificate Rotation

Kubelet serving certificates are used for the kubelet's HTTPS endpoint (port 10250). They can be rotated automatically if the kubelet is bootstrapped with `serverTLSBootstrap: true` and CSRs are approved. The kubelet requests a new serving certificate when the current one is near expiry.

Monitor pending CSRs to ensure rotation happens:

```bash
kubectl get csr -o go-template='{{range .items}}{{.metadata.name}} {{.spec.signerName}} {{range .status.conditions}}{{.type}}={{.reason}}{{end}}{{"\n"}}{{end}}'
```

## Operations Checklist

Use this checklist to ensure certificate hygiene and operational safety.

### Daily/Weekly Checks

- Run `kubeadm certs check-expiration` on all control plane nodes. If any certificate expires within 30 days, plan renewal.
- Check for pending CSRs that may indicate node bootstrap issues: `kubectl get csr | grep Pending`.
- Verify kubelet client certificates on worker nodes: `sudo openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -dates`.
- Monitor API server logs for certificate errors: `sudo journalctl -u kube-apiserver --since "1 hour ago" | grep -i certificate`.

### Pre-Renewal Checklist

- Back up the PKI directory.
- Document the current certificate serial numbers and fingerprints.
- Identify all components affected by the renewal.
- Schedule a maintenance window if downtime is expected.

### Post-Renewal Verification

- Run `kubectl get nodes` and `kubectl get pods -n kube-system` to confirm cluster health.
- Check certificate dates with `openssl x509 -noout -dates`.
- Verify kubelet rotation status by checking for new CSRs and approved conditions.
- Test external access to the API server if using a load balancer.

## Conclusion

Kubernetes certificates are foundational to cluster security and availability. By following the inventory, safe configuration, verification, and recovery procedures in this article, you can manage certificates confidently. Always observe before changing, back up critical files, and verify each step with concrete commands. Start with low-risk checks like listing certificate expiry dates and monitoring CSRs, then move to controlled renewals and rotations. A disciplined approach to certificate management prevents outages and keeps your cluster secure.