E-NO
EN FR
Get Started
Kubernetes Certificates advanced concepts 7 Min Read

Kubernetes Certificates Advanced Concepts Explained with Practical Examples

calendar_today Published: 2026-09-05
update Last Updated: 2026-09-05
analytics SEO Efficiency: 100%
Technical guide illustration for Kubernetes Certificates Advanced Concepts Explained with Practical Examples.

Intro

Kubernetes certificates secure every control plane and worker node component. When certificates expire or are misconfigured, the cluster may become partially or completely unavailable. This article explains advanced certificate concepts with concrete commands, expected outputs, and recovery steps. You will learn how to inspect certificates, approve and sign Certificate Signing Requests (CSRs), configure kubelet client certificate rotation, set up mutual TLS (mTLS) for workloads, and troubleshoot common failures. The focus is on operational safety: observe before changing, limit the blast radius, and verify each step.

This guide is for developers, DevOps engineers, and technical startup teams running Kubernetes in production or preparing for it. You should have basic familiarity with kubectl and cluster architecture. All examples use Kubernetes v1.28 and OpenSSL 3.x, but the concepts apply to recent versions.

Version and Environment Inventory

Before touching certificates, gather the exact cluster version, certificate locations, and expiry dates. This read-only inventory prevents accidental changes and gives you a baseline for recovery.

Run the following commands and record the output:

kubectl version --short
# Client Version: v1.28.2
# Server Version: v1.28.2

List all certificates used by the control plane on a kubeadm cluster (default paths):

sudo ls -l /etc/kubernetes/pki/
# ca.crt, ca.key, apiserver.crt, apiserver.key, etc.

Check the expiry of the API server certificate:

sudo openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -dates
# notBefore=Jan  1 00:00:00 2024 GMT
# notAfter=Jan  1 00:00:00 2025 GMT

For kubelet client certificates on a worker node, inspect the kubelet config:

sudo cat /var/lib/kubelet/config.yaml | grep clientCAFile
# clientCAFile: /etc/kubernetes/pki/ca.crt

The kubelet's serving certificate is often auto-generated and stored in /var/lib/kubelet/pki/. Check its expiry and subject:

sudo openssl x509 -in /var/lib/kubelet/pki/kubelet-server-current.pem -noout -subject -dates
# subject=O = system:nodes, CN = system:node:worker-1
# notBefore=... notAfter=...

Record the current certificate serial numbers and fingerprint for later comparison:

sudo openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -serial -fingerprint -sha256

Safe Configuration Path

Certificate changes can lock you out of the cluster. Always work on a test cluster first, back up the existing PKI directory, and follow a minimal-change approach.

Backing Up Certificates

On each control plane node, create a timestamped backup of /etc/kubernetes/pki/:

sudo tar -czf /backup/kubernetes-pki-$(date +%Y%m%d%H%M%S).tar.gz -C /etc/kubernetes pki

For kubelet certificates on workers, back up /var/lib/kubelet/pki/ similarly.

Renewing a Control Plane Certificate with kubeadm

Kubeadm provides a safe renewal command. First, check which certificates are due for renewal:

sudo kubeadm certs check-expiration
# [apiserver] Certificate will expire on 2025-01-01
# [apiserver-etcd-client] Certificate will expire on 2025-01-01
# etc.

Renew all certificates at once (on each control plane node):

sudo kubeadm certs renew all

Restart the control plane components to pick up the new certificates:

sudo systemctl restart kubelet
# Wait a few seconds, then verify the API server is responsive
kubectl get nodes

If you only need to renew the API server certificate:

sudo kubeadm certs renew apiserver

After renewal, verify the new expiry date:

sudo openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -dates

Configuring Kubelet Client Certificate Rotation

Kubelet can automatically request new client certificates when they approach expiry. Enable rotation in the kubelet configuration file /var/lib/kubelet/config.yaml:

rotateCertificates: true
serverTLSBootstrap: true

Then restart the kubelet:

sudo systemctl restart kubelet

To approve the pending CSR automatically, you can configure the controller manager with --cluster-signing-duration=8760h (1 year) and set up an approver. For manual approval, see the next section.

Quick check 1 of 2

What is the purpose of the kube-controller-manager in the context of certificate signing?

According to the reference, the controller-manager is responsible for issuing actual signed certificates, while the apiserver receives and authenticates requests.

Verification and Diagnostics

After any certificate change, verify that all components are healthy and that certificates are valid.

Checking Certificate Expiry Across the Cluster

Use kubeadm certs check-expiration on control plane nodes, and for kubelet serving certificates, inspect them on each worker. A more automated approach is to use a tool like kube-cert-manager (deprecated) or a monitoring system with cert-exporter. For a quick check of all master certificates:

sudo kubeadm certs check-expiration
# Expected output: list of certificates with expiry dates

Verifying API Server Connectivity

From a workstation with kubectl, run:

kubectl get --raw='/healthz?verbose'
# [ok]etcd ok
# [ok]poststarthook/start-kube-apiserver-admission-initializer ok
# ...

If the API server certificate is invalid, you may see x509: certificate has expired or is not yet valid. Check the API server logs:

sudo journalctl -u kube-apiserver -n 50 --no-pager | grep -i certificate

Diagnosing CSR Approval Issues

When a node or user requests a certificate, a CertificateSigningRequest object is created. List pending CSRs:

kubectl get csr
# NAME        AGE   SIGNERNAME                     REQUESTOR          CONDITION
# node-csr-   10s   kubernetes.io/kube-apiserver-client-kubelet   system:node:worker-1   Pending

Inspect the CSR details:

kubectl describe csr node-csr-xxxx
# Events: ...

Approve the CSR after verifying the requestor and usage:

kubectl certificate approve node-csr-xxxx

If the CSR is denied, check the signer name and requestor. Common mistakes include missing RBAC permissions for the node bootstrap token or an incorrect signer name.

Failure Modes and Recovery

Certificates fail silently until they expire. Here are common failure scenarios and exact recovery steps.

Expired API Server Certificate

Symptoms: kubectl commands fail with Unable to connect to the server: x509: certificate has expired.

Recovery:

  1. On a control plane node, confirm the expiry:
sudo openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -dates
  1. Renew the certificate:
sudo kubeadm certs renew apiserver
  1. Restart the kubelet to trigger static pod restart:
sudo systemctl restart kubelet
  1. Verify connectivity:
kubectl get nodes

Kubelet Client Certificate Not Rotated

Symptoms: Kubelet logs show Failed to rotate client certificate and may stop posting status.

Check rotation settings:

sudo grep -E 'rotateCertificates|serverTLSBootstrap' /var/lib/kubelet/config.yaml

If rotateCertificates: false, set it to true, restart kubelet, and then check for a new CSR:

kubectl get csr | grep node-csr

Approve the CSR manually if no auto-approver is configured:

kubectl certificate approve <csr-name>

Verify the kubelet has a new client certificate:

sudo openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -dates

Workload mTLS Configuration Failure

If a pod fails to connect to a service with mTLS required, check the pod's mounted certificates. For example, an application expects /etc/tls/tls.crt and /etc/tls/tls.key:

kubectl exec -it <pod-name> -- ls -l /etc/tls/
# tls.crt, tls.key
kubectl exec -it <pod-name> -- openssl x509 -in /etc/tls/tls.crt -noout -subject -dates

If the certificate is for the wrong hostname, the connection will fail. Ensure the certificate's SANs include the service DNS name. You can check SANs with:

openssl x509 -in tls.crt -noout -ext subjectAltName

Implementing mTLS for a Workload

Mutual TLS (mTLS) verifies both client and server identities. In Kubernetes, this often involves creating a CA, issuing client and server certificates, and configuring the application to present them.

Creating a Private CA and Issuing Certificates with OpenSSL

  1. Generate a CA key and self-signed certificate:
openssl genrsa -out ca.key 2048
openssl req -x509 -new -nodes -key ca.key -sha256 -days 365 -out ca.crt -subj "/CN=my-ca"
  1. Generate a server key and certificate signing request (CSR) with SANs for the service DNS name:
openssl genrsa -out server.key 2048
openssl req -new -key server.key -out server.csr -subj "/CN=my-service.default.svc" -config <(cat <<EOF
[req]
distinguished_name=dn
[dn]
[SAN]
subjectAltName=DNS:my-service.default.svc,DNS:my-service.default.svc.cluster.local
EOF
)

Note: The exact syntax may vary; use a proper openssl config file for production. Sign the server certificate:

openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 365 -sha256 -extensions SAN -extfile <(echo "subjectAltName=DNS:my-service.default.svc,DNS:my-service.default.svc.cluster.local")
  1. Generate client certificate similarly, with a CN representing the client identity (e.g., my-client).

Storing Certificates in Kubernetes Secrets

Create a Secret containing the server certificate and key:

kubectl create secret tls my-service-tls --cert=server.crt --key=server.key

For the CA certificate that clients need to trust:

kubectl create secret generic my-ca --from-file=ca.crt

Mount the secret in your pod spec:

volumes:
- name: tls
  secret:
    secretName: my-service-tls
- name: ca
  secret:
    secretName: my-ca
containers:
- name: app
  volumeMounts:
  - name: tls
    mountPath: "/etc/tls"
    readOnly: true
  - name: ca
    mountPath: "/etc/ca"
    readOnly: true

Configure your application to use the server certificate and to require client certificates signed by your CA.

Quick check 2 of 2

What is the initial status of a CertificateSigningRequest (CSR) created by a kubelet?

The reference states that initially a CSR from a kubelet has a status of Pending.

Certificate Signing Request (CSR) API Deep Dive

The Kubernetes CSR API allows users and workloads to request certificates signed by the cluster's CA. Understanding the flow helps troubleshoot and automate.

CSR Lifecycle

  1. A client generates a key and CSR.
  2. The client submits a CertificateSigningRequest object to the Kubernetes API.
  3. An administrator or controller approves the CSR.
  4. The controller manager signs the CSR using the configured CA.
  5. The signed certificate is retrieved via the CSR's status.certificate field.

Inspecting a CSR Object

Get all CSRs:

kubectl get csr

View the CSR YAML:

kubectl get csr <name> -o yaml

Important fields:

  • spec.request: base64-encoded CSR.
  • spec.signerName: e.g., kubernetes.io/kube-apiserver-client.
  • spec.usages: e.g., client auth.
  • status.conditions: shows Approved, Denied, or Failed.

Approve or deny with:

kubectl certificate approve <name>
kubectl certificate deny <name>

To retrieve the signed certificate:

kubectl get csr <name> -o jsonpath='{.status.certificate}' | base64 -d > client.crt

Common CSR Signers

  • kubernetes.io/kube-apiserver-client: for user client certificates.
  • kubernetes.io/kube-apiserver-client-kubelet: for kubelet client certificates.
  • kubernetes.io/kubelet-serving: for kubelet serving certificates.

Ensure your cluster's controller manager has the appropriate signer flags:

--cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
--cluster-signing-key-file=/etc/kubernetes/pki/ca.key

Certificate Rotation Strategies

Beyond kubeadm, you may need to plan for rotation of CA certificates. Rotating the cluster CA is disruptive and requires careful orchestration.

Kubeadm CA Rotation (Manual)

Kubeadm does not support automatic CA rotation. To rotate the CA, you must:

  1. Generate a new CA.
  2. Re-issue all component certificates.
  3. Distribute the new CA to all nodes.
  4. Restart all components.

This is a high-risk operation. Always test on a non-production cluster and have a rollback plan. Some tools like kubeadm certs renew can help with component certs, but not the CA itself.

Kubelet Serving Certificate Rotation

Kubelet serving certificates are used for the kubelet's HTTPS endpoint (port 10250). They can be rotated automatically if the kubelet is bootstrapped with serverTLSBootstrap: true and CSRs are approved. The kubelet requests a new serving certificate when the current one is near expiry.

Monitor pending CSRs to ensure rotation happens:

kubectl get csr -o go-template='{{range .items}}{{.metadata.name}} {{.spec.signerName}} {{range .status.conditions}}{{.type}}={{.reason}}{{end}}{{"\n"}}{{end}}'

Operations Checklist

Use this checklist to ensure certificate hygiene and operational safety.

Daily/Weekly Checks

  • Run kubeadm certs check-expiration on all control plane nodes. If any certificate expires within 30 days, plan renewal.
  • Check for pending CSRs that may indicate node bootstrap issues: kubectl get csr | grep Pending.
  • Verify kubelet client certificates on worker nodes: sudo openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -dates.
  • Monitor API server logs for certificate errors: sudo journalctl -u kube-apiserver --since "1 hour ago" | grep -i certificate.

Pre-Renewal Checklist

  • Back up the PKI directory.
  • Document the current certificate serial numbers and fingerprints.
  • Identify all components affected by the renewal.
  • Schedule a maintenance window if downtime is expected.

Post-Renewal Verification

  • Run kubectl get nodes and kubectl get pods -n kube-system to confirm cluster health.
  • Check certificate dates with openssl x509 -noout -dates.
  • Verify kubelet rotation status by checking for new CSRs and approved conditions.
  • Test external access to the API server if using a load balancer.

Conclusion

Kubernetes certificates are foundational to cluster security and availability. By following the inventory, safe configuration, verification, and recovery procedures in this article, you can manage certificates confidently. Always observe before changing, back up critical files, and verify each step with concrete commands. Start with low-risk checks like listing certificate expiry dates and monitoring CSRs, then move to controlled renewals and rotations. A disciplined approach to certificate management prevents outages and keeps your cluster secure.

Related Research

Article Quality Score

Reader usefulness 100%
  • check_circle Reader-ready guide
  • check_circle Practical examples included
  • check_circle Clean SEO article URL