Kubernetes Certificates Security Hardening: A Practical Guide

Intro

Certificates are the backbone of trust in Kubernetes. They authenticate users, service accounts, and control plane components, and they encrypt communication between the API server, kubelet, etcd, and other critical services. A misconfigured or expired certificate can bring your cluster to a halt or, worse, expose it to man-in-the-middle attacks. Yet many operators treat certificates as a set-and-forget configuration.

This guide is for developers, DevOps consultants, and technical startup teams who need to harden Kubernetes certificate security with practical, verifiable steps. We will connect Kubernetes certificate hardening, access control, secrets management, and permissions to concrete commands, expected outputs, failure signals, and recovery decisions.

The operational goal is safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached. Every step assumes a running Kubernetes cluster (version 1.20+ recommended) with kubectl configured and, for some tasks, kubeadm available. We focus on self-managed clusters, but the principles apply to managed offerings as well.

Version and Environment Inventory

Before touching any certificate, you must know your environment. Inventory the Kubernetes version, the certificate authorities (CAs) in use, and where certificates are stored.

Identify the cluster version and certificate locations

Run:

kubectl version --short

Expected output (example for v1.27):

Client Version: v1.27.3
Server Version: v1.27.3

If your cluster was set up with kubeadm, certificates are stored in /etc/kubernetes/pki on control plane nodes. List them:

sudo ls -la /etc/kubernetes/pki

You should see files like ca.crt, ca.key, apiserver.crt, apiserver.key, front-proxy-ca.crt, etcd/ca.crt, etc.

Check certificate expiry with kubeadm:

sudo kubeadm certs check-expiration

Sample output:

[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Oct 05, 2024 13:02 UTC   364d            ca                      no
apiserver                  Oct 05, 2024 13:02 UTC   364d            ca                      no
apiserver-etcd-client      Oct 05, 2024 13:02 UTC   364d            etcd-ca                 no
apiserver-kubelet-client   Oct 05, 2024 13:02 UTC   364d            ca                      no
controller-manager.conf    Oct 05, 2024 13:02 UTC   364d            ca                      no
etcd-healthcheck-client    Oct 05, 2024 13:02 UTC   364d            etcd-ca                 no
etcd-peer                  Oct 05, 2024 13:02 UTC   364d            etcd-ca                 no
etcd-server                Oct 05, 2024 13:02 UTC   364d            etcd-ca                 no
front-proxy-client         Oct 05, 2024 13:02 UTC   364d            front-proxy-ca          no
scheduler.conf             Oct 05, 2024 13:02 UTC   364d            ca                      no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Oct 03, 2033 13:02 UTC   9y              no
etcd-ca                 Oct 03, 2033 13:02 UTC   9y              no
front-proxy-ca         Oct 03, 2033 13:02 UTC   9y              no

For clusters not created with kubeadm, inspect the API server pod or static pod manifest to find certificate paths. For example, on a control plane node, look at /etc/kubernetes/manifests/kube-apiserver.yaml and note the --tls-cert-file and --tls-private-key-file flags.

Check certificate details with OpenSSL

To see the subject, issuer, and validity of a certificate file, use:

sudo openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -subject -issuer -dates

Example output:

subject=CN=kube-apiserver
issuer=CN=kubernetes
notBefore=Oct  5 13:02:00 2023 GMT
notAfter=Oct  5 13:02:00 2024 GMT

Audit certificate-related Kubernetes Secrets

Kubernetes stores some certificates in Secrets, such as the service account token signing cert. Check for suspicious or unexpected secrets in kube-system:

kubectl get secrets -n kube-system

Look for names like bootstrap-token-, default-token-, or any custom TLS secrets. Inspect a specific secret's metadata (without revealing data) with:

kubectl describe secret <secret-name> -n kube-system

Replace <secret-name> with a real name from the list (e.g., default-token-abcde).

Safe Configuration Path

Now that you have an inventory, you can make safe configuration changes. The safest path is to rely on Kubernetes' built-in certificate management mechanisms and avoid manual file edits unless absolutely necessary.

Use kubeadm to renew certificates (recommended)

If your cluster uses kubeadm, renewing certificates is a controlled operation.

First, back up the existing /etc/kubernetes/pki directory:

sudo tar -czf /root/k8s-pki-backup-$(date +%Y%m%d-%H%M%S).tar.gz /etc/kubernetes/pki

Then renew all certificates with a single command (you can also renew individual ones):

sudo kubeadm certs renew all

You will see output like:

[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'

certificate for serving the Kubernetes API renewed
certificate for the API server to connect to etcd renewed
certificate for the API server to connect to kubelet renewed
certificate for the Kubernetes API to connect to etcd renewed
certificate for the etcd server renewed
certificate for the etcd peer renewed
certificate for etcd healthcheck renewed
certificate for the apiserver to connect to the kubelet renewed
certificate for the controller-manager renewed
certificate for the scheduler renewed
certificate for the front-proxy client renewed

Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.

After renewal, restart the control plane components. On a static pod cluster, moving the manifests temporarily out of the manifests directory will trigger restart:

sudo mv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/
sudo mv /etc/kubernetes/manifests/kube-controller-manager.yaml /tmp/
sudo mv /etc/kubernetes/manifests/kube-scheduler.yaml /tmp/
sudo mv /etc/kubernetes/manifests/etcd.yaml /tmp/
sleep 20
sudo mv /tmp/kube-apiserver.yaml /etc/kubernetes/manifests/
sudo mv /tmp/kube-controller-manager.yaml /etc/kubernetes/manifests/
sudo mv /tmp/kube-scheduler.yaml /etc/kubernetes/manifests/
sudo mv /tmp/etcd.yaml /etc/kubernetes/manifests/

Then verify the API server is responding:

kubectl get nodes

If you see the node list without errors, the control plane is up.

Approve Certificate Signing Requests (CSRs) securely

When a kubelet or other component requests a certificate via CSR, you must approve it carefully.

First, list pending CSRs:

kubectl get csr

Example output:

NAME        AGE     SIGNERNAME                     REQUESTOR          REQUESTEDDURATION   CONDITION
csr-abcde   2m      kubernetes.io/kubelet-serving  system:node:node1  <none>              Pending

Inspect the CSR details before approving:

kubectl describe csr csr-abcde

Check the Requestor, Subject, DNS Names, and IP Addresses to ensure they match the node. Then approve:

kubectl certificate approve csr-abcde

Expected output:

certificatesigningrequest.certificates.k8s.io/csr-abcde approved

To deny if something looks wrong:

kubectl certificate deny csr-abcde

Set restrictive RBAC for CSR approval

Only cluster administrators should be able to approve CSRs. Verify your RBAC policies. For example, to see who can approve CSRs:

kubectl get clusterrolebinding | grep -E 'csr|approve'

If you want to create a dedicated role for CSR approval, use this YAML:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: csr-approver
rules:
- apiGroups: ["certificates.k8s.io"]
  resources: ["certificatesigningrequests/approval"]
  verbs: ["update"]
- apiGroups: ["certificates.k8s.io"]
  resources: ["certificatesigningrequests"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: csr-approver-binding
subjects:
- kind: User
  name: [email protected]
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: csr-approver
  apiGroup: rbac.authorization.k8s.io

Apply it with kubectl apply -f csr-approver.yaml.

Protect private keys

Private keys for CAs and serving certificates should have strict file permissions. For example:

sudo chmod 600 /etc/kubernetes/pki/ca.key
sudo chmod 600 /etc/kubernetes/pki/apiserver.key
sudo chown root:root /etc/kubernetes/pki/ca.key
sudo chown root:root /etc/kubernetes/pki/apiserver.key

Also, ensure that any Kubernetes Secret containing TLS keys is encrypted at rest and access-controlled. Check if encryption at rest is enabled by looking at the API server flags:

ps aux | grep kube-apiserver | grep encryption-provider-config

If no output, encryption at rest is not configured. You can set it up by creating an EncryptionConfiguration and referencing it in the API server manifest.

Verification and Diagnostics

After any change, verify that certificates are valid, rotation is complete, and components trust each other.

Verify certificate expiration after renewal

Run the expiration check again:

sudo kubeadm certs check-expiration

The renewed certificates should show a RESIDUAL TIME close to the original validity period (e.g., 364d).

Test component communication

Check that the API server can reach kubelets with the new certificates. For a specific node, run:

kubectl get --raw="/api/v1/nodes/<node-name>/proxy/healthz"

Replace <node-name> with an actual node name (e.g., node1). Expected output: ok.

For etcd health:

sudo ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key endpoint health

Expected output:

https://127.0.0.1:2379 is healthy: successfully committed proposal: took = 12.345678ms

Diagnose certificate errors in logs

If a component fails, check its logs. For the API server pod:

kubectl logs -n kube-system kube-apiserver-<control-plane-hostname> | grep -i cert

Replace <control-plane-hostname> with your hostname. You might see errors like:

x509: certificate has expired or is not yet valid

or

tls: failed to verify client certificate: x509: certificate signed by unknown authority

These indicate expiry or trust chain issues.

Verify CSR status

For a specific CSR, check if it was issued:

kubectl get csr csr-abcde -o jsonpath='{.status.certificate}' | base64 -d | openssl x509 -noout -subject -issuer -dates

This decodes the issued certificate and shows its details.

Failure Modes and Recovery

Certificates can fail in several ways. Be prepared to diagnose and recover.

Expired certificates

If a certificate expires, affected components will fail to authenticate. The API server may become unavailable, or kubelets may stop reporting.

Diagnosis: Run sudo kubeadm certs check-expiration and look for certificates with RESIDUAL TIME of 0d or negative.

Recovery: Renew the certificates immediately using kubeadm certs renew all (see previous section). If the API server is down and kubeadm cannot connect, you may need to manually renew using the CA key:

sudo kubeadm certs renew apiserver --use-api=false --config=/etc/kubernetes/kubeadm-config.yaml

Then restart the control plane components as described.

Mismatched or untrusted CA

If a component presents a certificate not signed by the cluster CA, you'll see errors like x509: certificate signed by unknown authority.

Diagnosis: Check the certificate issuer with openssl x509 -in <cert> -noout -issuer. Compare with the cluster CA's subject (usually CN=kubernetes).

Recovery: You must issue a new certificate signed by the correct CA. For kubeadm clusters, use kubeadm certs renew. For manual setups, use the CA key to sign a new CSR.

Lost private key

If a private key is lost or compromised, the corresponding certificate is unusable and should be rotated immediately.

Diagnosis: You cannot decrypt traffic or authenticate. Logs may show tls: failed to find any PEM data in certificate input or similar.

Recovery: Generate a new key pair and certificate. For kubeadm, you can force regeneration by deleting the certificate and key files in /etc/kubernetes/pki and running kubeadm init phase certs all (only on a new cluster) or kubeadm certs renew. For a running cluster, you may need to use kubeadm init phase certs with --config carefully, or regenerate manually with openssl and update all references.

Certificate rotation causing service disruption

Improper rotation can leave components with mixed certificates, leading to intermittent failures.

Diagnosis: Check component logs for TLS errors. Use curl -v https://<api-server-ip>:6443/healthz to see the certificate presented.

Recovery: Ensure all control plane components are restarted after renewal. Verify with kubectl get pods -n kube-system that all pods are running and not crash-looping.

Operations Checklist

Use this checklist to maintain certificate hygiene on an ongoing basis.

Replace placeholders like <api-server-ip> with your actual IP (e.g., 192.168.1.10).

Conclusion

Kubernetes certificate security hardening is not a one-time task; it requires regular inspection, controlled rotation, and strict access controls. By following the practices in this guide, you can prevent outages and security breaches caused by mismanaged certificates.

Start with a low-risk verification: run kubeadm certs check-expiration on your cluster today. Record the current state, set up a monthly reminder for renewal, and review who has permission to approve CSRs. These small steps will build a solid foundation for certificate security.

Remember: a reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision.