## Intro Certificates are the backbone of trust in Kubernetes. They authenticate users, service accounts, and control plane components, and they encrypt communication between the API server, kubelet, etcd, and other critical services. A misconfigured or expired certificate can bring your cluster to a halt or, worse, expose it to man-in-the-middle attacks. Yet many operators treat certificates as a set-and-forget configuration. This guide is for developers, DevOps consultants, and technical startup teams who need to harden Kubernetes certificate security with practical, verifiable steps. We will connect Kubernetes certificate hardening, access control, secrets management, and permissions to concrete commands, expected outputs, failure signals, and recovery decisions. The operational goal is safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached. Every step assumes a running Kubernetes cluster (version 1.20+ recommended) with kubectl configured and, for some tasks, kubeadm available. We focus on self-managed clusters, but the principles apply to managed offerings as well. ## Version and Environment Inventory Before touching any certificate, you must know your environment. Inventory the Kubernetes version, the certificate authorities (CAs) in use, and where certificates are stored. Identify the cluster version and certificate locations Run: kubectl version --short Expected output (example for v1.27): Client Version: v1.27.3 Server Version: v1.27.3 If your cluster was set up with kubeadm, certificates are stored in /etc/kubernetes/pki on control plane nodes. List them: sudo ls -la /etc/kubernetes/pki You should see files like ca.crt , ca.key , apiserver.crt , apiserver.key , front-proxy-ca.crt , etcd/ca.crt , etc. Check certificate expiry with kubeadm: sudo kubeadm certs check-expiration Sample output: [check-expiration] Reading configuration from the cluster... [check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED admin.conf Oct 05, 2024 13:02 UTC 364d ca no apiserver Oct 05, 2024 13:02 UTC 364d ca no apiserver-etcd-client Oct 05, 2024 13:02 UTC 364d etcd-ca no apiserver-kubelet-client Oct 05, 2024 13:02 UTC 364d ca no controller-manager.conf Oct 05, 2024 13:02 UTC 364d ca no etcd-healthcheck-client Oct 05, 2024 13:02 UTC 364d etcd-ca no etcd-peer Oct 05, 2024 13:02 UTC 364d etcd-ca no etcd-server Oct 05, 2024 13:02 UTC 364d etcd-ca no front-proxy-client Oct 05, 2024 13:02 UTC 364d front-proxy-ca no scheduler.conf Oct 05, 2024 13:02 UTC 364d ca no CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED ca Oct 03, 2033 13:02 UTC 9y no etcd-ca Oct 03, 2033 13:02 UTC 9y no front-proxy-ca Oct 03, 2033 13:02 UTC 9y no For clusters not created with kubeadm, inspect the API server pod or static pod manifest to find certificate paths. For example, on a control plane node, look at /etc/kubernetes/manifests/kube-apiserver.yaml and note the --tls-cert-file and --tls-private-key-file flags. Check certificate details with OpenSSL To see the subject, issuer, and validity of a certificate file, use: sudo openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -subject -issuer -dates Example output: subject=CN=kube-apiserver issuer=CN=kubernetes notBefore=Oct 5 13:02:00 2023 GMT notAfter=Oct 5 13:02:00 2024 GMT Audit certificate-related Kubernetes Secrets Kubernetes stores some certificates in Secrets, such as the service account token signing cert. Check for suspicious or unexpected secrets in kube-system : kubectl get secrets -n kube-system Look for names like bootstrap-token- , default-token- , or any custom TLS secrets. Inspect a specific secret's metadata (without revealing data) with: kubectl describe secret -n kube-system Replace with a real name from the list (e.g., default-token-abcde ). ## Safe Configuration Path Now that you have an inventory, you can make safe configuration changes. The safest path is to rely on Kubernetes' built-in certificate management mechanisms and avoid manual file edits unless absolutely necessary. Use kubeadm to renew certificates (recommended) If your cluster uses kubeadm, renewing certificates is a controlled operation. First, back up the existing /etc/kubernetes/pki directory: sudo tar -czf /root/k8s-pki-backup-$(date +%Y%m%d-%H%M%S).tar.gz /etc/kubernetes/pki Then renew all certificates with a single command (you can also renew individual ones): sudo kubeadm certs renew all You will see output like: [renew] Reading configuration from the cluster... [renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' certificate for serving the Kubernetes API renewed certificate for the API server to connect to etcd renewed certificate for the API server to connect to kubelet renewed certificate for the Kubernetes API to connect to etcd renewed certificate for the etcd server renewed certificate for the etcd peer renewed certificate for etcd healthcheck renewed certificate for the apiserver to connect to the kubelet renewed certificate for the controller-manager renewed certificate for the scheduler renewed certificate for the front-proxy client renewed Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates. After renewal, restart the control plane components. On a static pod cluster, moving the manifests temporarily out of the manifests directory will trigger restart: sudo mv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/ sudo mv /etc/kubernetes/manifests/kube-controller-manager.yaml /tmp/ sudo mv /etc/kubernetes/manifests/kube-scheduler.yaml /tmp/ sudo mv /etc/kubernetes/manifests/etcd.yaml /tmp/ sleep 20 sudo mv /tmp/kube-apiserver.yaml /etc/kubernetes/manifests/ sudo mv /tmp/kube-controller-manager.yaml /etc/kubernetes/manifests/ sudo mv /tmp/kube-scheduler.yaml /etc/kubernetes/manifests/ sudo mv /tmp/etcd.yaml /etc/kubernetes/manifests/ Then verify the API server is responding: kubectl get nodes If you see the node list without errors, the control plane is up. Approve Certificate Signing Requests (CSRs) securely When a kubelet or other component requests a certificate via CSR, you must approve it carefully. First, list pending CSRs: kubectl get csr Example output: NAME AGE SIGNERNAME REQUESTOR REQUESTEDDURATION CONDITION csr-abcde 2m kubernetes.io/kubelet-serving system:node:node1 Pending Inspect the CSR details before approving: kubectl describe csr csr-abcde Check the Requestor , Subject , DNS Names , and IP Addresses to ensure they match the node. Then approve: kubectl certificate approve csr-abcde Expected output: certificatesigningrequest.certificates.k8s.io/csr-abcde approved To deny if something looks wrong: kubectl certificate deny csr-abcde Set restrictive RBAC for CSR approval Only cluster administrators should be able to approve CSRs. Verify your RBAC policies. For example, to see who can approve CSRs: kubectl get clusterrolebinding | grep -E 'csr|approve' If you want to create a dedicated role for CSR approval, use this YAML: apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: csr-approver rules: - apiGroups: ["certificates.k8s.io"] resources: ["certificatesigningrequests/approval"] verbs: ["update"] - apiGroups: ["certificates.k8s.io"] resources: ["certificatesigningrequests"] verbs: ["get", "list", "watch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: csr-approver-binding subjects: - kind: User name: alice@example.com apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: csr-approver apiGroup: rbac.authorization.k8s.io Apply it with kubectl apply -f csr-approver.yaml . Protect private keys Private keys for CAs and serving certificates should have strict file permissions. For example: sudo chmod 600 /etc/kubernetes/pki/ca.key sudo chmod 600 /etc/kubernetes/pki/apiserver.key sudo chown root:root /etc/kubernetes/pki/ca.key sudo chown root:root /etc/kubernetes/pki/apiserver.key Also, ensure that any Kubernetes Secret containing TLS keys is encrypted at rest and access-controlled. Check if encryption at rest is enabled by looking at the API server flags: ps aux | grep kube-apiserver | grep encryption-provider-config If no output, encryption at rest is not configured. You can set it up by creating an EncryptionConfiguration and referencing it in the API server manifest. ## Verification and Diagnostics After any change, verify that certificates are valid, rotation is complete, and components trust each other. Verify certificate expiration after renewal Run the expiration check again: sudo kubeadm certs check-expiration The renewed certificates should show a RESIDUAL TIME close to the original validity period (e.g., 364d). Test component communication Check that the API server can reach kubelets with the new certificates. For a specific node, run: kubectl get --raw="/api/v1/nodes//proxy/healthz" Replace with an actual node name (e.g., node1 ). Expected output: ok . For etcd health: sudo ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key endpoint health Expected output: https://127.0.0.1:2379 is healthy: successfully committed proposal: took = 12.345678ms Diagnose certificate errors in logs If a component fails, check its logs. For the API server pod: kubectl logs -n kube-system kube-apiserver- | grep -i cert Replace with your hostname. You might see errors like: x509: certificate has expired or is not yet valid or tls: failed to verify client certificate: x509: certificate signed by unknown authority These indicate expiry or trust chain issues. Verify CSR status For a specific CSR, check if it was issued: kubectl get csr csr-abcde -o jsonpath='{.status.certificate}' | base64 -d | openssl x509 -noout -subject -issuer -dates This decodes the issued certificate and shows its details. ## Failure Modes and Recovery Certificates can fail in several ways. Be prepared to diagnose and recover. Expired certificates If a certificate expires, affected components will fail to authenticate. The API server may become unavailable, or kubelets may stop reporting. Diagnosis: Run sudo kubeadm certs check-expiration and look for certificates with RESIDUAL TIME of 0d or negative. Recovery: Renew the certificates immediately using kubeadm certs renew all (see previous section). If the API server is down and kubeadm cannot connect, you may need to manually renew using the CA key: sudo kubeadm certs renew apiserver --use-api=false --config=/etc/kubernetes/kubeadm-config.yaml Then restart the control plane components as described. Mismatched or untrusted CA If a component presents a certificate not signed by the cluster CA, you'll see errors like x509: certificate signed by unknown authority . Diagnosis: Check the certificate issuer with openssl x509 -in -noout -issuer . Compare with the cluster CA's subject (usually CN=kubernetes ). Recovery: You must issue a new certificate signed by the correct CA. For kubeadm clusters, use kubeadm certs renew . For manual setups, use the CA key to sign a new CSR. Lost private key If a private key is lost or compromised, the corresponding certificate is unusable and should be rotated immediately. Diagnosis: You cannot decrypt traffic or authenticate. Logs may show tls: failed to find any PEM data in certificate input or similar. Recovery: Generate a new key pair and certificate. For kubeadm, you can force regeneration by deleting the certificate and key files in /etc/kubernetes/pki and running kubeadm init phase certs all (only on a new cluster) or kubeadm certs renew . For a running cluster, you may need to use kubeadm init phase certs with --config carefully, or regenerate manually with openssl and update all references. Certificate rotation causing service disruption Improper rotation can leave components with mixed certificates, leading to intermittent failures. Diagnosis: Check component logs for TLS errors. Use curl -v https://:6443/healthz to see the certificate presented. Recovery: Ensure all control plane components are restarted after renewal. Verify with kubectl get pods -n kube-system that all pods are running and not crash-looping. ## Operations Checklist Use this checklist to maintain certificate hygiene on an ongoing basis.

Task	Command / Action	Frequency	Expected Result
Check certificate expiration	`sudo kubeadm certs check-expiration`	Monthly	All certificates have at least 30 days residual time
Review CSR approvals	`kubectl get csr --field-selector=spec.signerName=kubernetes.io/kubelet-serving`	Weekly	No unexpected pending CSRs
Audit RBAC for certificate operations	`kubectl get clusterrolebindings -o yaml \| grep -B5 -A5 certificates.k8s.io`	Quarterly	Only authorized users have approve/update permissions
Verify file permissions on PKI	`sudo stat -c '%a %n' /etc/kubernetes/pki/*.key`	Monthly	Private keys are 600 and owned by root
Backup PKI directory	`sudo tar -czf /backup/pki-$(date +%F).tar.gz /etc/kubernetes/pki`	Before any change	Backup completes without errors
Test API server certificate	`echo \| openssl s_client -connect <api-server-ip>:6443 2>/dev/null \| openssl x509 -noout -dates`	Monthly	Certificate not expired
Check for encryption at rest	`ps aux \| grep kube-apiserver \| grep encryption-provider-config`	Quarterly	Flag present if secrets should be encrypted
Monitor for failed TLS handshakes	Search logs for `x509` errors	Weekly	No recent errors in control plane logs

Replace placeholders like with your actual IP (e.g., 192.168.1.10 ). ## Conclusion Kubernetes certificate security hardening is not a one-time task; it requires regular inspection, controlled rotation, and strict access controls. By following the practices in this guide, you can prevent outages and security breaches caused by mismanaged certificates. Start with a low-risk verification: run kubeadm certs check-expiration on your cluster today. Record the current state, set up a monthly reminder for renewal, and review who has permission to approve CSRs. These small steps will build a solid foundation for certificate security. Remember: a reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision.