Kubernetes Certificates Monitoring and Alerts: Practical Examples for Reliable Operations

Intro

Certificate failures in Kubernetes are usually silent until a control plane component stops trusting its own TLS identity or a workload rejects an expired serving certificate. This guide gives you a practical, command-first workflow for Kubernetes certificate monitoring and alerting: from discovering what certificates exist in your cluster, to watching their expiry with metrics and dashboards, to responding safely when an alert fires.

You will find concrete commands, expected output, Prometheus queries, alert rules, and recovery steps. The goal is operational safety: observe before you change, limit the blast radius, protect sensitive material, and verify every fix.

This article is written for developers, DevOps consultants, and technical startup teams that operate Kubernetes clusters and need to prevent certificate-related outages.

Version and Environment Inventory

Before you monitor or change anything, capture the cluster environment. Different Kubernetes distributions and versions manage certificates differently, so your first step is to identify what you are working with.

Identify the cluster version and distribution

Run the following command to get the server version:

kubectl version --short

Expected output (example for a vanilla Kubernetes cluster):

Client Version: v1.27.3
Kustomize Version: v5.0.1
Server Version: v1.27.3

If you are using a managed service such as Amazon EKS, Google GKE, or Azure AKS, the server version is also returned, but certificate management differs. For example, EKS automatically rotates the kubelet serving certificates, but the control plane API server certificate is managed by the cloud provider and you cannot inspect it directly. On a kubeadm cluster, the control plane certificates are stored as files in /etc/kubernetes/pki and can be checked with openssl.

To determine if your cluster was created with kubeadm, look for the kubeadm ConfigMap:

kubectl get configmap kubeadm-config -n kube-system

If this returns a ConfigMap, you are on a kubeadm cluster. If you get NotFound, you are likely on a managed or custom distribution.

List certificate-related resources

Inspect CertificateSigningRequest objects. These are used when certificates are requested via the Kubernetes API.

kubectl get csr

Expected output when there are pending requests:

NAME        AGE   SIGNERNAME                     REQUESTOR          REQUESTEDDURATION   CONDITION
my-csr      2m    kubernetes.io/kube-apiserver-client   system:node:worker1   <none>              Pending

A Pending condition means a cluster administrator must approve the CSR. A Denied condition means the request was rejected, and you should investigate why.

If you are using cert-manager, list Certificate resources across all namespaces:

kubectl get certificates --all-namespaces

Possible output:

NAMESPACE   NAME          READY   SECRET               AGE
app-team    web-tls       True    web-tls-secret       30d
default     internal-ca   False   internal-ca-secret   10d

A False readiness means cert-manager has not issued or renewed that certificate. The reason can be found in the Certificate's status conditions or Events.

Capture current state without changing anything

Before any intervention, record the current state and timestamp. Use a script or manual log:

kubectl get certificates --all-namespaces -o wide > cert-inventory-$(date +%Y%m%d-%H%M%S).txt
kubectl get csr -o wide > csr-inventory-$(date +%Y%m%d-%H%M%S).txt

This gives you a rollback point and an audit trail. Protect files containing private key material: never store them in a shared location without encryption.

Prerequisites for monitoring

To follow the examples in this article, ensure you have:

• kubectl configured with cluster-admin or sufficient RBAC permissions to read secrets and metrics.
• openssl installed for direct certificate file inspection.
• curl and jq for querying metrics endpoints.
• If using Prometheus, access to the Prometheus expression browser or Grafana.

Safe Configuration Path

Changing certificate configuration can break cluster communication. Follow a safe path: inspect, backup, change one scoped item, verify, and know how to revert.

Inspect certificate expiry dates

For a kubeadm cluster, the control plane certificates are usually in /etc/kubernetes/pki. Check their expiry using a loop on a control plane node.

SSH to the control plane node and run:

find /etc/kubernetes/pki -type f -name "*.crt" -print0 | while IFS= read -r -d '' cert; do
  echo "$cert"
  openssl x509 -in "$cert" -noout -enddate
 done

Example output:

/etc/kubernetes/pki/apiserver.crt
notAfter=Oct 12 09:30:00 2024 GMT
/etc/kubernetes/pki/ca.crt
notAfter=Nov 1 10:00:00 2029 GMT

The ca.crt usually has a long validity, but apiserver.crt expires yearly if not renewed.

Renew a certificate safely with kubeadm

If you need to renew the API server certificate before it expires (or after), use kubeadm's built-in renewal commands. First, check what would be renewed (dry run):

kubeadm certs renew apiserver --dry-run

If the command is supported, it shows what certificates would be renewed without making changes. If --dry-run is not available in your kubeadm version, make a backup of /etc/kubernetes/pki first:

cp -a /etc/kubernetes/pki /etc/kubernetes/pki-backup-$(date +%Y%m%d)

Then run the renewal:

kubeadm certs renew apiserver

After renewal, restart the kube-apiserver static pod. Usually kubeadm does this automatically, but you can force it:

crictl ps | grep kube-apiserver
# Note the container ID and kill it; kubelet will restart it.

Alternatively, on systems with systemd, restart kubelet:

systemctl restart kubelet

Verify the new certificate is loaded by checking the kube-apiserver logs or querying the API server:

kubectl get --raw /readyz

Expected output: ok

Rotate a serving certificate with cert-manager

If you use cert-manager, do not manually edit a Certificate resource unless you fully understand the implications. To trigger a renewal manually (for testing or emergency), you can delete the Secret associated with the Certificate.

First, back up the Secret:

kubectl get secret web-tls-secret -n app-team -o yaml > web-tls-secret-backup.yaml

Then delete it. cert-manager will issue a new certificate and create a new Secret within minutes, depending on the issuer.

kubectl delete secret web-tls-secret -n app-team

Watch the Certificate status:

kubectl get certificate web-tls -n app-team -w

Wait until READY becomes True. Then verify the new certificate expiry:

kubectl get secret web-tls-secret -n app-team -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -enddate

Expected output: a new notAfter date.

Protect secrets and private material

Never echo a private key to the terminal in shared sessions. When inspecting a TLS secret, use jsonpath to extract only the certificate, not the private key, unless absolutely necessary. If you need the private key, redirect to a file with restrictive permissions:

umask 077
kubectl get secret web-tls-secret -n app-team -o jsonpath='{.data.tls\.key}' | base64 -d > tls.key

Then remove the file after use.

Verification and Diagnostics

Monitoring is not just about collecting metrics; you must verify that your monitoring stack sees the correct data and that alerts trigger when expected.

Verify metrics exposure

Kubernetes components expose metrics for their own certificates. For example, the kube-apiserver exposes apiserver_certificate_expiration_seconds as a gauge. To check this metric from a control plane node, use kubectl get --raw:

kubectl get --raw /metrics | grep apiserver_certificate_expiration_seconds

Example output:

# HELP apiserver_certificate_expiration_seconds [ALPHA] Expiration timestamps of API server certificates.
# TYPE apiserver_certificate_expiration_seconds gauge
apiserver_certificate_expiration_seconds{name="apiserver"} 2.592e+06
apiserver_certificate_expiration_seconds{name="apiserver-etcd-client"} 2.592e+06

If you do not see this metric, your kube-apiserver version may not expose it (it was introduced in Kubernetes 1.23 with alpha state, moved to beta in 1.26, and is stable in later versions). Enable the CertificateExpiration feature gate if needed.

For cert-manager, it exposes metrics at /metrics on its controller pod (default port 9402). To check, port-forward to the cert-manager pod:

kubectl port-forward -n cert-manager deployment/cert-manager 9402:9402

Then in another terminal:

curl -s localhost:9402/metrics | grep certmanager_certificate_expiration_timestamp_seconds

Expected output example:

certmanager_certificate_expiration_timestamp_seconds{name="web-tls",namespace="app-team"} 1.728e+09

The value is a Unix timestamp; you can convert it with date -d @1728000000.

Query Prometheus for certificate expiry

If you have Prometheus scraping these metrics, you can write queries to alert on upcoming expirations. For kube-apiserver certificates, the metric is apiserver_certificate_expiration_seconds. To get certificates expiring in less than 30 days, use:

apiserver_certificate_expiration_seconds - time() < 30 * 24 * 3600

This returns a vector of certificates with time until expiry less than 30 days. If the query returns no data, either your scrape is not working or all certificates have more than 30 days left.

For cert-manager, the metric is certmanager_certificate_expiration_timestamp_seconds. The query for expiring in less than 30 days is:

(certmanager_certificate_expiration_timestamp_seconds - time()) < 30 * 24 * 3600

Also, cert-manager has a certmanager_certificate_ready_status metric that is 1 when ready and 0 when not. Alert on that as well.

Set up alert rules

Here is an example Prometheus alert rule for cert-manager certificates expiring within 30 days:

groups:
- name: kubernetes-certificates
  rules:
  - alert: CertificateExpiringSoon
    expr: (certmanager_certificate_expiration_timestamp_seconds - time()) < 30 * 24 * 3600
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Certificate {{ $labels.name }} in namespace {{ $labels.namespace }} expires in less than 30 days"
      description: "The certificate {{ $labels.name }} in namespace {{ $labels.namespace }} will expire at {{ $value | humanizeTimestamp }}."
  - alert: CertificateNotReady
    expr: certmanager_certificate_ready_status == 0
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "Certificate {{ $labels.name }} in namespace {{ $labels.namespace }} is not ready"

For kube-apiserver certificates, a similar rule:

  - alert: ApiServerCertificateExpiringSoon
    expr: (apiserver_certificate_expiration_seconds - time()) < 30 * 24 * 3600
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "API server certificate {{ $labels.name }} expires in less than 30 days"

Diagnostic checks when something fails

If an alert fires or you suspect a certificate problem, run these diagnostics:

1. Check the certificate status directly:

kubectl get certificate <name> -n <namespace> -o yaml

Look for status.conditions with type: Ready and status: False. The message field often says why it failed.

1. Check events in the namespace:

kubectl get events -n app-team --sort-by=.lastTimestamp

1. For kubelet serving certificate issues on a node, check the kubelet logs:

journalctl -u kubelet -n 100 --no-pager

1. For control plane components, check the static pod logs:

kubectl logs -n kube-system kube-apiserver-<node-name>

Replace <node-name> with the actual control plane node name.

Failure Modes and Recovery

Certificates fail in predictable ways. Here are common failure modes and step-by-step recovery.

Expired API server certificate

Symptom: kubectl cannot connect; the API server returns TLS errors or does not start. In the kube-apiserver logs you may see:

TLS handshake error from 10.0.0.5:53210: remote error: tls: bad certificate

or

Error: x509: certificate has expired or is not yet valid

Recovery on a kubeadm cluster:

1. SSH to the control plane node.
2. Backup the PKI directory:

cp -a /etc/kubernetes/pki /etc/kubernetes/pki-backup-$(date +%Y%m%d-%H%M%S)

1. Run kubeadm certs renew all to renew all certificates:

kubeadm certs renew all

1. Restart the kube-apiserver and other control plane components:

crictl ps | grep -E 'kube-apiserver|kube-controller-manager|kube-scheduler' | awk '{print $1}' | xargs -r crictl stop

Or if using systemd, restart kubelet:

systemctl restart kubelet

1. Verify by checking the API server availability:

kubectl get --raw /readyz

Expired workload serving certificate issued by cert-manager

Symptom: Ingress or service returns certificate expiry errors in browser or clients refuse connection. The cert-manager may have failed to renew due to DNS or issuer issues.

Recovery:

1. Check the Certificate resource:

kubectl describe certificate web-tls -n app-team

Look for Status events indicating renewal failure, such as Issuing or Failed.

1. Check cert-manager logs:

kubectl logs -n cert-manager -l app=cert-manager --tail=100

1. Common reasons: ACME DNS01 challenge failed, issuer misconfigured, rate limit exceeded. Fix the underlying issue (e.g., update DNS credentials, increase rate limit).

1. If the certificate is already expired and cert-manager cannot renew automatically, you can force a new issuance by deleting the Secret as described in Safe Configuration Path. But first ensure the issuer is healthy.

Kubelet client certificate expiration

On nodes, the kubelet uses a client certificate to authenticate to the API server. If it expires, the node becomes NotReady and pods are evicted. Check node status:

kubectl get nodes

If a node is NotReady, inspect kubelet logs:

journalctl -u kubelet -n 200 | grep -i certificate

On kubeadm clusters, the kubelet client certificate is auto-rotated by default if the RotateKubeletClientCertificate feature gate is enabled (it is enabled by default from Kubernetes 1.20). If not, you may need to manually approve CSRs or rotate the kubelet certificate on the node by removing the old certificate files and restarting kubelet.

Preventing failures: monitoring and alerts

The best recovery is prevention. Ensure your alert rules cover:

• Expiry warning thresholds: 60 days and 30 days.
• Certificates not ready (for cert-manager).
• Failed renewal attempts (e.g., high error rate in cert-manager metrics).
• Certificate signing requests pending for too long.

For pending CSRs, you can set an alert using the Kubernetes API via a custom exporter or using kube-state-metrics if it exposes CSR metrics (not available by default). Instead, you can have a cron job that checks for pending CSRs and sends an alert.

Example cron job command:

kubectl get csr | grep Pending && echo "Alert: pending CSR" | mail -s "Pending CSR" [email protected]

Better to integrate with your alerting system.

Operations Checklist

Use this checklist before and during certificate operations. It is designed to minimize risk and ensure verification.

Before any change

• [ ] Record current cluster version and distribution (kubectl version --short).
• [ ] Inventory all Certificate and CSR resources (kubectl get certificates --all-namespaces, kubectl get csr).
• [ ] Backup relevant Secrets and PKI files (e.g., cp -a /etc/kubernetes/pki /etc/kubernetes/pki-backup-<date>).
• [ ] Confirm you have permissions to perform the operation and a rollback plan.
• [ ] Notify stakeholders if the change may cause a brief downtime.

After any change

• [ ] Verify the resource is in the expected state (e.g., Certificate READY=True).
• [ ] Check the new certificate expiry date with openssl x509 -noout -enddate.
• [ ] Test client connections to the service or API server.
• [ ] Check logs for errors in the relevant component.
• [ ] Update documentation and monitoring thresholds if needed.

Example worked scenario

Suppose the certificate web-tls in namespace app-team is expiring in 10 days and is managed by cert-manager. You receive an alert from Prometheus.

Follow this sequence:

1. Verify the alert is real:

kubectl get certificate web-tls -n app-team -o jsonpath='{.status.notAfter}'

Expected output: a timestamp within 10 days.

1. Check cert-manager logs for renewal failures:

kubectl logs -n cert-manager -l app=cert-manager --tail=100 | grep web-tls

If you see an error like failed to perform self check, it may be a temporary issue.

1. Trigger manual renewal by deleting the Secret (after backup):

kubectl get secret web-tls-secret -n app-team -o yaml > web-tls-secret-backup.yaml
kubectl delete secret web-tls-secret -n app-team

1. Wait for readiness:

kubectl wait --for=condition=ready certificate/web-tls -n app-team --timeout=300s

1. Verify new certificate expiry:

kubectl get secret web-tls-secret -n app-team -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -enddate

1. Confirm HTTP service works (if exposed via Ingress):

curl -v https://your-app.example.com 2>&1 | grep -E 'expire date|SSL certificate verify ok'

This completes the recovery.

Conclusion

Kubernetes certificate monitoring is not a one-time setup; it requires continuous observation, alerting, and a tested recovery path. By following the version inventory, safe configuration changes, verification techniques, and failure recovery steps in this article, you can reduce the risk of certificate-related outages.

Start with a low-risk verification: choose one certificate, record its current expiry, set up a metric scrape for it, and create a test alert. Then simulate an expiry by adjusting the alert threshold to confirm the notification works. This gives you confidence in your system before a real incident occurs.

Remember the principles: observe before you change, limit the blast radius, protect secrets, verify results, and document recovery. With these practices, your Kubernetes certificates will not be the cause of your next 3 a.m. page.