## Intro

Certificate failures in Kubernetes are usually silent until a control plane component stops trusting its own TLS identity or a workload rejects an expired serving certificate. This guide gives you a practical, command-first workflow for Kubernetes certificate monitoring and alerting: from discovering what certificates exist in your cluster, to watching their expiry with metrics and dashboards, to responding safely when an alert fires.

You will find concrete commands, expected output, Prometheus queries, alert rules, and recovery steps. The goal is operational safety: observe before you change, limit the blast radius, protect sensitive material, and verify every fix.

This article is written for developers, DevOps consultants, and technical startup teams that operate Kubernetes clusters and need to prevent certificate-related outages.

## Version and Environment Inventory

Before you monitor or change anything, capture the cluster environment. Different Kubernetes distributions and versions manage certificates differently, so your first step is to identify what you are working with.

### Identify the cluster version and distribution

Run the following command to get the server version:

```bash
kubectl version --short
```

Expected output (example for a vanilla Kubernetes cluster):

```
Client Version: v1.27.3
Kustomize Version: v5.0.1
Server Version: v1.27.3
```

If you are using a managed service such as Amazon EKS, Google GKE, or Azure AKS, the server version is also returned, but certificate management differs. For example, EKS automatically rotates the kubelet serving certificates, but the control plane API server certificate is managed by the cloud provider and you cannot inspect it directly. On a kubeadm cluster, the control plane certificates are stored as files in `/etc/kubernetes/pki` and can be checked with `openssl`.

To determine if your cluster was created with kubeadm, look for the kubeadm ConfigMap:

```bash
kubectl get configmap kubeadm-config -n kube-system
```

If this returns a ConfigMap, you are on a kubeadm cluster. If you get `NotFound`, you are likely on a managed or custom distribution.

### List certificate-related resources

Inspect CertificateSigningRequest objects. These are used when certificates are requested via the Kubernetes API.

```bash
kubectl get csr
```

Expected output when there are pending requests:

```
NAME        AGE   SIGNERNAME                     REQUESTOR          REQUESTEDDURATION   CONDITION
my-csr      2m    kubernetes.io/kube-apiserver-client   system:node:worker1   <none>              Pending
```

A `Pending` condition means a cluster administrator must approve the CSR. A `Denied` condition means the request was rejected, and you should investigate why.

If you are using cert-manager, list Certificate resources across all namespaces:

```bash
kubectl get certificates --all-namespaces
```

Possible output:

```
NAMESPACE   NAME          READY   SECRET               AGE
app-team    web-tls       True    web-tls-secret       30d
default     internal-ca   False   internal-ca-secret   10d
```

A `False` readiness means cert-manager has not issued or renewed that certificate. The reason can be found in the Certificate's status conditions or Events.

### Capture current state without changing anything

Before any intervention, record the current state and timestamp. Use a script or manual log:

```bash
kubectl get certificates --all-namespaces -o wide > cert-inventory-$(date +%Y%m%d-%H%M%S).txt
kubectl get csr -o wide > csr-inventory-$(date +%Y%m%d-%H%M%S).txt
```

This gives you a rollback point and an audit trail. Protect files containing private key material: never store them in a shared location without encryption.

### Prerequisites for monitoring

To follow the examples in this article, ensure you have:

- `kubectl` configured with cluster-admin or sufficient RBAC permissions to read secrets and metrics.
- `openssl` installed for direct certificate file inspection.
- `curl` and `jq` for querying metrics endpoints.
- If using Prometheus, access to the Prometheus expression browser or Grafana.

## Safe Configuration Path

Changing certificate configuration can break cluster communication. Follow a safe path: inspect, backup, change one scoped item, verify, and know how to revert.

### Inspect certificate expiry dates

For a kubeadm cluster, the control plane certificates are usually in `/etc/kubernetes/pki`. Check their expiry using a loop on a control plane node.

SSH to the control plane node and run:

```bash
find /etc/kubernetes/pki -type f -name "*.crt" -print0 | while IFS= read -r -d '' cert; do
  echo "$cert"
  openssl x509 -in "$cert" -noout -enddate
 done
```

Example output:

```
/etc/kubernetes/pki/apiserver.crt
notAfter=Oct 12 09:30:00 2024 GMT
/etc/kubernetes/pki/ca.crt
notAfter=Nov 1 10:00:00 2029 GMT
```

The `ca.crt` usually has a long validity, but `apiserver.crt` expires yearly if not renewed.

### Renew a certificate safely with kubeadm

If you need to renew the API server certificate before it expires (or after), use kubeadm's built-in renewal commands. First, check what would be renewed (dry run):

```bash
kubeadm certs renew apiserver --dry-run
```

If the command is supported, it shows what certificates would be renewed without making changes. If `--dry-run` is not available in your kubeadm version, make a backup of `/etc/kubernetes/pki` first:

```bash
cp -a /etc/kubernetes/pki /etc/kubernetes/pki-backup-$(date +%Y%m%d)
```

Then run the renewal:

```bash
kubeadm certs renew apiserver
```

After renewal, restart the kube-apiserver static pod. Usually kubeadm does this automatically, but you can force it:

```bash
crictl ps | grep kube-apiserver
# Note the container ID and kill it; kubelet will restart it.
```

Alternatively, on systems with systemd, restart kubelet:

```bash
systemctl restart kubelet
```

Verify the new certificate is loaded by checking the kube-apiserver logs or querying the API server:

```bash
kubectl get --raw /readyz
```

Expected output: `ok`

### Rotate a serving certificate with cert-manager

If you use cert-manager, do not manually edit a Certificate resource unless you fully understand the implications. To trigger a renewal manually (for testing or emergency), you can delete the Secret associated with the Certificate.

First, back up the Secret:

```bash
kubectl get secret web-tls-secret -n app-team -o yaml > web-tls-secret-backup.yaml
```

Then delete it. cert-manager will issue a new certificate and create a new Secret within minutes, depending on the issuer.

```bash
kubectl delete secret web-tls-secret -n app-team
```

Watch the Certificate status:

```bash
kubectl get certificate web-tls -n app-team -w
```

Wait until `READY` becomes `True`. Then verify the new certificate expiry:

```bash
kubectl get secret web-tls-secret -n app-team -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -enddate
```

Expected output: a new `notAfter` date.

### Protect secrets and private material

Never echo a private key to the terminal in shared sessions. When inspecting a TLS secret, use `jsonpath` to extract only the certificate, not the private key, unless absolutely necessary. If you need the private key, redirect to a file with restrictive permissions:

```bash
umask 077
kubectl get secret web-tls-secret -n app-team -o jsonpath='{.data.tls\.key}' | base64 -d > tls.key
```

Then remove the file after use.

## Verification and Diagnostics

Monitoring is not just about collecting metrics; you must verify that your monitoring stack sees the correct data and that alerts trigger when expected.

### Verify metrics exposure

Kubernetes components expose metrics for their own certificates. For example, the kube-apiserver exposes `apiserver_certificate_expiration_seconds` as a gauge. To check this metric from a control plane node, use `kubectl get --raw`:

```bash
kubectl get --raw /metrics | grep apiserver_certificate_expiration_seconds
```

Example output:

```
# HELP apiserver_certificate_expiration_seconds [ALPHA] Expiration timestamps of API server certificates.
# TYPE apiserver_certificate_expiration_seconds gauge
apiserver_certificate_expiration_seconds{name="apiserver"} 2.592e+06
apiserver_certificate_expiration_seconds{name="apiserver-etcd-client"} 2.592e+06
```

If you do not see this metric, your kube-apiserver version may not expose it (it was introduced in Kubernetes 1.23 with alpha state, moved to beta in 1.26, and is stable in later versions). Enable the `CertificateExpiration` feature gate if needed.

For cert-manager, it exposes metrics at `/metrics` on its controller pod (default port 9402). To check, port-forward to the cert-manager pod:

```bash
kubectl port-forward -n cert-manager deployment/cert-manager 9402:9402
```

Then in another terminal:

```bash
curl -s localhost:9402/metrics | grep certmanager_certificate_expiration_timestamp_seconds
```

Expected output example:

```
certmanager_certificate_expiration_timestamp_seconds{name="web-tls",namespace="app-team"} 1.728e+09
```

The value is a Unix timestamp; you can convert it with `date -d @1728000000`.

### Query Prometheus for certificate expiry

If you have Prometheus scraping these metrics, you can write queries to alert on upcoming expirations. For kube-apiserver certificates, the metric is `apiserver_certificate_expiration_seconds`. To get certificates expiring in less than 30 days, use:

```promql
apiserver_certificate_expiration_seconds - time() < 30 * 24 * 3600
```

This returns a vector of certificates with time until expiry less than 30 days. If the query returns no data, either your scrape is not working or all certificates have more than 30 days left.

For cert-manager, the metric is `certmanager_certificate_expiration_timestamp_seconds`. The query for expiring in less than 30 days is:

```promql
(certmanager_certificate_expiration_timestamp_seconds - time()) < 30 * 24 * 3600
```

Also, cert-manager has a `certmanager_certificate_ready_status` metric that is 1 when ready and 0 when not. Alert on that as well.

### Set up alert rules

Here is an example Prometheus alert rule for cert-manager certificates expiring within 30 days:

```yaml
groups:
- name: kubernetes-certificates
  rules:
  - alert: CertificateExpiringSoon
    expr: (certmanager_certificate_expiration_timestamp_seconds - time()) < 30 * 24 * 3600
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Certificate {{ $labels.name }} in namespace {{ $labels.namespace }} expires in less than 30 days"
      description: "The certificate {{ $labels.name }} in namespace {{ $labels.namespace }} will expire at {{ $value | humanizeTimestamp }}."
  - alert: CertificateNotReady
    expr: certmanager_certificate_ready_status == 0
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "Certificate {{ $labels.name }} in namespace {{ $labels.namespace }} is not ready"
```

For kube-apiserver certificates, a similar rule:

```yaml
  - alert: ApiServerCertificateExpiringSoon
    expr: (apiserver_certificate_expiration_seconds - time()) < 30 * 24 * 3600
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "API server certificate {{ $labels.name }} expires in less than 30 days"
```

### Diagnostic checks when something fails

If an alert fires or you suspect a certificate problem, run these diagnostics:

1. Check the certificate status directly:

```bash
kubectl get certificate <name> -n <namespace> -o yaml
```

Look for `status.conditions` with `type: Ready` and `status: False`. The `message` field often says why it failed.

2. Check events in the namespace:

```bash
kubectl get events -n app-team --sort-by=.lastTimestamp
```

3. For kubelet serving certificate issues on a node, check the kubelet logs:

```bash
journalctl -u kubelet -n 100 --no-pager
```

4. For control plane components, check the static pod logs:

```bash
kubectl logs -n kube-system kube-apiserver-<node-name>
```

Replace `<node-name>` with the actual control plane node name.

## Failure Modes and Recovery

Certificates fail in predictable ways. Here are common failure modes and step-by-step recovery.

### Expired API server certificate

Symptom: `kubectl` cannot connect; the API server returns TLS errors or does not start. In the kube-apiserver logs you may see:

```
TLS handshake error from 10.0.0.5:53210: remote error: tls: bad certificate
```

or

```
Error: x509: certificate has expired or is not yet valid
```

Recovery on a kubeadm cluster:

1. SSH to the control plane node.
2. Backup the PKI directory:

```bash
cp -a /etc/kubernetes/pki /etc/kubernetes/pki-backup-$(date +%Y%m%d-%H%M%S)
```

3. Run `kubeadm certs renew all` to renew all certificates:

```bash
kubeadm certs renew all
```

4. Restart the kube-apiserver and other control plane components:

```bash
crictl ps | grep -E 'kube-apiserver|kube-controller-manager|kube-scheduler' | awk '{print $1}' | xargs -r crictl stop
```

Or if using systemd, restart kubelet:

```bash
systemctl restart kubelet
```

5. Verify by checking the API server availability:

```bash
kubectl get --raw /readyz
```

### Expired workload serving certificate issued by cert-manager

Symptom: Ingress or service returns certificate expiry errors in browser or clients refuse connection. The cert-manager may have failed to renew due to DNS or issuer issues.

Recovery:

1. Check the Certificate resource:

```bash
kubectl describe certificate web-tls -n app-team
```

Look for `Status` events indicating renewal failure, such as `Issuing` or `Failed`.

2. Check cert-manager logs:

```bash
kubectl logs -n cert-manager -l app=cert-manager --tail=100
```

3. Common reasons: ACME DNS01 challenge failed, issuer misconfigured, rate limit exceeded. Fix the underlying issue (e.g., update DNS credentials, increase rate limit).

4. If the certificate is already expired and cert-manager cannot renew automatically, you can force a new issuance by deleting the Secret as described in Safe Configuration Path. But first ensure the issuer is healthy.

### Kubelet client certificate expiration

On nodes, the kubelet uses a client certificate to authenticate to the API server. If it expires, the node becomes `NotReady` and pods are evicted. Check node status:

```bash
kubectl get nodes
```

If a node is `NotReady`, inspect kubelet logs:

```bash
journalctl -u kubelet -n 200 | grep -i certificate
```

On kubeadm clusters, the kubelet client certificate is auto-rotated by default if the `RotateKubeletClientCertificate` feature gate is enabled (it is enabled by default from Kubernetes 1.20). If not, you may need to manually approve CSRs or rotate the kubelet certificate on the node by removing the old certificate files and restarting kubelet.

### Preventing failures: monitoring and alerts

The best recovery is prevention. Ensure your alert rules cover:

- Expiry warning thresholds: 60 days and 30 days.
- Certificates not ready (for cert-manager).
- Failed renewal attempts (e.g., high error rate in cert-manager metrics).
- Certificate signing requests pending for too long.

For pending CSRs, you can set an alert using the Kubernetes API via a custom exporter or using kube-state-metrics if it exposes CSR metrics (not available by default). Instead, you can have a cron job that checks for pending CSRs and sends an alert.

Example cron job command:

```bash
kubectl get csr | grep Pending && echo "Alert: pending CSR" | mail -s "Pending CSR" ops@example.com
```

Better to integrate with your alerting system.

## Operations Checklist

Use this checklist before and during certificate operations. It is designed to minimize risk and ensure verification.

### Before any change

- [ ] Record current cluster version and distribution (`kubectl version --short`).
- [ ] Inventory all Certificate and CSR resources (`kubectl get certificates --all-namespaces`, `kubectl get csr`).
- [ ] Backup relevant Secrets and PKI files (e.g., `cp -a /etc/kubernetes/pki /etc/kubernetes/pki-backup-<date>`).
- [ ] Confirm you have permissions to perform the operation and a rollback plan.
- [ ] Notify stakeholders if the change may cause a brief downtime.

### After any change

- [ ] Verify the resource is in the expected state (e.g., Certificate READY=True).
- [ ] Check the new certificate expiry date with `openssl x509 -noout -enddate`.
- [ ] Test client connections to the service or API server.
- [ ] Check logs for errors in the relevant component.
- [ ] Update documentation and monitoring thresholds if needed.

### Example worked scenario

Suppose the certificate `web-tls` in namespace `app-team` is expiring in 10 days and is managed by cert-manager. You receive an alert from Prometheus.

Follow this sequence:

1. Verify the alert is real:

```bash
kubectl get certificate web-tls -n app-team -o jsonpath='{.status.notAfter}'
```

Expected output: a timestamp within 10 days.

2. Check cert-manager logs for renewal failures:

```bash
kubectl logs -n cert-manager -l app=cert-manager --tail=100 | grep web-tls
```

If you see an error like `failed to perform self check`, it may be a temporary issue.

3. Trigger manual renewal by deleting the Secret (after backup):

```bash
kubectl get secret web-tls-secret -n app-team -o yaml > web-tls-secret-backup.yaml
kubectl delete secret web-tls-secret -n app-team
```

4. Wait for readiness:

```bash
kubectl wait --for=condition=ready certificate/web-tls -n app-team --timeout=300s
```

5. Verify new certificate expiry:

```bash
kubectl get secret web-tls-secret -n app-team -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -enddate
```

6. Confirm HTTP service works (if exposed via Ingress):

```bash
curl -v https://your-app.example.com 2>&1 | grep -E 'expire date|SSL certificate verify ok'
```

This completes the recovery.

## Conclusion

Kubernetes certificate monitoring is not a one-time setup; it requires continuous observation, alerting, and a tested recovery path. By following the version inventory, safe configuration changes, verification techniques, and failure recovery steps in this article, you can reduce the risk of certificate-related outages.

Start with a low-risk verification: choose one certificate, record its current expiry, set up a metric scrape for it, and create a test alert. Then simulate an expiry by adjusting the alert threshold to confirm the notification works. This gives you confidence in your system before a real incident occurs.

Remember the principles: observe before you change, limit the blast radius, protect secrets, verify results, and document recovery. With these practices, your Kubernetes certificates will not be the cause of your next 3 a.m. page.