Kubeadm Monitoring and Alerts: A Hands-On Implementation Guide

Introduction

Monitoring a kubeadm cluster is not about collecting every metric possible. It is about answering three questions reliably: what is the current state, what changed, and what should be done when a threshold is crossed. This guide walks through a practical implementation using Prometheus, Alertmanager, and Grafana on a kubeadm-deployed cluster, with concrete commands, expected outputs, and recovery steps.

The target audience is developers, DevOps consultants, and technical startup teams who operate their own Kubernetes clusters. You will learn how to deploy a monitoring stack, define alert rules, verify alerts fire, and respond to common failure modes. Every step follows an operational safety pattern: observe before changing, scope the change, verify the result, and document recovery.

Throughout the article, replace placeholder values (like <your-namespace>, <your-email>) with your own. Commands assume a working kubeadm cluster with kubectl configured and Helm v3 installed. No real credentials or production identifiers are used.

Version and Environment Inventory

Before touching any component, establish a baseline. You need to know the Kubernetes version, the kubeadm version, the node operating system, and the current state of the cluster. Run these read-only commands:

kubectl version --short
kubeadm version -o short
kubectl get nodes -o wide

Expected output for the first command on a typical cluster:

Client Version: v1.27.3
Kustomize Version: v5.0.1
Server Version: v1.27.3

If kubectl get nodes shows any node in NotReady state, resolve that before installing monitoring tools. A healthy node should show Ready and the correct version in the VERSION column.

Prerequisites for the monitoring stack:

• Cluster created with kubeadm (version 1.22 or later recommended for stable metrics APIs).
• kubectl configured to access the cluster with cluster-admin privileges.
• Helm 3 installed locally.
• Internet access from the cluster to pull container images (or a pre-populated registry).
• At least 2 GB of free memory and 2 CPU cores available on worker nodes for the monitoring components.

Record the following in a runbook: cluster name, kubeadm version, Kubernetes version, node list with roles, and the date of the baseline. Use a timestamped command like date -u to record when the baseline was taken. Store this in a secure location, not in a public repository.

Deploying Prometheus with Helm

The standard way to install Prometheus on kubeadm is via the prometheus-community/kube-prometheus-stack Helm chart. It bundles Prometheus, Alertmanager, Grafana, and a set of Kubernetes-specific exporters. First, add the repository:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

Create a namespace dedicated to monitoring:

kubectl create namespace monitoring

Install the stack with sensible defaults, but first review the chart values. Do not install without customizing the retention and storage size, because default settings may not suit your environment. Create a values.yaml file with the following content:

prometheus:
  prometheusSpec:
    retention: 15d
    storageSpec:
      volumeClaimTemplate:
        spec:
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 20Gi

grafana:
  adminPassword: "admin"  # change this in a secrets manager
alertmanager:
  config:
    global:
      resolve_timeout: 5m
    route:
      group_by: ['alertname']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 4h
      receiver: 'default-receiver'
    receivers:
      - name: 'default-receiver'
        email_configs:
          - to: '[email protected]'
            from: '[email protected]'
            smarthost: 'smtp.example.com:587'
            auth_username: '[email protected]'
            auth_password: 'SMTP_PASSWORD'

Replace the email and SMTP settings with your own. Never store plaintext passwords in values files; use Kubernetes secrets or a secret management tool. For this guide, we will temporarily set adminPassword to admin but you should change it immediately after installation.

Install the chart:

helm install monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --values values.yaml

Verify the pods are running:

kubectl get pods -n monitoring

Expected output (names may vary slightly):

NAME                                                     READY   STATUS    RESTARTS   AGE
alertmanager-monitoring-kube-prometheus-alertmanager-0   2/2     Running   0          2m
monitoring-grafana-7c9c8b7d9b-xyz12                      2/2     Running   0          2m
monitoring-kube-prometheus-operator-6f9f8b7c9d-abc34     1/1     Running   0          2m
monitoring-kube-state-metrics-7b8c9d8f6f-def56           1/1     Running   0          2m
monitoring-prometheus-node-exporter-9g8h7f6d5e-ghi78    1/1     Running   0          2m
monitoring-prometheus-monitoring-kube-prometheus-0      2/2     Running   0          2m

If any pod is in CrashLoopBackOff, check logs with kubectl logs -n monitoring <pod-name> and ensure the PVC for Prometheus was created successfully (kubectl get pvc -n monitoring).

Configuring Alert Rules

The kube-prometheus-stack includes default alert rules, but you should add your own for cluster health. Create a custom PrometheusRule. For example, an alert for high node CPU usage:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: custom-node-alerts
  namespace: monitoring
  labels:
    release: monitoring
spec:
  groups:
    - name: node-health
      rules:
        - alert: NodeHighCPUUsage
          expr: 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "High CPU usage on {{ $labels.instance }}"
            description: "CPU usage has been above 80% for 10 minutes (current value: {{ $value }}%)."

Apply this rule:

kubectl apply -f custom-node-alerts.yaml

Verify the rule was loaded by querying the Prometheus API. Port-forward to Prometheus:

kubectl port-forward -n monitoring svc/monitoring-kube-prometheus-prometheus 9090:9090

In another terminal, query the rule:

curl -s http://localhost:9090/api/v1/rules | jq '.data.groups[] | select(.name=="node-health")'

The rule should appear with its name and expression. If not, check the Prometheus logs for config errors.

Testing Alerts and Alertmanager Routing

To verify that alerts actually reach your notification channel, use the Alertmanager UI or the API to send a test alert. First, access Alertmanager:

kubectl port-forward -n monitoring svc/monitoring-kube-prometheus-alertmanager 9093:9093

Open http://localhost:9093 and click "New Silence" or use the API to post a test alert. Alternatively, generate load on a node to trigger the CPU alert. For example, run a stress container on a node (careful: this is a real intervention):

kubectl run stress --image=polinux/stress --restart=Never --command -- stress --cpu 2 --timeout 600s

After a few minutes, the alert should enter Pending and then Firing. Check Prometheus alerts page at http://localhost:9090/alerts or query the API:

curl -s http://localhost:9090/api/v1/alerts | jq '.data.alerts[] | select(.labels.alertname=="NodeHighCPUUsage")'

Expected response includes an alert with "state": "firing". If not, verify that the alert expression returns data using the Prometheus query UI. Once confirmed, delete the stress pod:

kubectl delete pod stress

After the alert resolves, you should receive a resolution notification according to the Alertmanager route (after repeat_interval).

Building Dashboards with Grafana

Grafana is included in the stack and pre-configured with dashboards for Kubernetes. Access Grafana:

kubectl port-forward -n monitoring svc/monitoring-grafana 3000:80

Open http://localhost:3000 and log in with admin and the password from values.yaml. The default dashboards include "Kubernetes / Nodes" and "Kubernetes / Compute Resources / Pod". These give you immediate visibility into node and pod metrics.

To create a custom dashboard for a specific workload, use Prometheus queries. For example, to track request rate for a deployment, add a panel with the query:

sum(rate(container_cpu_usage_seconds_total{namespace="default", pod=~"my-app-.*"}[5m])) by (pod)

Set the panel to refresh every 30 seconds and use a time series graph. This gives you a live view of CPU usage per pod.

Dashboards are also useful for incident response: when you receive an alert, open the relevant dashboard and compare current metrics with the baseline to identify what changed.

Common Pitfalls and How to Avoid Them

1. Ignoring retention and storage settings. Default Prometheus installation may use a small emptyDir volume, meaning data is lost when the pod restarts. This makes historical analysis impossible. Why it happens: Users accept defaults without reviewing. How to avoid: Always set a persistent volume claim (as shown in the values.yaml) and a retention period that fits your operational needs. For most clusters, 15 days is a good starting point. Recovery: If you already installed without persistence, reinstall with correct storage settings after backing up any important configurations.

2. Alert fatigue from too many low-severity alerts. If every small blip triggers a page, the team starts ignoring alerts. This defeats the purpose. Why it happens: Default rules are often not tuned to your environment; thresholds may be too low or for durations too short. How to avoid: Review default rules and disable or adjust those that are noisy. Use severity labels and route only critical alerts to immediate paging. Set repeat_interval appropriately (e.g., 4 hours) to avoid duplicate notifications. Recovery: Tune the alert expressions based on baseline metrics over a week. Raise thresholds or increase the for duration to filter transient spikes.

3. Prometheus and Alertmanager not accessible due to network policies or service misconfigurations. Why it happens: In kubeadm, default network policies are not installed, but if you add them later, they can block traffic to monitoring components. Also, port-forwarding may fail if the service selector is wrong. How to avoid: Test access immediately after installation. Use kubectl describe svc -n monitoring to verify endpoints are populated. Recovery: If endpoints are empty, check the service selector matches the pod labels. For network policies, create policies allowing ingress from the monitoring namespace to the relevant ports.

4. Storing secrets in plaintext in Helm values or config files. Why it happens: Convenience. Developers often put SMTP passwords, Grafana admin passwords, or API tokens directly in files that are committed to version control. How to avoid: Use Kubernetes secrets and reference them in the chart. For example, create a secret for Alertmanager SMTP password and use existingSecret in values. For Grafana, use admin.existingSecret. Recovery: If secrets are leaked, rotate them immediately. Remove from version control, force-push to clean history, and update the secret in the cluster.

5. Not testing recovery procedures. Having an alert is not enough; you must know what to do when it fires. Teams often scramble during an incident because they never practiced. Why it happens: Monitoring is treated as a set-and-forget system. How to avoid: Schedule regular fire drills: pick a low-risk alert, trigger it intentionally (like the stress test above), and walk through the response. Document the steps in a runbook. Recovery: If you find gaps during a drill, update the runbook and assign an owner to fix the issue. The owner should be a single named engineer, e.g., "Priya Shah, DevOps Engineer", and the review should happen quarterly.

Operations Checklist and Ownership

Here is a checklist for setting up and maintaining kubeadm monitoring. Each item has an owner and a review cadence.

For each item, the owner is a single named person, not a team. The owner is responsible for the outcome and for escalating if they cannot complete the task. Revisit the checklist itself every six months to ensure it matches the cluster reality.

Failure Modes and Recovery

Prometheus pod crash-loops due to corrupted storage. Symptom: Prometheus is in CrashLoopBackOff. Logs show errors like opening storage failed or corrupt TSDB. Recovery: First, scale down Prometheus to prevent further writes: kubectl scale statefulset -n monitoring monitoring-kube-prometheus-prometheus --replicas=0. Then, inspect the storage. If using a PVC, you can attach a debug pod to the volume. If data is corrupted and not needed, delete the PVC and recreate. If data is needed, use promtool tsdb to attempt repair. Restore from backup if available. After repair, scale back up and verify using kubectl get pods -n monitoring.

Alertmanager fails to send notifications. Symptom: Alerts are firing but no notifications received. Alertmanager logs show SMTP errors. Recovery: Check the Alertmanager configuration: kubectl get secret -n monitoring alertmanager-monitoring-kube-prometheus-alertmanager -o jsonpath='{.data.alertmanager\.yaml}' | base64 --decode. Validate SMTP settings. Test SMTP connectivity from within the cluster using a debug pod with nc -vz smtp.example.com 587. Fix credentials in the secret and restart Alertmanager.

Grafana dashboard shows no data. Symptom: Panels display "No data" despite metrics existing. Recovery: Check the Grafana datasource configuration. In Grafana, go to Configuration > Data Sources and test the Prometheus connection. If it fails, verify the service monitoring-kube-prometheus-prometheus is reachable from Grafana pod. Check network policies. Also ensure the time range of the dashboard is appropriate.

Alerts not firing due to missing labels. Symptom: You created a PrometheusRule, but it does not appear in Prometheus alerts. Recovery: Verify the rule was picked up: kubectl get prometheusrule -n monitoring. If it exists, check the Prometheus config: kubectl describe prometheus -n monitoring monitoring-kube-prometheus-prometheus. Look for the rule selector. The rule must have labels matching the selector (in our chart, the label release: monitoring is required). If missing, add the label and reapply.

Node exporter not collecting metrics. Symptom: No node-level metrics in Prometheus. Recovery: Check node-exporter daemonset: kubectl get ds -n monitoring monitoring-prometheus-node-exporter. If pods are not running on all nodes, inspect logs. Common issues include hostPath mount problems or resource constraints. Adjust resource requests if needed. Ensure the service monitoring-prometheus-node-exporter is present and Prometheus is scraping it: query Prometheus for up{job="node-exporter"}; if up is 0, check the scrape target.

Conclusion

Effective kubeadm monitoring is a continuous process, not a one-time setup. By following the steps in this guide, you have deployed a robust monitoring stack, configured meaningful alerts, built dashboards, and prepared for common failure modes. The key is to treat monitoring as code: version your configurations, review alert rules regularly, and practice incident response.

Start small: install the stack, verify the default dashboards, and create one custom alert. Then gradually tune thresholds and add more rules. Assign clear ownership for monitoring tasks and review the system monthly. With this foundation, your kubeadm cluster will be more observable, manageable, and resilient.