Kubernetes monitoring and alerting should help you answer three questions fast: is the platform healthy, are workloads healthy, and do users feel it? This practical guide shows you what to measure, how to install a small-but-reliable stack, which alerts to start with, how to verify and troubleshoot, and how to recover when something goes wrong. You will finish with a pilot that is safe to run in a single cluster and easy to extend.

## Version and Environment Inventory

Example environment (adjust to your reality):

- Kubernetes: v1.27 to v1.29

- Prometheus: v2.46+ (single instance for pilot)

- Alertmanager: v0.26+

- kube-state-metrics: v2.10+

- node-exporter: v1.6+

- Grafana: v10+

Prerequisites:

- A cluster-admin or equivalent RBAC to create a monitoring namespace, install Helm charts or manifests, and create ClusterRoles.

- Network access for Prometheus to scrape targets on cluster IPs. If you use NetworkPolicies, allow scraping from the monitoring namespace to node-exporter, kube-state-metrics, and Kubernetes components.

- Storage: start with 10-20 Gi for Prometheus PV with 24-48h retention in a pilot. Increase later.

- Time sync: nodes should have NTP enabled to avoid skewed timestamps.

Safe scoping for a first pilot:

- Single cluster, single namespace: monitoring.

- Core targets only: kube-state-metrics, node-exporter, apiserver, kubelet, and default workloads in 1-2 namespaces.

- One Grafana data source (Prometheus), one folder of dashboards, and a small set of alerts.

## What To Monitor And Why

Start with the smallest set of high-signal metrics that explain most incidents. Expand as you learn.

<div class="my-stack-md overflow-x-auto">
<table class="min-w-[42rem] border-collapse text-left">
<thead><tr><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Layer</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">High-signal metrics or events</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Why it matters</th></tr></thead>
<tbody><tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Control plane</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">apiserver_request_total, apiserver_request_duration_seconds, apiserver_storage_objects, scheduler_pending_pods</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Detect API errors, latency spikes, and scheduling backlogs that block deployments</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Nodes</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">node_cpu_seconds_total, node_memory_MemAvailable_bytes, node_filesystem_avail_bytes, kube_node_status_condition (Ready, DiskPressure)</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Catch node exhaustion and pressure before pods evict or fail</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Workloads</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">container_cpu_usage_seconds_total, container_memory_working_set_bytes, kube_pod_container_status_restarts_total, kube_deployment_status_replicas_unavailable</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Spot runaway CPU/memory and crash loops that impact services</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Networking</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">node_network_receive_errs_total, node_network_transmit_errs_total, pod network RTT if available</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Identify packet errors and degraded east-west traffic</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Storage</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">kubelet_volume_stats_used_bytes, kubelet_volume_stats_capacity_bytes, persistentvolumeclaim status</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Prevent write failures and data loss when PVCs fill</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Scheduling/Autoscaling</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">kube_pod_status_phase, kube_horizontalpodautoscaler_status_desired_replicas</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Explain backlog and scale mismatches</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Events/Logs</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">CrashLoopBackOff, OOMKilled, ImagePullBackOff, back-off restarting</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Fast context when metrics show symptoms</td></tr></tbody>
</table>
</div>
Notes:

- Prefer rates and ratios over absolute counters (e.g., rate(apiserver_request_total[5m]) with error code filters) to avoid false positives.

- Use labels sparingly in queries. Filter by namespace or app where possible.

## Safe Configuration Path

This path stands up a scoped, low-risk stack in a dedicated namespace. It assumes Helm is available. Adjust resource requests to your cluster size.

- Create a monitoring namespace

kubectl create namespace monitoring 

- Add the chart repo and create a values file

 helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

cat > values-monitoring.yaml <<'EOF'
global:
 scrape_interval: 30s

alertmanager:
 config:
 route:
 receiver: 'null'
 group_by: ['alertname', 'namespace']
 group_wait: 30s
 group_interval: 5m
 repeat_interval: 3h
 receivers:
 - name: 'null'
 alertmanagerSpec:
 replicas: 1

prometheus:
 prometheusSpec:
 replicas: 1
 retention: 36h
 resources:
 requests:
 cpu: 200m
 memory: 1Gi
 limits:
 cpu: 1
 memory: 4Gi
 enableAdminAPI: false
 scrapeInterval: 30s
 ruleSelectorNilUsesHelmValues: false
 serviceMonitorSelectorNilUsesHelmValues: false
 podMonitorSelectorNilUsesHelmValues: false
 additionalScrapeConfigs: []
 service:
 type: ClusterIP

kube-state-metrics:
 selfMonitor:
 enabled: false

nodeExporter:
 tolerations:
 - operator: Exists
 resources:
 requests:
 cpu: 50m
 memory: 64Mi

kubelet:
 serviceMonitor:
 cAdvisor: true

grafana:
 enabled: true
 adminUser: admin
 adminPassword: admin
 service:
 type: ClusterIP
 defaultDashboardsEnabled: true
 sidecar:
 dashboards:
 enabled: true
 resources:
 requests:
 cpu: 100m
 memory: 256Mi
EOF 

- Install kube-prometheus-stack

 helm upgrade --install monitoring prometheus-community/kube-prometheus-stack \
 --namespace monitoring \
 --values values-monitoring.yaml 

- Scope scraping to a couple of namespaces (optional but recommended in larger clusters)

 Label monitored namespaces:

kubectl label namespace default monitor=true --overwrite
kubectl label namespace kube-system monitor=true --overwrite 
 Constrain discovery by enabling label selectors on ServiceMonitors/PodMonitors if you introduce custom monitors later. For the core components shipped by the chart, start with defaults and review target counts in Prometheus.

- Add a pilot alert rule file

Create a minimal PrometheusRule with a few high-value alerts (see next section) and apply it to the monitoring namespace.

## Practical Alert Rules

Start with a handful of actionable alerts. Keep severities consistent: critical pages for user impact or data risk; warning for degradation worth investigating during business hours. The expressions below are examples; tune thresholds to your environment.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
 name: pilot-k8s-alerts
 namespace: monitoring
 labels:
 role: alert-rules
spec:
 groups:
 - name: k8s.pilot.rules
 rules:
 - alert: KubeAPIServerHighErrorRate
 expr: |
 sum(rate(apiserver_request_total{code=~"5.."}[5m]))
 /
 sum(rate(apiserver_request_total[5m])) > 0.05
 for: 10m
 labels:
 severity: critical
 annotations:
 summary: High API server 5xx error ratio (>5% for 10m)
 runbook: Check apiserver logs, etcd health, and load spikes.

 - alert: NodeNotReady
 expr: kube_node_status_condition{condition="Ready",status="true"} == 0
 for: 5m
 labels:
 severity: critical
 annotations:
 summary: Node not ready for 5m
 runbook: kubectl describe node; check pressure and kubelet.

 - alert: NodeDiskPressure
 expr: kube_node_status_condition{condition="DiskPressure",status="true"} == 1
 for: 5m
 labels:
 severity: warning
 annotations:
 summary: Node under disk pressure
 runbook: Free space on node; check image GC and logs.

 - alert: PodCrashLooping
 expr: increase(kube_pod_container_status_restarts_total[10m]) > 5
 for: 5m
 labels:
 severity: warning
 annotations:
 summary: Pod restarting frequently (>5 in 10m)
 runbook: kubectl logs; inspect OOMKilled or config errors.

 - alert: PVCFillingUp
 expr: |
 (kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes) > 0.9
 for: 10m
 labels:
 severity: warning
 annotations:
 summary: PVC usage >90%
 runbook: Expand PVC or clean data; verify app retention.

 - alert: HighPodCPU
 expr: |
 sum by (namespace, pod) (rate(container_cpu_usage_seconds_total{image!=""}[5m])) > 2
 for: 10m
 labels:
 severity: warning
 annotations:
 summary: Pod using >2 CPU cores for 10m (example threshold)
 runbook: Check autoscaling or throttle hot paths. 
 A compact mapping of common alerts to first steps:

<div class="my-stack-md overflow-x-auto">
<table class="min-w-[42rem] border-collapse text-left">
<thead><tr><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Alert</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">When it fires (example)</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">First response</th></tr></thead>
<tbody><tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">KubeAPIServerHighErrorRate</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">API 5xx ratio &gt; 5% for 10m</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Check apiserver and etcd logs; reduce load if spiking</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">NodeNotReady</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Node Ready=false 5m</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">kubectl describe node; check pressure and kubelet status</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">NodeDiskPressure</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">DiskPressure=true 5m</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Free disk, prune images/logs; consider tainting the node</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">PodCrashLooping</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">&gt;5 restarts in 10m</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">kubectl logs and events; look for OOMKilled, config errors</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">PVCFillingUp</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">PVC &gt;90% for 10m</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Clean data or expand volume; verify retention policies</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">HighPodCPU</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Pod CPU &gt;2 cores for 10m</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Confirm autoscaling; check hot endpoints and limits</td></tr></tbody>
</table>
</div>

## Dashboards That Matter

Dashboards should answer where the problem is, not just show pretty lines. Recommended starting set:

- Cluster Health Overview: API server latency and error ratio, scheduler pending pods, total pods and nodes Ready, PVC usage percentiles, alert summary panel.

- Node Heatmap: CPU, memory available, disk available, and pressure conditions per node; highlight top offenders.

- Workload SLO: For your top 3 services, chart requests per second, error rate, latency percentiles (from app metrics if available), and pod restarts.

- Deployments Status: Desired vs available replicas, rollout age, and recent failures for targeted namespaces.

- Storage Watch: PVC usage rate, filesystem available, and top 10 fastest-growing volumes.

Tip: Put links to runbooks and kubectl one-liners in panel descriptions for on-call speed.

## Verification and Diagnostics

After installation and rule creation, verify end-to-end.

- Check pods and services

kubectl get pods -n monitoring
kubectl get svc -n monitoring 
 Expect Prometheus, Alertmanager, Grafana, node-exporter DaemonSets, and kube-state-metrics to be Running.

- Verify Prometheus targets and basic query

Port-forward temporarily and open http://localhost:9090.

kubectl -n monitoring port-forward svc/monitoring-kube-prometheus-prometheus 9090:9090 
 Navigate to Status -> Targets; expect targets Up with last scrape recent.

- Verify Grafana

Port-forward and log in (admin/admin from the example values; change in real use):

kubectl -n monitoring port-forward svc/monitoring-grafana 3000:80 
 Add the Prometheus data source if not auto-provisioned (URL http://monitoring-kube-prometheus-prometheus:9090 inside the cluster). Load a cluster overview dashboard and confirm data renders.

- Verify an alert fires

Create a temporary always-on test alert and check Alertmanager UI.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
 name: test-alert
 namespace: monitoring
spec:
 groups:
 - name: test
 rules:
 - alert: AlwaysOnTest
 expr: vector(1)
 for: 0m
 labels:
 severity: warning
 annotations:
 summary: This is a test alert 
 Apply it, wait up to 1 minute, then open Alertmanager via port-forward:

kubectl apply -f test-alert.yaml
kubectl -n monitoring port-forward svc/monitoring-kube-prometheus-alertmanager 9093:9093 
 Open http://localhost:9093 and confirm the alert appears. Remove the test rule when done:

kubectl delete -f test-alert.yaml 

- Diagnose common issues

- If some targets are Down in Prometheus -> Targets: click a target to see the error. Often a ServiceMonitor label mismatch or NetworkPolicy block.

- If no series appear for kube-state-metrics: check that the service is in kube-system or monitoring and that ServiceMonitor selectors match.

- If dashboards load slowly: check Prometheus resource limits; reduce dashboard auto-refresh; simplify queries.

## Incident Response Workflow

 When an alert fires, use a short, repeatable flow to cut time-to-mitigation.

- Classify quickly

- Is this platform (control plane, nodes) or workload (namespaces, deployments)? Check the alert labels and cluster overview dashboard.

- Confirm user impact

- Look for elevated error rate and latency in the workload SLO dashboard. If only infrastructure metrics spike but SLOs hold, downgrade urgency.

- Gather recent changes

- List recent rollouts in the affected namespace:

kubectl get deploy -n <ns>
kubectl rollout history deploy/<name> -n <ns> 

- Check events and logs

- Events:

 kubectl get events -A --sort-by=.lastTimestamp | tail -n 50 

- Pod logs for a crashing container:

 kubectl -n <ns> logs deploy/<name> --tail=200
kubectl -n <ns> describe pod <pod> 
 Look for OOMKilled, CrashLoopBackOff, ImagePullBackOff, back-off restarting.

- Stabilize

- For node pressure: cordon and drain the worst node if you have spare capacity:

kubectl cordon <node>
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data 

- For CrashLoopBackOff due to config: roll back to last working ReplicaSet:

 kubectl -n <ns> rollout undo deploy/<name> 

- Close the loop

- Add a note to the alert or runbook with what worked. Tune thresholds if the alert was noisy or too late.

## Failure Modes and Recovery

 Common pitfalls and how to recover safely.

- Prometheus memory spikes or OOMKills

Symptoms: Prometheus restarts; dashboards time out.

- Mitigation: lower retention (e.g., 24-36h), increase scrape interval (e.g., 30s to 60s), and drop high-cardinality labels via relabelings.

- Recovery: edit values, then upgrade the Helm release. If needed, scale Prometheus down and up to clear overload:

kubectl -n monitoring scale statefulset monitoring-kube-prometheus-prometheus --replicas=0
sleep 10
kubectl -n monitoring scale statefulset monitoring-kube-prometheus-prometheus --replicas=1 

- No data from kube-state-metrics or node-exporter

- Cause: ServiceMonitor label mismatches, NetworkPolicy blocks, or RBAC.

- Fix: check Service/ServiceMonitor labels, allow egress from monitoring namespace, and ensure ClusterRole/Binding exist for the operator.

- Alert noise and fatigue

- Cause: thresholds too low, no grouping, no dead-time between repeats.

- Fix: increase group_wait to 30s, group_interval to 5m; add for: durations; raise thresholds; route warnings to chat, critical to pager.

- Grafana dashboards broken after upgrade

- Fix: re-validate data source; import minimal dashboards first. Keep custom dashboards in source control as JSON.

- Rollback and uninstall

- Silence alerts before major changes:

 # Use Alertmanager UI to create a silence for matcher severity=~".*" for 1h 

- Helm rollback to a known-good revision:

 helm -n monitoring history monitoring
helm -n monitoring rollback monitoring <REVISION> 

- Full uninstall (keeps PVCs by default; delete them only if you are sure):

 helm -n monitoring uninstall monitoring
kubectl -n monitoring get pvc | awk 'NR>1 {print $1}' | xargs -I{} kubectl -n monitoring delete pvc {}
kubectl delete namespace monitoring 

- Clock skew breaks alert timing

- Symptom: scraped samples look stale; alert for: conditions fire unpredictably.

- Fix: ensure NTP on nodes; restart node time services if skewed.

## Operations Checklist

 Daily

- Check Alertmanager for open critical alerts and acknowledge or resolve.

- Scan cluster overview dashboard for API error ratio and node pressure.

- Review top 10 pods by restarts in the last 24h.

Weekly

- Tune any alert that paged more than twice without action.

- Review PVC usage growth; plan expansions before 85%.

- Validate that new namespaces or services are scraped (targets view in Prometheus).

Before releases or cluster upgrades

- Silence non-critical alerts for the change window.

- Confirm backup of Prometheus PV if you store long-lived data elsewhere.

- Validate Grafana dashboards against staging clusters first.

Capacity and hygiene

- Keep Prometheus retention small in a single-instance pilot (24-48h). Use remote write for long-term storage only after the pilot stabilizes.

- Enforce label hygiene in app metrics. Avoid unbounded labels (e.g., user_id) which explode series count.

## Conclusion

You stood up a scoped Kubernetes monitoring and alerting stack, verified data and alert paths, and put initial dashboards and runbooks in place. From here, expand in small, safe steps: add one namespace at a time, one new alert per week, and one dashboard per team. As you grow, consider high availability for Prometheus and Alertmanager, remote storage for longer retention, and app-level SLO dashboards for your top services. The same disciplined approach you used in the pilot will keep monitoring reliable as you scale.