Kubernetes Monitoring and Alerts with Practical Examples: Practical Implementation Guide

Kubernetes monitoring and alerting should help you answer three questions fast: is the platform healthy, are workloads healthy, and do users feel it? This practical guide shows you what to measure, how to install a small-but-reliable stack, which alerts to start with, how to verify and troubleshoot, and how to recover when something goes wrong. You will finish with a pilot that is safe to run in a single cluster and easy to extend.

Version and Environment Inventory

Example environment (adjust to your reality):

• Kubernetes: v1.27 to v1.29
• Prometheus: v2.46+ (single instance for pilot)
• Alertmanager: v0.26+
• kube-state-metrics: v2.10+
• node-exporter: v1.6+
• Grafana: v10+

Prerequisites:

• A cluster-admin or equivalent RBAC to create a monitoring namespace, install Helm charts or manifests, and create ClusterRoles.
• Network access for Prometheus to scrape targets on cluster IPs. If you use NetworkPolicies, allow scraping from the monitoring namespace to node-exporter, kube-state-metrics, and Kubernetes components.
• Storage: start with 10-20 Gi for Prometheus PV with 24-48h retention in a pilot. Increase later.
• Time sync: nodes should have NTP enabled to avoid skewed timestamps.

Safe scoping for a first pilot:

• Single cluster, single namespace: monitoring.
• Core targets only: kube-state-metrics, node-exporter, apiserver, kubelet, and default workloads in 1-2 namespaces.
• One Grafana data source (Prometheus), one folder of dashboards, and a small set of alerts.

What To Monitor And Why

Start with the smallest set of high-signal metrics that explain most incidents. Expand as you learn.

Notes:

• Prefer rates and ratios over absolute counters (e.g., rate(apiserver_request_total[5m]) with error code filters) to avoid false positives.
• Use labels sparingly in queries. Filter by namespace or app where possible.

Safe Configuration Path

This path stands up a scoped, low-risk stack in a dedicated namespace. It assumes Helm is available. Adjust resource requests to your cluster size.

1. Create a monitoring namespace

kubectl create namespace monitoring

1. Add the chart repo and create a values file

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

cat > values-monitoring.yaml <<'EOF'
global:
  scrape_interval: 30s

alertmanager:
  config:
    route:
      receiver: 'null'
      group_by: ['alertname', 'namespace']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 3h
    receivers:
      - name: 'null'
  alertmanagerSpec:
    replicas: 1

prometheus:
  prometheusSpec:
    replicas: 1
    retention: 36h
    resources:
      requests:
        cpu: 200m
        memory: 1Gi
      limits:
        cpu: 1
        memory: 4Gi
    enableAdminAPI: false
    scrapeInterval: 30s
    ruleSelectorNilUsesHelmValues: false
    serviceMonitorSelectorNilUsesHelmValues: false
    podMonitorSelectorNilUsesHelmValues: false
    additionalScrapeConfigs: []
    service:
      type: ClusterIP

kube-state-metrics:
  selfMonitor:
    enabled: false

nodeExporter:
  tolerations:
    - operator: Exists
  resources:
    requests:
      cpu: 50m
      memory: 64Mi

kubelet:
  serviceMonitor:
    cAdvisor: true

grafana:
  enabled: true
  adminUser: admin
  adminPassword: admin
  service:
    type: ClusterIP
  defaultDashboardsEnabled: true
  sidecar:
    dashboards:
      enabled: true
  resources:
    requests:
      cpu: 100m
      memory: 256Mi
EOF

1. Install kube-prometheus-stack

helm upgrade --install monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --values values-monitoring.yaml

1. Scope scraping to a couple of namespaces (optional but recommended in larger clusters)

Label monitored namespaces:

kubectl label namespace default monitor=true --overwrite
kubectl label namespace kube-system monitor=true --overwrite

Constrain discovery by enabling label selectors on ServiceMonitors/PodMonitors if you introduce custom monitors later. For the core components shipped by the chart, start with defaults and review target counts in Prometheus.

1. Add a pilot alert rule file

Create a minimal PrometheusRule with a few high-value alerts (see next section) and apply it to the monitoring namespace.

Practical Alert Rules

Start with a handful of actionable alerts. Keep severities consistent: critical pages for user impact or data risk; warning for degradation worth investigating during business hours. The expressions below are examples; tune thresholds to your environment.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: pilot-k8s-alerts
  namespace: monitoring
  labels:
    role: alert-rules
spec:
  groups:
    - name: k8s.pilot.rules
      rules:
        - alert: KubeAPIServerHighErrorRate
          expr: |
            sum(rate(apiserver_request_total{code=~"5.."}[5m]))
            /
            sum(rate(apiserver_request_total[5m])) > 0.05
          for: 10m
          labels:
            severity: critical
          annotations:
            summary: High API server 5xx error ratio (>5% for 10m)
            runbook: Check apiserver logs, etcd health, and load spikes.

        - alert: NodeNotReady
          expr: kube_node_status_condition{condition="Ready",status="true"} == 0
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: Node not ready for 5m
            runbook: kubectl describe node; check pressure and kubelet.

        - alert: NodeDiskPressure
          expr: kube_node_status_condition{condition="DiskPressure",status="true"} == 1
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: Node under disk pressure
            runbook: Free space on node; check image GC and logs.

        - alert: PodCrashLooping
          expr: increase(kube_pod_container_status_restarts_total[10m]) > 5
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: Pod restarting frequently (>5 in 10m)
            runbook: kubectl logs; inspect OOMKilled or config errors.

        - alert: PVCFillingUp
          expr: |
            (kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes) > 0.9
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: PVC usage >90%
            runbook: Expand PVC or clean data; verify app retention.

        - alert: HighPodCPU
          expr: |
            sum by (namespace, pod) (rate(container_cpu_usage_seconds_total{image!=""}[5m])) > 2
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: Pod using >2 CPU cores for 10m (example threshold)
            runbook: Check autoscaling or throttle hot paths.

A compact mapping of common alerts to first steps:

Dashboards That Matter

Dashboards should answer where the problem is, not just show pretty lines. Recommended starting set:

• Cluster Health Overview: API server latency and error ratio, scheduler pending pods, total pods and nodes Ready, PVC usage percentiles, alert summary panel.
• Node Heatmap: CPU, memory available, disk available, and pressure conditions per node; highlight top offenders.
• Workload SLO: For your top 3 services, chart requests per second, error rate, latency percentiles (from app metrics if available), and pod restarts.
• Deployments Status: Desired vs available replicas, rollout age, and recent failures for targeted namespaces.
• Storage Watch: PVC usage rate, filesystem available, and top 10 fastest-growing volumes.

Tip: Put links to runbooks and kubectl one-liners in panel descriptions for on-call speed.

Verification and Diagnostics

After installation and rule creation, verify end-to-end.

1. Check pods and services

kubectl get pods -n monitoring
kubectl get svc -n monitoring

Expect Prometheus, Alertmanager, Grafana, node-exporter DaemonSets, and kube-state-metrics to be Running.

1. Verify Prometheus targets and basic query

Port-forward temporarily and open http://localhost:9090.

kubectl -n monitoring port-forward svc/monitoring-kube-prometheus-prometheus 9090:9090

Navigate to Status -> Targets; expect targets Up with last scrape recent.

1. Verify Grafana

Port-forward and log in (admin/admin from the example values; change in real use):

kubectl -n monitoring port-forward svc/monitoring-grafana 3000:80

Add the Prometheus data source if not auto-provisioned (URL http://monitoring-kube-prometheus-prometheus:9090 inside the cluster). Load a cluster overview dashboard and confirm data renders.

1. Verify an alert fires

Create a temporary always-on test alert and check Alertmanager UI.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: test-alert
  namespace: monitoring
spec:
  groups:
    - name: test
      rules:
        - alert: AlwaysOnTest
          expr: vector(1)
          for: 0m
          labels:
            severity: warning
          annotations:
            summary: This is a test alert

Apply it, wait up to 1 minute, then open Alertmanager via port-forward:

kubectl apply -f test-alert.yaml
kubectl -n monitoring port-forward svc/monitoring-kube-prometheus-alertmanager 9093:9093

Open http://localhost:9093 and confirm the alert appears. Remove the test rule when done:

kubectl delete -f test-alert.yaml

1. Diagnose common issues

• If some targets are Down in Prometheus -> Targets: click a target to see the error. Often a ServiceMonitor label mismatch or NetworkPolicy block.
• If no series appear for kube-state-metrics: check that the service is in kube-system or monitoring and that ServiceMonitor selectors match.
• If dashboards load slowly: check Prometheus resource limits; reduce dashboard auto-refresh; simplify queries.

Incident Response Workflow

When an alert fires, use a short, repeatable flow to cut time-to-mitigation.

1. Classify quickly

• Is this platform (control plane, nodes) or workload (namespaces, deployments)? Check the alert labels and cluster overview dashboard.

1. Confirm user impact

• Look for elevated error rate and latency in the workload SLO dashboard. If only infrastructure metrics spike but SLOs hold, downgrade urgency.

1. Gather recent changes

• List recent rollouts in the affected namespace:

kubectl get deploy -n <ns>
kubectl rollout history deploy/<name> -n <ns>

• Check events and logs
• Events:

kubectl get events -A --sort-by=.lastTimestamp | tail -n 50

• Pod logs for a crashing container:

kubectl -n <ns> logs deploy/<name> --tail=200
kubectl -n <ns> describe pod <pod>

Look for OOMKilled, CrashLoopBackOff, ImagePullBackOff, back-off restarting.

1. Stabilize

• For node pressure: cordon and drain the worst node if you have spare capacity:

kubectl cordon <node>
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data

• For CrashLoopBackOff due to config: roll back to last working ReplicaSet:

kubectl -n <ns> rollout undo deploy/<name>

1. Close the loop

• Add a note to the alert or runbook with what worked. Tune thresholds if the alert was noisy or too late.

Failure Modes and Recovery

Common pitfalls and how to recover safely.

1. Prometheus memory spikes or OOMKills

Symptoms: Prometheus restarts; dashboards time out.

• Mitigation: lower retention (e.g., 24-36h), increase scrape interval (e.g., 30s to 60s), and drop high-cardinality labels via relabelings.
• Recovery: edit values, then upgrade the Helm release. If needed, scale Prometheus down and up to clear overload:

kubectl -n monitoring scale statefulset monitoring-kube-prometheus-prometheus --replicas=0
sleep 10
kubectl -n monitoring scale statefulset monitoring-kube-prometheus-prometheus --replicas=1

1. No data from kube-state-metrics or node-exporter

• Cause: ServiceMonitor label mismatches, NetworkPolicy blocks, or RBAC.
• Fix: check Service/ServiceMonitor labels, allow egress from monitoring namespace, and ensure ClusterRole/Binding exist for the operator.

1. Alert noise and fatigue

• Cause: thresholds too low, no grouping, no dead-time between repeats.
• Fix: increase group_wait to 30s, group_interval to 5m; add for: durations; raise thresholds; route warnings to chat, critical to pager.

1. Grafana dashboards broken after upgrade

• Fix: re-validate data source; import minimal dashboards first. Keep custom dashboards in source control as JSON.

1. Rollback and uninstall

• Silence alerts before major changes:

# Use Alertmanager UI to create a silence for matcher severity=~".*" for 1h

• Helm rollback to a known-good revision:

helm -n monitoring history monitoring
helm -n monitoring rollback monitoring <REVISION>

• Full uninstall (keeps PVCs by default; delete them only if you are sure):

helm -n monitoring uninstall monitoring
kubectl -n monitoring get pvc | awk 'NR>1 {print $1}' | xargs -I{} kubectl -n monitoring delete pvc {}
kubectl delete namespace monitoring

1. Clock skew breaks alert timing

• Symptom: scraped samples look stale; alert for: conditions fire unpredictably.
• Fix: ensure NTP on nodes; restart node time services if skewed.

Operations Checklist

Daily

• Check Alertmanager for open critical alerts and acknowledge or resolve.
• Scan cluster overview dashboard for API error ratio and node pressure.
• Review top 10 pods by restarts in the last 24h.

Weekly

• Tune any alert that paged more than twice without action.
• Review PVC usage growth; plan expansions before 85%.
• Validate that new namespaces or services are scraped (targets view in Prometheus).

Before releases or cluster upgrades

• Silence non-critical alerts for the change window.
• Confirm backup of Prometheus PV if you store long-lived data elsewhere.
• Validate Grafana dashboards against staging clusters first.

Capacity and hygiene

• Keep Prometheus retention small in a single-instance pilot (24-48h). Use remote write for long-term storage only after the pilot stabilizes.
• Enforce label hygiene in app metrics. Avoid unbounded labels (e.g., user_id) which explode series count.

Conclusion

You stood up a scoped Kubernetes monitoring and alerting stack, verified data and alert paths, and put initial dashboards and runbooks in place. From here, expand in small, safe steps: add one namespace at a time, one new alert per week, and one dashboard per team. As you grow, consider high availability for Prometheus and Alertmanager, remote storage for longer retention, and app-level SLO dashboards for your top services. The same disciplined approach you used in the pilot will keep monitoring reliable as you scale.