Kubectl Monitoring and Alerts: A Practical Guide with Examples

Intro

Kubectl monitoring and alerts with practical examples should help operators move from an observed problem to a verified result. This guide focuses on the day-to-day reality of running Kubernetes: checking cluster health, diagnosing failing workloads, setting up lightweight alerts, and recovering safely when something breaks.

You will learn a repeatable workflow: start by identifying the installed version, deployment topology, prerequisites, and the exact component being inspected. Then use read-only commands to observe, make the smallest justified change, verify the result, and document how to recover if the expected state is not reached.

The article targets developers, DevOps consultants, and technical startup teams who use kubectl as their primary Kubernetes interface. It connects kubectl alerts, metrics, dashboards, and incident response to concrete commands, expected output, failure signals, and recovery decisions.

The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached.

Version and Environment Inventory

Before monitoring anything, know what you are dealing with. Version and Environment Inventory should name the relevant component, the supported version range, prerequisites, a read-only observation, the smallest justified change, and the command or signal that verifies the outcome.

Within Version and Environment Inventory, separate observation from intervention. Capture current state and timestamps first, protect credentials and private material, then change one scoped item only when its blast radius and recovery path are understood.

The important concepts for Version and Environment Inventory are kubectl monitoring, alerts, metrics, dashboards, and incident response. Related areas such as namespace, pod, and deployment should be included only when they affect prerequisites, compatibility, security, observability, or recovery for this topic.

Practical Kubernetes check for Version and Environment Inventory: start with kubectl get pods -o wide, then use kubectl describe pod <name> for scheduling and event details, kubectl logs <name> --previous for crash loops, and kubectl rollout status deployment/<name> before assuming a release succeeded.

For Version and Environment Inventory, keep the local test small. Apply one manifest, inspect the generated resources, and verify traffic with kubectl port-forward or a local service type before moving to a cloud load balancer or ingress controller.

Step-by-step version inventory

1. Check client and server versions:

    kubectl version --short

Expected output example:

    Client Version: v1.28.2
    Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
    Server Version: v1.27.4

If the server version is not shown, your kubeconfig may not have access to the cluster. Check kubectl config current-context to confirm which cluster you are talking to.

1. Verify cluster access and API resources:

    kubectl auth can-i list pods --all-namespaces

Expected: yes or no. If no, you know immediately that some monitoring commands will fail due to RBAC.

1. List namespaces to understand the environment topology:

    kubectl get namespaces

Identify whether you are in a development, staging, or production namespace. For example, a production namespace might be named prod-eu-west, a staging namespace staging, and development dev-team1.

1. Check if any prerequisites are missing, such as metrics-server for kubectl top:

    kubectl get deployment metrics-server -n kube-system

If you see NAME READY UP-TO-DATE AVAILABLE AGE then metrics-server is present. If you see Error from server (NotFound): deployments.apps "metrics-server" not found, you need to install it before using kubectl top.

1. Document the supported version range for your cluster. For example, you might support Kubernetes 1.24 to 1.28. Always check the official Kubernetes version skew policy: kubectl supports one minor version newer or older than the API server. Example: if your cluster runs 1.27, use kubectl 1.26, 1.27, or 1.28.

Safe Configuration Path

Monitoring often requires configuration changes: setting up alerts, dashboards, or logging. The Safe Configuration Path ensures you do not break a running system while changing it. It should name the relevant component, the supported version range, prerequisites, a read-only observation, the smallest justified change, and the command or signal that verifies the outcome.

Separate observation from intervention. Capture current state and timestamps first, protect credentials and private material, then change one scoped item only when its blast radius and recovery path are understood.

Important concepts include kubectl monitoring, alerts, metrics, dashboards, and incident response. Namespace, pod, and deployment appear when they affect prerequisites, compatibility, security, observability, or recovery.

Practical Kubernetes check: start with kubectl get pods -o wide, then kubectl describe pod <name>, kubectl logs <name> --previous, and kubectl rollout status deployment/<name>.

Keep the local test small. Apply one manifest, inspect the generated resources, and verify traffic with kubectl port-forward or a local service type before moving to a cloud load balancer or ingress controller.

Configuring a Prometheus alert rule safely

Assume you want to add a Prometheus alert for pods stuck in CrashLoopBackOff. The safe path is:

1. Observe current state: Before changing anything, see if any pod is already crashing:

    kubectl get pods --all-namespaces --field-selector=status.phase=Running

Filter for restarts:

    kubectl get pods --all-namespaces | awk '$4 > 0 {print $1, $2, $4}'

This command lists pods with restart count greater than zero. Note the output:

    kube-system   coredns-64897985d-4x2g2   1
    default       nginx-deployment-7c79c4bd97-9f8pk   3

This tells you which pods are already in a restart loop.

1. Identify the alert rule file location. Suppose Prometheus is deployed via the kube-prometheus-stack Helm chart. The rules are in a ConfigMap in the monitoring namespace. List ConfigMaps:

    kubectl get configmap -n monitoring | grep rules

Output example:

    prometheus-kube-prometheus-alertmanager-rules   1      28d
    prometheus-kube-prometheus-prometheus-rulefiles-0   1      28d

1. Make a minimal change. Instead of editing the live ConfigMap, create a new rule file or edit a test copy. For example, create a local file crashloop-alert.yaml:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: custom-alert-rules
      namespace: monitoring
    data:
      crashloop.rules: |
        groups:
        - name: pod-crashloop
          rules:
          - alert: PodCrashLooping
            expr: increase(kube_pod_container_status_restarts_total[10m]) > 0
            for: 5m
            labels:
              severity: warning
            annotations:
              summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} is crash looping"
              description: "Container {{ $labels.container }} has restarted {{ $value }} times in the last 10 minutes."

1. Apply and verify the ConfigMap without disrupting existing rules:

    kubectl apply -f crashloop-alert.yaml

Verify:

    kubectl get configmap custom-alert-rules -n monitoring -o yaml

Confirm that the rule file appears under data.

1. Check Prometheus picks up the rule. If using the Prometheus Operator, it automatically reloads rules from ConfigMaps labeled appropriately. You can check the Prometheus UI or API:

    kubectl port-forward -n monitoring svc/prometheus-kube-prometheus-prometheus 9090:9090

Then open http://localhost:9090/alerts in a browser. Your new alert should appear in the list, initially inactive.

1. Trigger a test alert. To verify the alert works, deploy a crashing pod in a test namespace:

    kubectl create namespace test-alert
    kubectl run crashpod --image=busybox --restart=Never --command -- sh -c 'exit 1'

Wait a few minutes and check the Prometheus alerts page or query the alert state via the API:

    curl -s http://localhost:9090/api/v1/alerts | jq '.data.alerts[] | select(.labels.alertname=="PodCrashLooping")'

You should see the alert move to pending and then firing.

1. Clean up: Delete the test pod and namespace after verification.

    kubectl delete namespace test-alert

Verification and Diagnostics

Verification and diagnostics are the heart of kubectl monitoring. You need to know whether an observed symptom is real, what caused it, and whether a fix worked.

For Kubectl monitoring, Verification and Diagnostics should name the relevant component, the supported version range, prerequisites, a read-only observation, the smallest justified change, and the command or signal that verifies the outcome.

Separate observation from intervention. Capture current state and timestamps first, protect credentials and private material, then change one scoped item only when its blast radius and recovery path are understood.

Important concepts include kubectl monitoring, alerts, metrics, dashboards, and incident response. Related areas such as namespace, pod, and deployment are included when they affect prerequisites, compatibility, security, observability, or recovery.

Practical check: start with kubectl get pods -o wide, then kubectl describe pod <name>, kubectl logs <name> --previous, and kubectl rollout status deployment/<name>.

Keep local tests small. Apply one manifest, inspect resources, verify traffic with kubectl port-forward or a local service before moving to cloud load balancers or ingress controllers.

Diagnosing a failed deployment

Let us walk through a realistic scenario: your deployment web-app in namespace production is not becoming ready.

1. Get a quick overview:

    kubectl get pods -n production -l app=web-app

Output shows a pod in CrashLoopBackOff:

    NAME                        READY   STATUS             RESTARTS   AGE
    web-app-7c9f8b6d4-abcde    0/1     CrashLoopBackOff   6          5m
    web-app-7c9f8b6d4-fghij    1/1     Running            0          5m

1. Describe the crashing pod to see events and container state:

    kubectl describe pod web-app-7c9f8b6d4-abcde -n production

Look for the Events section at the bottom. Example event that indicates an image pull problem:

    Warning  Failed     5m    kubelet            Failed to pull image "myrepo/web-app:latest": rpc error: code = NotFound desc = failed to pull and unpack image

Or an OOM kill:

    Warning  BackOff    4m    kubelet            Back-off restarting failed container
    Normal   Killing    4m    kubelet            Container web-app failed liveness probe, will be restarted

1. Check previous logs if the container restarted before you could capture its output:

    kubectl logs web-app-7c9f8b6d4-abcde -n production --previous

Example log line indicating a missing environment variable:

    ERROR: DATABASE_URL environment variable is not set

1. Check rollout status:

    kubectl rollout status deployment/web-app -n production

If the rollout is stuck, you may see:

    Waiting for deployment "web-app" rollout to finish: 1 out of 2 new replicas have been updated...

Use kubectl rollout history deployment/web-app -n production to see previous revisions.

1. Check resource usage if metrics-server is available:

    kubectl top pods -n production -l app=web-app

Output might show high memory usage:

    NAME                        CPU(cores)   MEMORY(bytes)
    web-app-7c9f8b6d4-abcde    10m          350Mi
    web-app-7c9f8b6d4-fghij    5m           120Mi

If the pod memory limit is 300Mi, the first pod is near its limit. Check limits:

    kubectl get pod web-app-7c9f8b6d4-abcde -n production -o jsonpath='{.spec.containers[*].resources}'

Example output:

    {"limits":{"cpu":"500m","memory":"300Mi"},"requests":{"cpu":"100m","memory":"150Mi"}}

1. Confirm the diagnosis: The pod is crashing because the database URL environment variable is missing (from logs) and/or memory limit is too low (from top). Verify by checking the deployment environment:

    kubectl get deployment web-app -n production -o yaml | grep -A5 env:

You may find no DATABASE_URL entry. Or check the ConfigMap/Secret referenced. This is a read-only verification that isolates the root cause before making changes.

Failure Modes and Recovery

Even with careful monitoring, failures happen. Failure Modes and Recovery should name the relevant component, supported version range, prerequisites, a read-only observation, the smallest justified change, and the command or signal that verifies the outcome.

Separate observation from intervention. Capture current state and timestamps first, protect credentials and private material, then change one scoped item only when its blast radius and recovery path are understood.

Important concepts include kubectl monitoring, alerts, metrics, dashboards, and incident response. Namespace, pod, and deployment are included when they affect prerequisites, compatibility, security, observability, or recovery.

Practical check: start with kubectl get pods -o wide, then kubectl describe pod <name>, kubectl logs <name> --previous, and kubectl rollout status deployment/<name>.

Keep local tests small. Apply one manifest, inspect resources, verify traffic with kubectl port-forward or a local service before moving to cloud load balancers or ingress controllers.

Common failure modes and recovery steps

Recovery example: fixing a CrashLoopBackOff

Continuing from the previous diagnosis, suppose the pod web-app-7c9f8b6d4-abcde is crash looping because DATABASE_URL is missing. Recovery steps:

1. Add the environment variable to the deployment. Use kubectl edit deployment web-app -n production or apply a patch:

    kubectl set env deployment/web-app -n production DATABASE_URL=postgres://user:[email protected]:5432/webdb

Or if the value is in a secret:

    kubectl set env deployment/web-app -n production --from=secret/db-secret --keys=DATABASE_URL

Note: Never put plain text secrets in command line if avoidable. Use a secret reference.

1. Verify the rollout triggers new pods:

    kubectl rollout status deployment/web-app -n production

Wait for success message:

    deployment "web-app" successfully rolled out

1. Check the new pod is running and not restarting:

    kubectl get pods -n production -l app=web-app

Output should show all pods with RESTARTS staying at 0 or not increasing:

    NAME                        READY   STATUS    RESTARTS   AGE
    web-app-7c9f8b6d4-abcde    1/1     Running   0          2m
    web-app-7c9f8b6d4-fghij    1/1     Running   0          26m

1. Check logs to confirm the app started correctly:

    kubectl logs web-app-7c9f8b6d4-abcde -n production | tail -20

Look for successful startup messages, e.g., Connected to database, Server started on port 8080.

1. Check the service endpoints if applicable:

    kubectl get endpoints web-app-service -n production

Should list pod IPs and port.

1. Document the recovery in your incident log: what was observed, what changed, how verified, and any follow-up (e.g., add alert for crash loops).

Operations Checklist

A checklist ensures consistent monitoring and recovery without relying on memory. For operations, you should name the relevant component, supported version range, prerequisites, a read-only observation, the smallest justified change, and the command or signal that verifies the outcome.

Separate observation from intervention. Capture current state and timestamps first, protect credentials and private material, then change one scoped item only when its blast radius and recovery path are understood.

Important concepts include kubectl monitoring, alerts, metrics, dashboards, and incident response. Namespace, pod, and deployment are included when they affect prerequisites, compatibility, security, observability, or recovery.

Practical check: start with kubectl get pods -o wide, then kubectl describe pod <name>, kubectl logs <name> --previous, and kubectl rollout status deployment/<name>.

Keep local tests small. Apply one manifest, inspect resources, verify traffic with kubectl port-forward or a local service before moving to cloud load balancers or ingress controllers.

Daily monitoring checklist

Run these commands at the start of a monitoring shift or after any cluster change. Replace placeholders with your actual namespace and deployment names.

1. Cluster health overview:

    kubectl get nodes

Verify all nodes are Ready. If a node is NotReady, run:

    kubectl describe node <node-name>

Look for conditions and events, e.g., Kubelet stopped posting node status.

1. Pod status across namespaces:

    kubectl get pods --all-namespaces --field-selector=status.phase!=Running,status.phase!=Succeeded

This shows pods that are not in a healthy state (Pending, Failed, Unknown). Investigate any output.

1. Deployment rollout status for critical applications:

    kubectl rollout status deployment/<name> -n <namespace>

Replace with your actual deployment names, e.g., kubectl rollout status deployment/web-app -n production.

1. Check events for recent warnings:

    kubectl get events --all-namespaces --sort-by='.lastTimestamp' | tail -20

Or filter for warnings:

    kubectl get events --all-namespaces --field-selector type=Warning --sort-by='.lastTimestamp'

1. Resource usage (if metrics-server installed):

    kubectl top nodes
    kubectl top pods --all-namespaces --sort-by=memory

Look for nodes or pods exceeding 80% of capacity.

1. Alert status in Prometheus (if configured):

    kubectl port-forward -n monitoring svc/prometheus-kube-prometheus-prometheus 9090:9090
    # then query http://localhost:9090/api/v1/alerts

Or use curl:

    curl -s http://localhost:9090/api/v1/alerts | jq '.data.alerts[] | select(.state=="firing")'

Incident response checklist

When an alert fires or a user reports an issue, follow this ordered list.

1. Acknowledge and record time in your incident management tool. Example entry: 2025-08-01T14:23:00Z - Received alert PodCrashLooping for namespace=production, pod=web-app-abcde.

1. Isolate the blast radius: Which namespace, which service, how many users affected? Example command:

    kubectl get pods -n production -o wide

Determine if it is a single pod or all replicas.

1. Collect evidence (read-only) before changing anything:

    kubectl describe pod <pod-name> -n production > incident-<timestamp>-describe.txt
    kubectl logs <pod-name> -n production --previous > incident-<timestamp>-logs.txt
    kubectl get events -n production --sort-by='.lastTimestamp' > incident-<timestamp>-events.txt

1. Form a hypothesis based on evidence. For example: "Pod is crash looping due to missing DATABASE_URL environment variable, as shown in logs."

1. Apply the smallest fix (with approval if needed). Example:

    kubectl set env deployment/web-app -n production --from=secret/db-secret --keys=DATABASE_URL

1. Verify recovery:

    kubectl rollout status deployment/web-app -n production
    kubectl get pods -n production -l app=web-app

Ensure pods are Running and restarts not increasing.

1. Document root cause and prevention. Example: "Root cause: missing env var in deployment spec. Prevention: add CI check to validate deployment env vars against required list. Add alert for crash loops as done in Safe Configuration Path."

Conclusion

Kubectl monitoring and alerts with practical examples is useful only when each recommendation is version-scoped, observable, and reversible where the technology permits. Copying a command without checking prerequisites and expected output is not an operations procedure.

As a next step, choose one low-risk verification for kubectl monitoring, record the current state, run the documented check, compare the result with the expected signal, and review dependencies such as namespace, pod, and deployment.

A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision.

Implement the daily monitoring checklist and the safe configuration path for alert rules. Test your incident response with a simulated crash pod. Then expand to custom metrics and dashboards as your confidence grows.