Kubernetes Pod Lifecycle Monitoring and Alerts: A Practical Implementation Guide

Intro

Kubernetes Pod Lifecycle monitoring with practical examples helps operators move from an observed problem to a verified result. The pod lifecycle is the sequence of states a pod passes through from creation to termination: Pending, Running, Succeeded, Failed, and CrashLoopBackOff. Monitoring these states and alerting on unexpected transitions is essential for service reliability.

This article provides a practical, command-driven guide for developers, DevOps consultants, and technical startup teams. It covers how to:

Inspect pod lifecycle states and events with kubectl
Collect pod metrics with kube-state-metrics and query them in Prometheus
Build alerts that fire when pods fail, crash, or remain pending
Diagnose common failure modes and recover safely
Create a reusable operations checklist for incident response

Every command includes expected output and a signal to verify. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document recovery.

Version and Environment Inventory

Before monitoring pod lifecycles, inventory your environment. Name the relevant component, supported version range, prerequisites, and the read-only commands that verify the current state.

Example baseline inventory:

Kubernetes cluster: v1.28
kubectl client: v1.28 (matches server minor version)
kube-state-metrics: v2.10
Prometheus: v2.50
Alertmanager: v0.26
Monitoring namespace: monitoring

Start with read-only observation. Capture current pods across all namespaces:

kubectl get pods -A -o wide

Expected output shows pod name, namespace, status, restarts, age, IP, and node.

To inspect a specific pod's scheduling and lifecycle events, use:

kubectl describe pod <pod-name> -n <namespace>

The Events section at the bottom reveals warnings like FailedScheduling, Failed to pull image, or Back-off restarting failed container. This is the first place to look for failures.

Version compatibility: kube-state-metrics v2.x requires Kubernetes v1.24 or later. Prometheus v2.50 supports all current Kubernetes versions. Always match kubectl to your cluster's minor version to avoid API mismatches.

Blast radius rule: Before applying any change, record the current pod count and restarts. Use kubectl get pods -A --no-headers | wc -l and kubectl get pods -A -o jsonpath='{.items[].status.containerStatuses[].restartCount}'. Store these values for comparison after changes.

Safe Configuration Path

Monitoring pod lifecycles often requires deploying or configuring kube-state-metrics, Prometheus, and alert rules. Follow a safe configuration path: apply one manifest, inspect the generated resources, and verify before moving to production.

Step 1: Deploy kube-state-metrics

Use the official manifest, pinned to a version:

kubectl apply -f https://github.com/kubernetes/kube-state-metrics/releases/download/v2.10.0/standard.yaml

Expected output confirms the deployment, service account, and ClusterRole.

Verify the pods are running:

kubectl get pods -n kube-system -l app.kubernetes.io/name=kube-state-metrics

Expected: one pod with Running status and zero restarts.

Step 2: Expose metrics

Create a ServiceMonitor if using the Prometheus Operator, or add a scrape config to your Prometheus config map.

Example ServiceMonitor snippet:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: kube-state-metrics
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: kube-state-metrics
  endpoints:
  - port: http-metrics
    interval: 30s

Apply and verify Prometheus discovers the target:

kubectl apply -f servicemonitor.yaml
kubectl port-forward -n monitoring svc/prometheus-k8s 9090:9090

Open http://localhost:9090/targets in a browser and confirm the kube-state-metrics target is up.

Step 3: Set up alerting rules

Create a PrometheusRule for pod failures. Example rule for CrashLoopBackOff:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: pod-lifecycle-alerts
  namespace: monitoring
spec:
  groups:
  - name: pod-lifecycle
    rules:
    - alert: PodCrashLooping
      expr: |
        increase(kube_pod_container_status_restarts_total[10m]) > 5
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} is crash looping"
        description: "Container {{ $labels.container }} has restarted {{ $value }} times in the last 10 minutes."

Apply and verify the rule appears in Prometheus:

kubectl apply -f prometheusrule.yaml
kubectl port-forward -n monitoring svc/prometheus-k8s 9090:9090
# Navigate to http://localhost:9090/rules to see the rule.

Rollback path: Keep the previous manifest in version control. If the new rule causes false positives, delete it with kubectl delete prometheusrule pod-lifecycle-alerts -n monitoring and restore the previous file.

Verification and Diagnostics

Once monitoring is configured, verify it works by simulating failures and observing metrics and alerts.

Check Pod Status Manually

Use kubectl get pods to see lifecycle states. To watch state transitions in real time:

kubectl get pods -n default --watch

This streams changes until you press Ctrl+C.

Inspect Logs and History

For a restarting container, view current and previous logs:

kubectl logs <pod-name> -n <namespace> --previous

The --previous flag shows logs from the last terminated container, which often contains the error that caused the crash.

Check rollout status for a Deployment:

kubectl rollout status deployment/<deployment-name> -n <namespace>

Expected output: deployment "<name>" successfully rolled out. If the deployment is stuck, the command exits with an error, and you should inspect events.

Query Prometheus Metrics

kube-state-metrics exposes pod lifecycle metrics. Common queries:

Total pods per phase:

  sum by (phase) (kube_pod_status_phase)

Pods in Pending state longer than 5 minutes:

  sum(kube_pod_status_phase{phase="Pending"}) by (namespace, pod)

Pod restarts in the last hour:

  increase(kube_pod_container_status_restarts_total[1h])

Pods not ready:

  kube_pod_status_ready{condition="false"}

Run these in the Prometheus UI (http://localhost:9090/graph after port-forward) and verify the values match kubectl get pods.

Trigger a Test Alert

Deploy a pod that exits immediately to see CrashLoopBackOff:

apiVersion: v1
kind: Pod
metadata:
  name: crash-loop-test
spec:
  containers:
  - name: busybox
    image: busybox
    command: ["sh", "-c", "exit 1"]

After a few minutes, the PodCrashLooping alert should fire. Check with:

kubectl get pods crash-loop-test
# Output shows RESTARTS increasing and STATUS CrashLoopBackOff

Then in Prometheus Alerts page (http://localhost:9090/alerts), you should see the alert in Pending or Firing state.

Clean up the test pod to avoid noise:

kubectl delete pod crash-loop-test

Failure Modes and Recovery

Pods can fail for many reasons. Recognizing patterns speeds up recovery. The table below lists common failure modes, their typical symptoms, diagnostic commands, and recovery actions.

Failure Mode	Symptom	Diagnostic Command	Recovery Action
Image pull error	Status `ErrImagePull` or `ImagePullBackOff`	`kubectl describe pod <name>` shows `Failed to pull image`	Fix image name/tag or imagePullSecret, then delete pod to recreate
CrashLoopBackOff	Container starts then exits, restarts increasing	`kubectl logs <name> --previous`	Examine error in logs, fix application config or code, roll back Deployment
Pending (unschedulable)	Pod stuck in Pending	`kubectl describe pod <name>` shows `FailedScheduling` due to insufficient resources or node selector	Scale nodes, adjust resource requests, or fix node affinity
OOMKilled	Container terminated with reason OOMKilled	`kubectl describe pod <name>` shows `OOMKilled` in last state	Increase memory limit or reduce app memory usage, then restart pod
Liveness probe failure	Pod restarts repeatedly but container logs show no application error	`kubectl describe pod <name>` shows `Liveness probe failed`	Adjust probe timing or fix application health endpoint
Readiness probe failure	Pod Running but not Ready, service doesn't route traffic	`kubectl get pods` shows `0/1 Running`, `kubectl describe pod` shows `Readiness probe failed`	Fix readiness endpoint or delay probe, ensure dependencies ready
Node failure	Pods on a node go to Terminating or Unknown	`kubectl get nodes` shows NotReady	Drain or cordon node, pods reschedule automatically if part of a controller

Recovery procedure for CrashLoopBackOff:

Get pod details: kubectl describe pod <pod-name> -n <namespace>
Check last termination reason: look for Error or OOMKilled
View previous logs: kubectl logs <pod-name> -n <namespace> --previous
If caused by a recent deployment, roll back: kubectl rollout undo deployment/<deployment-name>
If resource issue, edit resource limits and apply: kubectl apply -f updated-deployment.yaml
Verify pod is Running and restarts don't increase: kubectl get pods -n <namespace> --watch

Preventive measures:

Set memory and CPU limits for all containers to avoid OOMKills
Configure liveness and readiness probes appropriate to your app startup time
Use spec.template.metadata.labels and pod disruption budgets for graceful eviction
Monitor kube_pod_status_phase and kube_pod_container_status_restarts_total with dashboards

Operations Checklist

Use this checklist during incident response or routine maintenance to systematically verify pod lifecycle health.

Pre-incident (routine):

[ ] Cluster version and monitoring stack versions recorded (e.g., K8s v1.28, kube-state-metrics v2.10)
[ ] kube-state-metrics pods healthy: kubectl get pods -n kube-system -l app.kubernetes.io/name=kube-state-metrics
[ ] Prometheus targets up: port-forward and check /targets
[ ] Alert rules loaded: check /rules
[ ] Baseline pod count and restart metrics stored

During incident:

[ ] Identify affected namespace and pod: kubectl get pods -A --field-selector=status.phase=Failed or =Pending
[ ] Check pod events: kubectl describe pod <name> -n <ns> | tail -20
[ ] Check container logs: kubectl logs <name> -n <ns> --previous
[ ] Check resource usage: kubectl top pod <name> -n <ns> (requires metrics-server)
[ ] Query relevant Prometheus metric (e.g., increase(kube_pod_container_status_restarts_total[10m]))
[ ] Determine if a recent change caused failure (deployment, config map, secret)
[ ] If yes, rollback: kubectl rollout undo deployment/<name> -n <ns>
[ ] If no, proceed with specific recovery from failure modes table
[ ] Verify recovery: kubectl get pods -n <ns> --watch until stable

Post-incident:

[ ] Document root cause and timeline
[ ] Adjust alerts if they missed or false triggered
[ ] Update runbooks with new diagnostic steps
[ ] Review resource limits and probes
[ ] Delete any test pods used

Example runbook entry for OOMKilled:

Symptom: Pod restarts with OOMKilled in kubectl describe pod.
Confirm: kubectl get pod <name> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}' returns OOMKilled.
Check current limits: kubectl get pod <name> -o jsonpath='{.spec.containers[0].resources}'
Increase memory limit by 25% in deployment manifest, apply.
Watch for stable restarts: kubectl get pods --watch.

Conclusion

Kubernetes Pod Lifecycle monitoring and alerts are effective only when each recommendation is version-scoped, observable, and reversible. Copying a command without checking prerequisites and expected output is not an operations procedure.

As a next step, choose one low-risk verification from this article: deploy kube-state-metrics, create a simple alert rule, or simulate a CrashLoopBackOff test pod. Record the current state, run the documented check, and compare the result with the expected signal. Review dependencies such as Pod, Node, and Event.

A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision.