## Intro

Kubernetes Pod Lifecycle monitoring with practical examples helps operators move from an observed problem to a verified result. The pod lifecycle is the sequence of states a pod passes through from creation to termination: Pending, Running, Succeeded, Failed, and CrashLoopBackOff. Monitoring these states and alerting on unexpected transitions is essential for service reliability.

This article provides a practical, command-driven guide for developers, DevOps consultants, and technical startup teams. It covers how to:

- Inspect pod lifecycle states and events with kubectl
- Collect pod metrics with kube-state-metrics and query them in Prometheus
- Build alerts that fire when pods fail, crash, or remain pending
- Diagnose common failure modes and recover safely
- Create a reusable operations checklist for incident response

Every command includes expected output and a signal to verify. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document recovery.

## Version and Environment Inventory

Before monitoring pod lifecycles, inventory your environment. Name the relevant component, supported version range, prerequisites, and the read-only commands that verify the current state.

**Example baseline inventory:**
- Kubernetes cluster: v1.28
- kubectl client: v1.28 (matches server minor version)
- kube-state-metrics: v2.10
- Prometheus: v2.50
- Alertmanager: v0.26
- Monitoring namespace: `monitoring`

Start with read-only observation. Capture current pods across all namespaces:

```bash
kubectl get pods -A -o wide
```

Expected output shows pod name, namespace, status, restarts, age, IP, and node.

To inspect a specific pod's scheduling and lifecycle events, use:

```bash
kubectl describe pod <pod-name> -n <namespace>
```

The Events section at the bottom reveals warnings like `FailedScheduling`, `Failed to pull image`, or `Back-off restarting failed container`. This is the first place to look for failures.

**Version compatibility:** kube-state-metrics v2.x requires Kubernetes v1.24 or later. Prometheus v2.50 supports all current Kubernetes versions. Always match kubectl to your cluster's minor version to avoid API mismatches.

**Blast radius rule:** Before applying any change, record the current pod count and restarts. Use `kubectl get pods -A --no-headers | wc -l` and `kubectl get pods -A -o jsonpath='{.items[*].status.containerStatuses[*].restartCount}'`. Store these values for comparison after changes.

## Safe Configuration Path

Monitoring pod lifecycles often requires deploying or configuring kube-state-metrics, Prometheus, and alert rules. Follow a safe configuration path: apply one manifest, inspect the generated resources, and verify before moving to production.

**Step 1: Deploy kube-state-metrics**

Use the official manifest, pinned to a version:

```bash
kubectl apply -f https://github.com/kubernetes/kube-state-metrics/releases/download/v2.10.0/standard.yaml
```

Expected output confirms the deployment, service account, and ClusterRole.

Verify the pods are running:

```bash
kubectl get pods -n kube-system -l app.kubernetes.io/name=kube-state-metrics
```

Expected: one pod with `Running` status and zero restarts.

**Step 2: Expose metrics**

Create a ServiceMonitor if using the Prometheus Operator, or add a scrape config to your Prometheus config map.

Example ServiceMonitor snippet:

```yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: kube-state-metrics
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: kube-state-metrics
  endpoints:
  - port: http-metrics
    interval: 30s
```

Apply and verify Prometheus discovers the target:

```bash
kubectl apply -f servicemonitor.yaml
kubectl port-forward -n monitoring svc/prometheus-k8s 9090:9090
```

Open `http://localhost:9090/targets` in a browser and confirm the kube-state-metrics target is up.

**Step 3: Set up alerting rules**

Create a PrometheusRule for pod failures. Example rule for CrashLoopBackOff:

```yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: pod-lifecycle-alerts
  namespace: monitoring
spec:
  groups:
  - name: pod-lifecycle
    rules:
    - alert: PodCrashLooping
      expr: |
        increase(kube_pod_container_status_restarts_total[10m]) > 5
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} is crash looping"
        description: "Container {{ $labels.container }} has restarted {{ $value }} times in the last 10 minutes."
```

Apply and verify the rule appears in Prometheus:

```bash
kubectl apply -f prometheusrule.yaml
kubectl port-forward -n monitoring svc/prometheus-k8s 9090:9090
# Navigate to http://localhost:9090/rules to see the rule.
```

**Rollback path:** Keep the previous manifest in version control. If the new rule causes false positives, delete it with `kubectl delete prometheusrule pod-lifecycle-alerts -n monitoring` and restore the previous file.

## Verification and Diagnostics

Once monitoring is configured, verify it works by simulating failures and observing metrics and alerts.

### Check Pod Status Manually

Use `kubectl get pods` to see lifecycle states. To watch state transitions in real time:

```bash
kubectl get pods -n default --watch
```

This streams changes until you press Ctrl+C.

### Inspect Logs and History

For a restarting container, view current and previous logs:

```bash
kubectl logs <pod-name> -n <namespace> --previous
```

The `--previous` flag shows logs from the last terminated container, which often contains the error that caused the crash.

Check rollout status for a Deployment:

```bash
kubectl rollout status deployment/<deployment-name> -n <namespace>
```

Expected output: `deployment "<name>" successfully rolled out`. If the deployment is stuck, the command exits with an error, and you should inspect events.

### Query Prometheus Metrics

kube-state-metrics exposes pod lifecycle metrics. Common queries:

- Total pods per phase:
  ```promql
  sum by (phase) (kube_pod_status_phase)
  ```
- Pods in Pending state longer than 5 minutes:
  ```promql
  sum(kube_pod_status_phase{phase="Pending"}) by (namespace, pod)
  ```
- Pod restarts in the last hour:
  ```promql
  increase(kube_pod_container_status_restarts_total[1h])
  ```
- Pods not ready:
  ```promql
  kube_pod_status_ready{condition="false"}
  ```

Run these in the Prometheus UI (http://localhost:9090/graph after port-forward) and verify the values match `kubectl get pods`.

### Trigger a Test Alert

Deploy a pod that exits immediately to see CrashLoopBackOff:

```yaml
apiVersion: v1
kind: Pod
metadata:
  name: crash-loop-test
spec:
  containers:
  - name: busybox
    image: busybox
    command: ["sh", "-c", "exit 1"]
```

After a few minutes, the `PodCrashLooping` alert should fire. Check with:

```bash
kubectl get pods crash-loop-test
# Output shows RESTARTS increasing and STATUS CrashLoopBackOff
```

Then in Prometheus Alerts page (http://localhost:9090/alerts), you should see the alert in Pending or Firing state.

Clean up the test pod to avoid noise:

```bash
kubectl delete pod crash-loop-test
```

## Failure Modes and Recovery

Pods can fail for many reasons. Recognizing patterns speeds up recovery. The table below lists common failure modes, their typical symptoms, diagnostic commands, and recovery actions.

| Failure Mode | Symptom | Diagnostic Command | Recovery Action |
|---|---|---|---|
| Image pull error | Status `ErrImagePull` or `ImagePullBackOff` | `kubectl describe pod <name>` shows `Failed to pull image` | Fix image name/tag or imagePullSecret, then delete pod to recreate |
| CrashLoopBackOff | Container starts then exits, restarts increasing | `kubectl logs <name> --previous` | Examine error in logs, fix application config or code, roll back Deployment |
| Pending (unschedulable) | Pod stuck in Pending | `kubectl describe pod <name>` shows `FailedScheduling` due to insufficient resources or node selector | Scale nodes, adjust resource requests, or fix node affinity |
| OOMKilled | Container terminated with reason OOMKilled | `kubectl describe pod <name>` shows `OOMKilled` in last state | Increase memory limit or reduce app memory usage, then restart pod |
| Liveness probe failure | Pod restarts repeatedly but container logs show no application error | `kubectl describe pod <name>` shows `Liveness probe failed` | Adjust probe timing or fix application health endpoint |
| Readiness probe failure | Pod Running but not Ready, service doesn't route traffic | `kubectl get pods` shows `0/1 Running`, `kubectl describe pod` shows `Readiness probe failed` | Fix readiness endpoint or delay probe, ensure dependencies ready |
| Node failure | Pods on a node go to Terminating or Unknown | `kubectl get nodes` shows NotReady | Drain or cordon node, pods reschedule automatically if part of a controller |

**Recovery procedure for CrashLoopBackOff:**

1. Get pod details: `kubectl describe pod <pod-name> -n <namespace>`
2. Check last termination reason: look for `Error` or `OOMKilled`
3. View previous logs: `kubectl logs <pod-name> -n <namespace> --previous`
4. If caused by a recent deployment, roll back: `kubectl rollout undo deployment/<deployment-name>`
5. If resource issue, edit resource limits and apply: `kubectl apply -f updated-deployment.yaml`
6. Verify pod is Running and restarts don't increase: `kubectl get pods -n <namespace> --watch`

**Preventive measures:**
- Set memory and CPU limits for all containers to avoid OOMKills
- Configure liveness and readiness probes appropriate to your app startup time
- Use `spec.template.metadata.labels` and pod disruption budgets for graceful eviction
- Monitor `kube_pod_status_phase` and `kube_pod_container_status_restarts_total` with dashboards

## Operations Checklist

Use this checklist during incident response or routine maintenance to systematically verify pod lifecycle health.

**Pre-incident (routine):**
- [ ] Cluster version and monitoring stack versions recorded (e.g., K8s v1.28, kube-state-metrics v2.10)
- [ ] kube-state-metrics pods healthy: `kubectl get pods -n kube-system -l app.kubernetes.io/name=kube-state-metrics`
- [ ] Prometheus targets up: port-forward and check /targets
- [ ] Alert rules loaded: check /rules
- [ ] Baseline pod count and restart metrics stored

**During incident:**
- [ ] Identify affected namespace and pod: `kubectl get pods -A --field-selector=status.phase=Failed` or `=Pending`
- [ ] Check pod events: `kubectl describe pod <name> -n <ns> | tail -20`
- [ ] Check container logs: `kubectl logs <name> -n <ns> --previous`
- [ ] Check resource usage: `kubectl top pod <name> -n <ns>` (requires metrics-server)
- [ ] Query relevant Prometheus metric (e.g., `increase(kube_pod_container_status_restarts_total[10m])`)
- [ ] Determine if a recent change caused failure (deployment, config map, secret)
- [ ] If yes, rollback: `kubectl rollout undo deployment/<name> -n <ns>`
- [ ] If no, proceed with specific recovery from failure modes table
- [ ] Verify recovery: `kubectl get pods -n <ns> --watch` until stable

**Post-incident:**
- [ ] Document root cause and timeline
- [ ] Adjust alerts if they missed or false triggered
- [ ] Update runbooks with new diagnostic steps
- [ ] Review resource limits and probes
- [ ] Delete any test pods used

**Example runbook entry for OOMKilled:**

1. Symptom: Pod restarts with `OOMKilled` in `kubectl describe pod`.
2. Confirm: `kubectl get pod <name> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'` returns `OOMKilled`.
3. Check current limits: `kubectl get pod <name> -o jsonpath='{.spec.containers[0].resources}'`
4. Increase memory limit by 25% in deployment manifest, apply.
5. Watch for stable restarts: `kubectl get pods --watch`.

## Conclusion

Kubernetes Pod Lifecycle monitoring and alerts are effective only when each recommendation is version-scoped, observable, and reversible. Copying a command without checking prerequisites and expected output is not an operations procedure.

As a next step, choose one low-risk verification from this article: deploy kube-state-metrics, create a simple alert rule, or simulate a CrashLoopBackOff test pod. Record the current state, run the documented check, and compare the result with the expected signal. Review dependencies such as Pod, Node, and Event.

A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision.