Intro
When production incidents happen, Kubernetes operators move from an observed problem to a verified result. This checklist helps developers, DevOps consultants, and technical startup teams work safely with Kubernetes Event workloads by connecting operations, best practices, and maintenance to specific commands, expected output, failure signals, and recovery decisions.
The core principle is operational safety: observe before changing, limit blast radius, never paste secrets into commands, verify the result, and document how to recover if the expected state is not reached. Each section names the relevant component, supported version range (Kubernetes 1.24 to 1.30 where applicable), prerequisites, read-only observations, the smallest justified change, and the verification command.
This guide is written for engineers who manage stateful event-driven services on Kubernetes. It assumes you have kubectl installed and configured for a cluster, and that you know the namespace and name of the workload you are investigating. All examples use placeholder values like <pod-name> or <namespace> which you should replace with your actual values.
Version and Environment Inventory
Before touching anything, build a version and environment inventory. This prevents the classic mistake of applying a fix designed for one Kubernetes version to another and makes it easier to reproduce issues or roll back changes.
Start with cluster and client versions:
kubectl version --short
# Client Version: v1.29.2
# Server Version: v1.28.7If the client and server differ by more than one minor version, upgrade or use a matching client. Kubernetes supports a skew of one minor version between client and server. A mismatch can cause parsing errors or unsupported API fields.
Next, inspect the nodes and pods that run your event workload:
kubectl get nodes -o wide
# NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
# node-1 Ready control-plane 10d v1.28.7 10.0.1.10 none Ubuntu 22.04.3 LTS 5.15.0-91-generic containerd://1.7.13
# node-2 Ready <none> 10d v1.28.7 10.0.1.11 none Ubuntu 22.04.3 LTS 5.15.0-91-generic containerd://1.7.13Look for nodes in NotReady, SchedulingDisabled, or with high CPU/memory pressure. These conditions can prevent event pods from being scheduled or cause them to be evicted.
Then list pods in your workload namespace:
kubectl get pods -n production -o wide
# NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
# event-processor-7f9c8b5d6-8x2w 0/1 CrashLoopBackOff 4 (2m ago) 15m 10.244.2.5 node-2 <none> <none>
# event-queue-5d4f6b7c8-9q3z 1/1 Running 0 15m 10.244.1.8 node-1 <none> <none>A pod in CrashLoopBackOff immediately tells you the application inside is failing. Record the exact restart count and age because they help determine whether the problem is new or recurring.
Also check the Kubernetes events for the namespace:
kubectl get events -n production --sort-by='.lastTimestamp'
# LAST SEEN TYPE REASON OBJECT MESSAGE
# 2m Warning BackOff pod/event-processor-7f9c8b5d6-8x2w Back-off restarting failed container
# 5m Normal Scheduled pod/event-processor-7f9c8b5d6-8x2w Successfully assigned production/event-processor-7f9c8b5d6-8x2w to node-2
# 15m Normal Created pod/event-queue-5d4f6b7c8-9q3z Created container event-queue
# 15m Normal Started pod/event-queue-5d4f6b7c8-9q3z Started container event-queueEvents are often the first place to look when a pod is not behaving. They show scheduling decisions, image pull failures, probe failures, and evictions.
For deep inspection of a specific pod, use kubectl describe:
kubectl describe pod event-processor-7f9c8b5d6-8x2w -n productionLook at the Events section at the bottom for recent lifecycle events, and the Conditions section for PodScheduled, Initialized, ContainersReady, and Ready conditions. If a condition is False, the accompanying message often explains why.
Finally, verify the Kubernetes API version your manifests use. For example, events.k8s.io/v1 is stable since Kubernetes 1.19, but older clusters may still be on v1beta1. Check with:
kubectl api-versions | grep events
# events.k8s.io/v1Use the correct API version for your cluster to avoid manifest validation errors.
Safe Configuration Path
Making configuration changes in production should follow a safe path: small, scoped, and reversible. Never edit a live Deployment directly with kubectl edit; instead, update the manifest file, apply it, and observe the rollout.
Assume you need to increase the memory limit for the event-processor container from 512Mi to 1Gi because it is being OOMKilled. Start by checking current resource usage:
kubectl top pod event-processor-7f9c8b5d6-8x2w -n production
# NAME CPU(cores) MEMORY(bytes)
# event-processor-7f9c8b5d6-8x2w 120m 495MiMemory usage is near the 512Mi limit, so the pod is likely being killed when usage spikes. Check recent pod termination reasons:
kubectl describe pod event-processor-7f9c8b5d6-8x2w -n production | grep -A 5 'Last State'
# Last State: Terminated
# Reason: OOMKilled
# Exit Code: 137Exit code 137 indicates the container was killed by the OOM killer. That confirms the need for a higher memory limit.
Next, edit the Deployment manifest file (never kubectl edit in production without a backup). Find the container spec:
# event-processor-deployment.yaml (excerpt)
containers:
- name: event-processor
image: registry.example.com/event-processor:1.4.2
resources:
limits:
memory: "512Mi"
requests:
memory: "256Mi"Change the limit to 1Gi and optionally the request to 512Mi to reflect expected usage. Then apply the change:
kubectl apply -f event-processor-deployment.yaml -n production
# deployment.apps/event-processor configuredAfter applying, watch the rollout status:
kubectl rollout status deployment/event-processor -n production
# Waiting for deployment "event-processor" rollout to finish: 1 out of 3 new replicas have been updated...
# deployment "event-processor" successfully rolled outIf the rollout fails, check the new pod's logs and events. For example, if the new pod is still OOMKilling because 1Gi is not enough, the rollout may be stuck. You can pause the rollout:
kubectl rollout pause deployment/event-processor -n productionInvestigate, adjust the limit again, then resume:
kubectl rollout resume deployment/event-processor -n productionIf the change causes problems, roll back to the previous revision:
kubectl rollout undo deployment/event-processor -n production
# deployment.apps/event-processor rolled backAlways record the previous revision number with kubectl rollout history deployment/event-processor -n production so you can specify --to-revision if needed.
For configuration changes that involve ConfigMaps or Secrets, use the same principle. Do not use kubectl create secret with plain text on the command line because it appears in shell history. Instead, create the Secret from a file:
echo -n 's3cr3tValue' > secret.txt
kubectl create secret generic event-processor-secret --from-file=api-key=secret.txt -n production
rm secret.txtThen reference the secret key in your pod spec. Use kubectl get secret event-processor-secret -n production -o jsonpath='{.data.api-key}' | base64 --decode only for verification, not in logs.
Verification and Diagnostics
Verification and diagnostics are the heart of Kubernetes operations. You need a systematic approach to confirm that your event workload is functioning correctly and to diagnose issues when it is not.
Start with the basics: check pod status, logs, and events.
kubectl get pods -n production -l app=event-processor -o wide
# NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
# event-processor-7f9c8b5d6-8x2w 1/1 Running 0 30m 10.244.2.5 node-2 <none> <none>If a pod is not Running, use kubectl describe to see events. If it is crash looping, fetch logs from the current and previous container instances:
kubectl logs event-processor-7f9c8b5d6-8x2w -n production --tail=50
# ... application logs ...
kubectl logs event-processor-7f9c8b5d6-8x2w -n production --previous --tail=50
# ... logs from the previous crashed instance, which may show the fatal error ...For multi-container pods, specify the container name with -c <container-name>.
Next, check the health of your event service. If you have a Service and Ingress, verify endpoints and connectivity.
kubectl get svc event-processor -n production
# NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
# event-processor ClusterIP 10.96.52.114 <none> 8080/TCP 30m
kubectl get endpoints event-processor -n production
# NAME ENDPOINTS AGE
# event-processor 10.244.2.5:8080 30mIf endpoints are empty, the Service selector does not match any pod labels, or the pods are not ready. Compare the Service selector and pod labels.
Test connectivity from within the cluster:
kubectl run curl-test --rm -it --image=curlimages/curl -n production -- sh
# From inside the pod:
curl http://event-processor.production.svc.cluster.local:8080/healthz
# Expected: {"status":"ok"}For event-driven systems, verify that messages are being consumed. If you use Kafka, check consumer group lag. For example, using kafka-consumer-groups.sh:
kafka-consumer-groups.sh --bootstrap-server kafka-broker:9092 --describe --group event-processor
# TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
# events 0 100 150 50 consumer-1-... /10.244.2.5 consumer-1
# events 1 200 250 50 consumer-1-... /10.244.2.5 consumer-1High lag means the consumer is falling behind. Check application logs for processing slowdowns or errors.
Probe configuration is critical. Ensure liveness and readiness probes are correctly set. For an HTTP service, your deployment might look like:
livenessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 15
periodSeconds: 20
readinessProbe:
httpGet:
path: /readyz
port: 8080
initialDelaySeconds: 5
periodSeconds: 10Check probe success with kubectl describe pod; if probes are failing, the pod restarts frequently or is marked unready.
For deeper diagnostics, use kubectl exec to run commands inside the container and inspect the filesystem or environment:
kubectl exec -it event-processor-7f9c8b5d6-8x2w -n production -- sh
# Inside container: check environment variables, config files, run diagnostic commandsAlways prefer kubectl logs and kubectl exec for read-only checks before making changes.
Failure Modes and Recovery
Production systems fail in predictable ways. Knowing the common failure modes for Kubernetes event workloads and how to recover from them will reduce downtime.
1. Pod CrashLoopBackOff
Symptom: Pod status is CrashLoopBackOff, restart count increases.
Diagnosis: Use kubectl logs <pod> --previous to see the error from the last crash. Common causes: missing configuration, invalid arguments, unhandled exceptions, out-of-memory.
Recovery: Fix the underlying cause in the image or configuration, then apply the change. If the pod is stuck in a crash loop and you need to quickly stop it, scale the deployment to zero:
kubectl scale deployment event-processor --replicas=0 -n productionInvestigate, fix, then scale back up. This stops the crash loop and frees resources.
2. ImagePullBackOff
Symptom: Pod status is ImagePullBackOff or ErrImagePull.
Diagnosis: Describe the pod and check events:
kubectl describe pod event-processor-7f9c8b5d6-8x2w -n production | grep -A 10 Events
# Events:
# Type Reason Age From Message
# ---- ------ ---- ---- -------
# Normal Scheduled 10m default-scheduler Successfully assigned production/event-processor-7f9c8b5d6-8x2w to node-2
# Warning Failed 9m kubelet Failed to pull image "registry.example.com/event-processor:1.4.3": rpc error: code = NotFound desc = failed to pull and unpack image
# Warning Failed 8m kubelet Error: ErrImagePull
# Normal BackOff 7m kubelet Back-off pulling image "registry.example.com/event-processor:1.4.3"Recovery: Check the image name and tag. Ensure the registry credentials are correct and the image exists. If the tag is wrong, update the deployment to use a valid tag. If it is a private registry, verify the imagePullSecret is correct and present in the namespace.
3. Out of Memory (OOMKilled)
Symptom: Pod terminates with reason OOMKilled and exit code 137.
Diagnosis: Check the pod's last state:
kubectl describe pod event-processor-7f9c8b5d6-8x2w -n production | grep -A 5 'Last State'
# Last State: Terminated
# Reason: OOMKilled
# Exit Code: 137Also check node memory pressure events.
Recovery: Increase the container's memory limit, or reduce its memory usage by optimizing the application or scaling horizontally. Ensure requests are set appropriately; a too-low request can lead to eviction under node pressure.
4. Node NotReady
Symptom: One or more nodes show NotReady, and pods on them may be evicted or unresponsive.
Diagnosis: Check node status and conditions:
kubectl get nodes
# NAME STATUS ROLES AGE VERSION
# node-1 Ready control-plane 10d v1.28.7
# node-2 NotReady <none> 10d v1.28.7
kubectl describe node node-2 | grep -A 10 Conditions
# Conditions:
# Type Status LastHeartbeatTime LastTransitionTime Reason Message
# ---- ------ ----------------- ------------------ ------ -------
# MemoryPressure False Thu, 13 Jun 2024 10:00:00 +0000 Thu, 13 Jun 2024 09:00:00 +0000 KubeletHasSufficientMemory kubelet has sufficient memory available
# DiskPressure False ...
# PIDPressure False ...
# Ready False Thu, 13 Jun 2024 10:05:00 +0000 Thu, 13 Jun 2024 10:05:00 +0000 KubeletNotReady container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin is not readyThe Ready condition is False with reason KubeletNotReady and a message about network plugin. This often indicates a CNI plugin failure on the node.
Recovery: Check the kubelet status on the node (if accessible): systemctl status kubelet, journalctl -u kubelet -n 100. If the network plugin is not ready, the pod network may need to be restarted. If the node cannot be recovered quickly, cordon and drain it to move pods to healthy nodes:
kubectl cordon node-2
kubectl drain node-2 --ignore-daemonsets --delete-emptydir-dataThen investigate and fix the node. After recovery, uncordon it:
kubectl uncordon node-25. Event Delivery Failure (Consumer Lag)
Symptom: Event consumers are not processing messages as fast as they are produced; consumer group lag increases.
Diagnosis: Check consumer group lag (as shown earlier). Also check consumer pod logs for processing errors.
Recovery: Scale up the consumer deployment to increase processing capacity:
kubectl scale deployment event-processor --replicas=5 -n productionEnsure the event source (broker) is healthy and not throttling. If there is a poison message causing repeated retries, consider moving it to a dead letter queue or fixing the consumer to handle it gracefully.
For all failure modes, document the incident and recovery steps. Use kubectl get events --sort-by='.lastTimestamp' -n production to capture the timeline. Store these records in a runbook for future reference.
Operations Checklist
Use this checklist as a daily or pre/post-change routine for Kubernetes event production workloads.
Read-only observation
- [ ] Check cluster and client versions with
kubectl version --short. Note any skew beyond one minor version. - [ ] List nodes and pods with
kubectl get nodes -o wideandkubectl get pods -n production -o wide. Record any non-Ready nodes or pods with restarts. - [ ] Review recent events with
kubectl get events -n production --sort-by='.lastTimestamp'. Look for warnings or errors. - [ ] For key pods, run
kubectl describe pod <pod-name> -n productionand inspect conditions and events. - [ ] Check resource usage with
kubectl top pod -n productionandkubectl top node. Identify pods or nodes exceeding thresholds (e.g., >80% memory or CPU). - [ ] Verify service endpoints with
kubectl get endpoints -n production. Ensure all expected pods are listed. - [ ] For event consumers, check consumer group lag and compare with thresholds.
Safe change procedure
- [ ] Before any change, capture current state:
kubectl get deployment <name> -o yaml -n production > backup.yaml. - [ ] Modify the manifest file, not live objects.
- [ ] Apply the change with
kubectl apply -f <file> -n production. - [ ] Monitor rollout with
kubectl rollout status deployment/<name> -n production. - [ ] If rollout fails, investigate logs and events of new pods. Use
kubectl rollout undoif necessary. - [ ] After successful rollout, verify the new pod's status and logs.
After-change verification
- [ ] Check pod readiness:
kubectl get pods -n production -l app=<app>should show all pods Ready. - [ ] Run a smoke test: exec a curl into a pod or use
kubectl runto test the service endpoint. - [ ] Check consumer lag again to ensure processing is catching up.
- [ ] Review events for any new warnings.
- [ ] If the change involved resources, check that limits are sufficient and not causing OOMKills or throttling.
Documentation
- [ ] Record the change in your runbook with the time, target, command, and observed result.
- [ ] If rollback was needed, note the revision and reason.
- [ ] Update any dashboards or alerts based on observed metrics.
This checklist is not exhaustive but covers the most critical operations tasks. Adapt it to your environment, adding specific checks for your event broker, storage, and network configurations.
Conclusion
A Kubernetes Event production operations checklist becomes valuable only when it is version-scoped, observable, and reversible. Copying commands without checking prerequisites and expected output is not an operations procedure; it is a risk.
Start with a low-risk verification: choose one deployment in a development namespace, record its current state using kubectl get deployment -o yaml, run a read-only diagnostic like kubectl logs --tail=50, compare the result to the expected behavior, and only then make a small change and verify the rollout. Review dependencies such as cluster version, node health, and pod readiness after each step.
A reliable technical workflow makes failures visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident occurs. By following the checklists in this article, teams can operate event-driven workloads on Kubernetes with confidence and reduce the mean time to recovery when problems inevitably arise.
Next steps:
- Choose one production-adjacent workload and run the full read-only observation checklist.
- Document the results and compare with your baselines.
- Practice a safe change and rollback in a staging environment.
- Create or update your runbook with the failure modes described here.
Kubernetes operations are an ongoing practice. Regular use of these checklists and procedures will build the muscle memory needed for incident response and continuous improvement.