Learn how to safely upgrade and migrate running pods in Kubernetes with practical examples, including version inventory, configuration paths, verification, rollback, and recovery. This guide covers kubectl debug , ephemeral containers, and live debugging techniques to minimize disruption. ## Intro Upgrading and migrating running pods in Kubernetes is a delicate operation that requires careful planning to avoid downtime and service disruption. Debugging a live pod often involves attaching a debug container or using kubectl exec to inspect the environment. This guide provides practical examples for upgrading and migrating a running pod, including how to use kubectl debug to create a debugging session without disrupting the production container. We'll cover version inventory, safe configuration, verification, failure modes, recovery, and an operations checklist. By following these steps, you can minimize risk and ensure a smooth transition. ## Version and Environment Inventory Before making any changes, establish a clear picture of your current environment. This includes Kubernetes version, pod configuration, and the application's dependencies. Knowing these details helps you plan the upgrade and migration path. Start by checking the Kubernetes server version: kubectl version --short Expected output (example): Client Version: v1.24.0 Server Version: v1.24.0 Next, list the current pods and their images. Suppose you have a pod named myapp-pod in namespace default : kubectl get pod myapp-pod -o yaml This will output the full pod spec. Note the image field under containers . For example: containers: - name: app image: myapp:1.0 Record the image version. Also check the pod's labels and selectors if it is managed by a Deployment or StatefulSet: kubectl get deployment myapp-deployment -o yaml If you need to debug a running pod, you can create a debug container using kubectl debug . This is useful for inspecting the filesystem or network without altering the original container. Example: kubectl debug myapp-pod -it --image=busybox --target=app This command attaches a new container based on busybox to the same pod, sharing the process namespace if needed. You can then run commands inside the debug container to examine the environment. For example, to inspect the filesystem of the target container: # Inside the debug container ls /proc/1/root Or to check network connectivity: # Inside the debug container wget -O- http://localhost:8080/health Make sure you have the necessary RBAC permissions to perform these operations. The minimal permissions typically include get , list , create , and delete on pods and deployments. A minimal Role for debugging might look like: apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: namespace: default name: pod-debugger rules: - apiGroups: [""] resources: ["pods"] verbs: ["get", "list", "create", "delete"] - apiGroups: ["apps"] resources: ["deployments"] verbs: ["get", "list", "update", "patch"] Quick check 1 of 2 According to the article, what is the main purpose of using `kubectl debug` with ephemeral containers? - To restart a pod in a controlled manner - To inspect the state of an existing pod and run arbitrary commands without restarting it - To permanently add a new container to a running pod - To delete and replace a pod

The article states that ephemeral containers can be used to inspect the state of an existing pod and run arbitrary commands, which is useful for troubleshooting hard-to-reproduce bugs.

## Safe Configuration Path When upgrading or migrating a pod, use a controlled, scoped approach. Avoid changing multiple parameters at once. Instead, make incremental changes and test each one. For a controlled upgrade, consider using a new image tag. For example, to upgrade from myapp:1.0 to myapp:2.0 , update the deployment: kubectl set image deployment/myapp-deployment app=myapp:2.0 This will trigger a rolling update. To monitor the rollout: kubectl rollout status deployment/myapp-deployment Expected output: Waiting for deployment "myapp-deployment" rollout to finish: 1 out of 3 new replicas have been updated... deployment "myapp-deployment" successfully rolled out If you need to migrate a pod from one node to another, you can use a node selector or taints/tolerations. Suppose you want to move the pod to a node with label disktype=ssd . Patch the deployment with a node selector: kubectl patch deployment myapp-deployment -p '{"spec":{"template":{"spec":{"nodeSelector":{"disktype":"ssd"}}}}}' This will cause the deployment to create new pods on nodes that match the selector and terminate old pods. To ensure the migration is smooth, you might want to drain the old node first: kubectl drain --ignore-daemonsets --delete-emptydir-data This cordons the node and evicts pods. Then the scheduler places pods on other nodes. After draining, you can uncordon the node if needed: kubectl uncordon For debugging a running pod during migration, you can use kubectl debug with a copy of the pod to test changes without affecting the live pod. Example: kubectl debug myapp-pod --copy-to=myapp-debug --image=myapp:2.0 -- sleep 1h This creates a debug pod myapp-debug with the new image but does not serve traffic. You can then inspect the debug pod to validate the new configuration. To attach to the debug pod: kubectl exec -it myapp-debug -- sh Inside, you can run application-specific checks, such as verifying database connectivity or checking configuration files. ## Verification and Diagnostics After applying changes, verify that the pod is running correctly. Check the pod status: kubectl get pods Look for the pod in Running state with the correct number of restarts. For example: NAME READY STATUS RESTARTS AGE myapp-deployment-7f8c9d5b6-abcde 1/1 Running 0 2m Inspect logs to ensure there are no errors: kubectl logs myapp-deployment-7f8c9d5b6-abcde If the application exposes a health endpoint, test it using kubectl exec : kubectl exec myapp-deployment-7f8c9d5b6-abcde -- curl -f http://localhost:8080/health Expected output on success (example): {"status":"ok"} For debugging, you can attach to the running pod with kubectl debug . This can help diagnose issues that are not apparent from logs alone. Example: kubectl debug myapp-deployment-7f8c9d5b6-abcde -it --image=nicolaka/netshoot --target=app Inside the debug container, you can run network diagnostics or inspect environment variables. For instance, to check DNS resolution: nslookup kubernetes.default Or to view environment variables of the target process: cat /proc/1/environ | tr '\0' '\n' Additionally, check events for the pod to see any warnings: kubectl describe pod myapp-deployment-7f8c9d5b6-abcde Look under Events for messages like FailedScheduling or Liveness probe failed . These indicate problems that need attention.

Quick check 2 of 2 Which command creates a copy of a pod and changes the container image to `ubuntu` for all containers? - kubectl debug myapp --copy-to=myapp-debug --set-image=*=ubuntu - kubectl debug myapp --copy-to=myapp-debug --image=ubuntu - kubectl debug myapp --set-image=*=ubuntu - kubectl debug myapp --copy-to=myapp-debug --target=ubuntu

The article provides the exact command: `kubectl debug myapp --copy-to=myapp-debug --set-image=*=ubuntu` to copy the pod and change all container images to ubuntu.

## Failure Modes and Recovery Even with careful planning, upgrades and migrations can fail. Common failure modes include: - Image pull errors : The new image may not exist or credentials are missing. Check with kubectl describe pod and look for ErrImagePull or ImagePullBackOff . - CrashLoopBackOff : The new version crashes on startup. Check logs to identify the cause. - Scheduling failures : If node selectors or resource requests cannot be satisfied, pods remain Pending . - Service disruption : Rolling update may be too fast, causing temporary loss of capacity. To recover, you can rollback a Deployment to a previous revision. First, check rollout history: kubectl rollout history deployment/myapp-deployment Expected output: deployment.apps/myapp-deployment REVISION CHANGE-CAUSE 1 2 Then rollback to the previous revision: kubectl rollout undo deployment/myapp-deployment This reverts to revision 1. Monitor the rollback with kubectl rollout status . If a pod is stuck in Terminating state during node drain, you can force delete it: kubectl delete pod myapp-pod --grace-period=0 --force But be cautious with force deletion as it may leave resources. For debugging a failed pod, you can use kubectl debug to create a copy of the pod with a different command or image to reproduce the issue. For example, to start a debug pod that overrides the entrypoint with a shell: kubectl debug myapp-pod --copy-to=myapp-debug --image=myapp:2.0 -- /bin/sh Always have a backup of your Deployment or StatefulSet YAML so you can restore the previous configuration quickly. You can export the current configuration before changes: kubectl get deployment myapp-deployment -o yaml > myapp-deployment-backup.yaml ## Operations Checklist Use this checklist before, during, and after the upgrade/migration:

Step	Action	Verification
1	Record current pod image and configuration	`kubectl get pod <name> -o yaml`
2	Check Kubernetes version compatibility	`kubectl version --short`
3	Create a debug pod with new image to test	`kubectl debug <pod> --copy-to=<debug-pod> --image=<new-image>`
4	Update deployment with new image	`kubectl set image deployment/<name> app=<new-image>`
5	Monitor rollout	`kubectl rollout status deployment/<name>`
6	Verify pod health and logs	`kubectl get pods`, `kubectl logs <pod>`
7	If migration, drain old node	`kubectl drain <node> --ignore-daemonsets`
8	Check events for warnings	`kubectl describe pod <pod>`
9	Rollback if needed	`kubectl rollout undo deployment/<name>`

After completing the steps, document the changes and update any relevant runbooks. For example, note the new image version, the date of change, and any issues encountered. This helps with future audits and troubleshooting. ## Conclusion Upgrading and migrating running pods in Kubernetes can be done safely with proper planning and the right tools. By using kubectl debug , you can inspect and test changes without affecting production traffic. Always start with a version inventory, make scoped configuration changes, verify thoroughly, and have a rollback plan. The operations checklist provided will help you repeat the process consistently. Remember to monitor the rollout and be ready to undo if anything goes wrong. With these practices, you can minimize downtime and keep your applications running smoothly.