Kubernetes Debug Running Pod Upgrade and Migration: A Practical Implementation Guide

Learn how to safely upgrade and migrate running pods in Kubernetes with practical examples, including version inventory, configuration paths, verification, rollback, and recovery. This guide covers kubectl debug, ephemeral containers, and live debugging techniques to minimize disruption.

Intro

Upgrading and migrating running pods in Kubernetes is a delicate operation that requires careful planning to avoid downtime and service disruption. Debugging a live pod often involves attaching a debug container or using kubectl exec to inspect the environment. This guide provides practical examples for upgrading and migrating a running pod, including how to use kubectl debug to create a debugging session without disrupting the production container. We'll cover version inventory, safe configuration, verification, failure modes, recovery, and an operations checklist. By following these steps, you can minimize risk and ensure a smooth transition.

Version and Environment Inventory

Before making any changes, establish a clear picture of your current environment. This includes Kubernetes version, pod configuration, and the application's dependencies. Knowing these details helps you plan the upgrade and migration path.

Start by checking the Kubernetes server version:

kubectl version --short

Expected output (example):

Client Version: v1.24.0
Server Version: v1.24.0

Next, list the current pods and their images. Suppose you have a pod named myapp-pod in namespace default:

kubectl get pod myapp-pod -o yaml

This will output the full pod spec. Note the image field under containers. For example:

containers:
- name: app
  image: myapp:1.0

Record the image version. Also check the pod's labels and selectors if it is managed by a Deployment or StatefulSet:

kubectl get deployment myapp-deployment -o yaml

If you need to debug a running pod, you can create a debug container using kubectl debug. This is useful for inspecting the filesystem or network without altering the original container. Example:

kubectl debug myapp-pod -it --image=busybox --target=app

This command attaches a new container based on busybox to the same pod, sharing the process namespace if needed. You can then run commands inside the debug container to examine the environment. For example, to inspect the filesystem of the target container:

# Inside the debug container
ls /proc/1/root

Or to check network connectivity:

# Inside the debug container
wget -O- http://localhost:8080/health

Make sure you have the necessary RBAC permissions to perform these operations. The minimal permissions typically include get, list, create, and delete on pods and deployments. A minimal Role for debugging might look like:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: pod-debugger
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "create", "delete"]
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["get", "list", "update", "patch"]

Safe Configuration Path

When upgrading or migrating a pod, use a controlled, scoped approach. Avoid changing multiple parameters at once. Instead, make incremental changes and test each one.

For a controlled upgrade, consider using a new image tag. For example, to upgrade from myapp:1.0 to myapp:2.0, update the deployment:

kubectl set image deployment/myapp-deployment app=myapp:2.0

This will trigger a rolling update. To monitor the rollout:

kubectl rollout status deployment/myapp-deployment

Expected output:

Waiting for deployment "myapp-deployment" rollout to finish: 1 out of 3 new replicas have been updated...
deployment "myapp-deployment" successfully rolled out

If you need to migrate a pod from one node to another, you can use a node selector or taints/tolerations. Suppose you want to move the pod to a node with label disktype=ssd. Patch the deployment with a node selector:

kubectl patch deployment myapp-deployment -p '{"spec":{"template":{"spec":{"nodeSelector":{"disktype":"ssd"}}}}}'

This will cause the deployment to create new pods on nodes that match the selector and terminate old pods. To ensure the migration is smooth, you might want to drain the old node first:

kubectl drain <old-node> --ignore-daemonsets --delete-emptydir-data

This cordons the node and evicts pods. Then the scheduler places pods on other nodes. After draining, you can uncordon the node if needed:

kubectl uncordon <old-node>

For debugging a running pod during migration, you can use kubectl debug with a copy of the pod to test changes without affecting the live pod. Example:

kubectl debug myapp-pod --copy-to=myapp-debug --image=myapp:2.0 -- sleep 1h

This creates a debug pod myapp-debug with the new image but does not serve traffic. You can then inspect the debug pod to validate the new configuration. To attach to the debug pod:

kubectl exec -it myapp-debug -- sh

Inside, you can run application-specific checks, such as verifying database connectivity or checking configuration files.

Verification and Diagnostics

After applying changes, verify that the pod is running correctly. Check the pod status:

kubectl get pods

Look for the pod in Running state with the correct number of restarts. For example:

NAME                                READY   STATUS    RESTARTS   AGE
myapp-deployment-7f8c9d5b6-abcde   1/1     Running   0          2m

Inspect logs to ensure there are no errors:

kubectl logs myapp-deployment-7f8c9d5b6-abcde

If the application exposes a health endpoint, test it using kubectl exec:

kubectl exec myapp-deployment-7f8c9d5b6-abcde -- curl -f http://localhost:8080/health

Expected output on success (example):

{"status":"ok"}

For debugging, you can attach to the running pod with kubectl debug. This can help diagnose issues that are not apparent from logs alone. Example:

kubectl debug myapp-deployment-7f8c9d5b6-abcde -it --image=nicolaka/netshoot --target=app

Inside the debug container, you can run network diagnostics or inspect environment variables. For instance, to check DNS resolution:

nslookup kubernetes.default

Or to view environment variables of the target process:

cat /proc/1/environ | tr '\0' '\n'

Additionally, check events for the pod to see any warnings:

kubectl describe pod myapp-deployment-7f8c9d5b6-abcde

Look under Events for messages like FailedScheduling or Liveness probe failed. These indicate problems that need attention.

Failure Modes and Recovery

Even with careful planning, upgrades and migrations can fail. Common failure modes include:

• Image pull errors: The new image may not exist or credentials are missing. Check with kubectl describe pod and look for ErrImagePull or ImagePullBackOff.
• CrashLoopBackOff: The new version crashes on startup. Check logs to identify the cause.
• Scheduling failures: If node selectors or resource requests cannot be satisfied, pods remain Pending.
• Service disruption: Rolling update may be too fast, causing temporary loss of capacity.

To recover, you can rollback a Deployment to a previous revision. First, check rollout history:

kubectl rollout history deployment/myapp-deployment

Expected output:

deployment.apps/myapp-deployment
REVISION  CHANGE-CAUSE
1         <none>
2         <none>

Then rollback to the previous revision:

kubectl rollout undo deployment/myapp-deployment

This reverts to revision 1. Monitor the rollback with kubectl rollout status.

If a pod is stuck in Terminating state during node drain, you can force delete it:

kubectl delete pod myapp-pod --grace-period=0 --force

But be cautious with force deletion as it may leave resources.

For debugging a failed pod, you can use kubectl debug to create a copy of the pod with a different command or image to reproduce the issue. For example, to start a debug pod that overrides the entrypoint with a shell:

kubectl debug myapp-pod --copy-to=myapp-debug --image=myapp:2.0 -- /bin/sh

Always have a backup of your Deployment or StatefulSet YAML so you can restore the previous configuration quickly. You can export the current configuration before changes:

kubectl get deployment myapp-deployment -o yaml > myapp-deployment-backup.yaml

Operations Checklist

Use this checklist before, during, and after the upgrade/migration:

After completing the steps, document the changes and update any relevant runbooks. For example, note the new image version, the date of change, and any issues encountered. This helps with future audits and troubleshooting.

Conclusion

Upgrading and migrating running pods in Kubernetes can be done safely with proper planning and the right tools. By using kubectl debug, you can inspect and test changes without affecting production traffic. Always start with a version inventory, make scoped configuration changes, verify thoroughly, and have a rollback plan. The operations checklist provided will help you repeat the process consistently. Remember to monitor the rollout and be ready to undo if anything goes wrong. With these practices, you can minimize downtime and keep your applications running smoothly.