## Intro Kubectl is the primary command-line tool for interacting with Kubernetes clusters. While many engineers are familiar with basic commands like `kubectl get pods` or `kubectl apply -f manifest.yaml`, advanced usage requires a deeper understanding of how kubectl works, how to diagnose issues safely, and how to recover from failures without causing further disruption. This article is written for developers, DevOps consultants, and technical startup teams who need to move beyond the basics. It connects kubectl internals, architecture, and deep-dive concepts to practical commands, expected output, failure signals, and recovery decisions. The focus is on operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify results, and document recovery paths. By the end of this guide, you will be able to: - Inventory your Kubernetes environment and inspect cluster state with precision. - Safely apply configuration changes and validate them before full rollout. - Diagnose common pod, deployment, and networking failures. - Recover from failed states using rollback and inspection techniques. - Follow a repeatable operations checklist for day-to-day reliability. ## Version and Environment Inventory Before making any changes to a Kubernetes cluster, you must know exactly what you are working with. A version and environment inventory identifies the relevant component, its supported version range, prerequisites, a read-only observation method, the smallest justified change, and the command or signal that verifies the outcome. Start by checking your kubectl version and cluster version. Mismatched versions can lead to unexpected API behavior and unsupported features. Run: ```bash kubectl version ``` Expected output will show client and server versions, for example: ``` Client Version: v1.29.1 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.28.3 ``` If the server version is not shown, kubectl cannot reach the cluster, or your kubeconfig lacks the necessary context. Verify your current context with: ```bash kubectl config current-context ``` This should return the name of the context you intend to use, such as `prod-cluster-admin`. Using the wrong context is a common cause of accidental changes. Always confirm the context before running any mutating command. Next, inventory the node and pod topology to understand where your workloads run. For a read-only observation, use: ```bash kubectl get nodes -o wide kubectl get pods -A -o wide ``` The output shows node names, roles, versions, internal and external IPs, and for pods, their namespace, status, restart count, age, and assigned node. For example: ``` NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE node-1 Ready control-plane 12d v1.28.3 10.0.0.1 Ubuntu 22.04 node-2 Ready 12d v1.28.3 10.0.0.2 Ubuntu 22.04 ``` This inventory helps you identify anomalies such as nodes in `NotReady` state, pods stuck in `Pending` or `CrashLoopBackOff`, and resource distribution issues. For a deeper dive into a specific pod, use `describe` to view events, conditions, and configuration details: ```bash kubectl describe pod -n ``` The output includes recent events that often point directly to the problem, such as insufficient CPU, image pull failures, or scheduling constraints. For example, an event might read: ``` Warning FailedScheduling 2m (x5 over 10m) default-scheduler 0/3 nodes are available: 3 Insufficient cpu. ``` This tells you exactly why the pod is not running. Remember the principle of separating observation from intervention. Capture the current state and timestamps before making any changes. Protect credentials and private material; never echo secrets in logs or command history. Use environment variables or secret references in manifests rather than plaintext values. ## Safe Configuration Path Applying configuration changes to a production cluster is one of the most dangerous operations. The key is to follow a safe configuration path that minimizes risk and ensures recoverability. This means applying changes in a controlled manner, validating them before full rollout, and having a rollback plan. Before applying any manifest, perform a dry run to see what would change without actually modifying the cluster: ```bash kubectl apply -f manifest.yaml --dry-run=client ``` This client-side dry run validates the syntax and shows the resources that would be created or updated. For server-side validation, use `--dry-run=server`. This is crucial when working with admission controllers or schema validation that may reject certain fields. If you are modifying an existing deployment, use the `diff` command to see a unified diff between the current state and the proposed manifest: ```bash kubectl diff -f manifest.yaml ``` The output will show additions, deletions, and modifications in a familiar diff format. For example: ```diff --- /tmp/LIVE-123/deployment.yaml +++ /tmp/MERGED-456/deployment.yaml @@ -12,7 +12,7 @@ image: myapp:1.0 - image: myapp:1.0 + image: myapp:1.1 ``` This allows you to review changes before they are applied. When you are ready to apply, rely on the deployment's revision history. For example, to update a deployment's image: ```bash kubectl set image deployment/myapp myapp=myapp:1.1 ``` Then check the rollout status: ```bash kubectl rollout status deployment/myapp ``` Expected output if successful: ``` deployment "myapp" successfully rolled out ``` If the rollout fails, you can inspect the revision history: ```bash kubectl rollout history deployment/myapp ``` This lists revisions with change causes (if recorded). To roll back to a previous revision: ```bash kubectl rollout undo deployment/myapp --to-revision=2 ``` Always verify that the rollback succeeded with `kubectl rollout status`. For small local tests, keep the blast radius minimal. Use a dedicated namespace for testing and clean up afterwards. Apply one manifest at a time and inspect the generated resources. Use `kubectl port-forward` to test connectivity locally before exposing services via a load balancer or ingress controller. For example: ```bash kubectl port-forward svc/my-service 8080:80 ``` This forwards local port 8080 to the service's port 80, allowing you to test the application without opening it to the internet. ## Verification and Diagnostics Verification and diagnostics are at the heart of advanced kubectl usage. When something goes wrong, your ability to quickly identify and resolve the issue depends on systematic inspection and interpretation of output. Start with the basics: check pod status and logs. For a pod in a crash loop, get its current status: ```bash kubectl get pods ``` If the pod is in `CrashLoopBackOff` or `Error`, the next step is to view its logs. For the current instance: ```bash kubectl logs ``` If the pod has crashed and restarted, you may need logs from the previous instance to see the error that caused the crash: ```bash kubectl logs --previous ``` For example, a Python application might log a traceback that reveals a missing dependency or configuration error. The logs will show the exact error, such as: ``` ModuleNotFoundError: No module named 'requests' ``` This tells you the container image is missing a required package. Use `describe` to get events and conditions that may explain the pod's state. Look for events like `FailedScheduling`, `FailedMount`, or `BackOff`. The output also includes readiness and liveness probe results, which can indicate why a pod is not receiving traffic. For networking issues, verify service endpoints and connectivity. Check that a service has endpoints: ```bash kubectl get endpoints -n ``` The output should list pod IPs and ports. If the list is empty, the service selector does not match any pods. Verify the selector in the service definition and the pod labels. Test connectivity from within the cluster using a temporary debug pod: ```bash kubectl run -it --rm debug --image=busybox -- sh ``` Inside the debug pod, you can use `wget` or `nc` to test internal service DNS and ports. For example: ```bash wget -O- http://my-service.namespace.svc.cluster.local:8080/health ``` This allows you to test the service without leaving the cluster. For deeper diagnostics, use `kubectl exec` to run commands inside a running container. For example, to check environment variables or files: ```bash kubectl exec -it -- env kubectl exec -it -- ls /app ``` This can reveal misconfigurations, missing files, or incorrect paths. Always compare expected behavior with actual behavior. For example, if a deployment is supposed to have 3 replicas but only 2 are running, use `kubectl describe deployment` to see why the third replica is not created. Look for events about quota limits or insufficient resources. ## Failure Modes and Recovery Kubernetes workloads fail in predictable ways, and advanced kubectl users should be familiar with common failure modes and how to recover from them. This section outlines several scenarios with concrete recovery steps. ### CrashLoopBackOff A pod repeatedly starts and crashes. The restart count in `kubectl get pods` increments, and the status becomes `CrashLoopBackOff`. **Why it happens:** The application exits with a non-zero code, often due to misconfiguration, missing dependencies, or failed startup probes. **Recovery steps:** 1. Check logs from the previous instance: `kubectl logs --previous`. 2. Describe the pod for events: `kubectl describe pod `. 3. If the error is due to a bad configuration, fix the ConfigMap or Secret and update the pod. 4. If the image is wrong, update the deployment with the correct image tag. ### ImagePullBackOff A pod cannot pull its container image. The status is `ImagePullBackOff` or `ErrImagePull`. **Why it happens:** Incorrect image name or tag, missing registry credentials, or network issues. **Recovery steps:** 1. Check the image name in the pod spec: `kubectl get pod -o jsonpath='{.spec.containers[*].image}'`. 2. Verify that the image exists in the registry, and that you have permission to pull it. 3. If using a private registry, ensure the imagePullSecret is correctly configured. 4. Check events with `kubectl describe pod` for detailed error messages. ### Pending Pods A pod is stuck in `Pending` because it cannot be scheduled. **Why it happens:** Insufficient resources, node selectors or affinities that cannot be met, or taints that repel the pod. **Recovery steps:** 1. Run `kubectl describe pod ` and look at the Events section. It will tell you why scheduling failed, e.g., `0/3 nodes are available: 3 Insufficient memory`. 2. If resources are the issue, adjust the pod's resource requests or add more nodes. 3. If selectors or taints are the problem, modify the pod spec or node taints accordingly. ### Failed Rollout A deployment update causes the new pods to fail or never become ready, so the rollout stalls. **Why it happens:** New image has bugs, readiness probe fails, or resource limits are too low. **Recovery steps:** 1. Check rollout status: `kubectl rollout status deployment/`. 2. Get the rollout history: `kubectl rollout history deployment/`. 3. Inspect the failing pods' logs and events. 4. If the new version is broken, roll back immediately: `kubectl rollout undo deployment/`. 5. Verify that the old pods are back and the rollout is complete. ### Service Connection Refused Applications cannot connect to a service, or the service does not route traffic correctly. **Why it happens:** Service selector does not match pods, targetPort is wrong, or network policies block traffic. **Recovery steps:** 1. Check that the service has endpoints: `kubectl get endpoints `. If empty, the selector is wrong. 2. Verify the targetPort in the service definition matches the container port. 3. Test from a debug pod as described earlier. 4. Check network policies that might be blocking ingress or egress. In all recovery scenarios, document the incident and the steps taken. This helps build a knowledge base for future issues and informs improvements to the operations checklist. ## Operations Checklist A consistent operations checklist helps prevent mistakes and ensures that routine tasks are performed safely and completely. Here is a practical checklist for common kubectl operations, with specific commands and verification steps. ### Before Any Change - [ ] Confirm the current context: `kubectl config current-context` - [ ] Verify cluster version compatibility: `kubectl version` - [ ] Take a note of current state: `kubectl get all -n -o wide > state-before.txt` - [ ] Ensure you have a backup or can revert the change. ### Applying Changes - [ ] Use dry-run: `kubectl apply -f manifest.yaml --dry-run=client` - [ ] Review diff: `kubectl diff -f manifest.yaml` - [ ] Apply the change: `kubectl apply -f manifest.yaml` - [ ] Watch the rollout: `kubectl rollout status deployment/` - [ ] Check pod health: `kubectl get pods -n ` - [ ] If using a Job, check completion: `kubectl get job ` ### Verifying After Change - [ ] Confirm pods are running: `kubectl get pods` - [ ] Check logs for errors: `kubectl logs ` - [ ] Test service endpoints: `kubectl get endpoints ` - [ ] Run application-level smoke tests. ### Rollback Plan - [ ] Identify the previous revision: `kubectl rollout history deployment/` - [ ] Determine the command to rollback: `kubectl rollout undo deployment/` - [ ] Document the expected time to rollback and who is responsible (e.g., the on-call engineer). For each major decision or change, assign a single accountable owner, not a group. For example, the deployment owner is [Priya Shah, Engineering Lead]. She is responsible for approving changes to production and for ensuring rollback procedures are tested. The decision to apply a change should be revisited monthly in the team's operations review, or immediately after a failed deployment. ## Common Pitfalls and How to Avoid Them Even experienced kubectl users fall into traps that can cause outages or wasted time. Here are some common pitfalls, why they happen, and how to avoid or recover from them. ### Operating in the Wrong Context **Why it happens:** You may have multiple clusters or namespaces, and your kubeconfig context is not the one you think. This can lead to applying changes to production instead of staging. **How to avoid:** Always run `kubectl config current-context` before any mutating command. Use tools like `kubectx` and `kubens` to switch contexts and namespaces explicitly. Consider using different shell prompts or terminal color schemes for different environments. **How to recover:** If you accidentally changed the wrong cluster, immediately roll back using `kubectl rollout undo` if possible, or restore from backups. Document the incident and review your context management practices. ### Applying Manifests Without Review **Why it happens:** You trust the manifest is correct and apply it directly, but it may have unintended changes due to a typo or a merge conflict. **How to avoid:** Always use `kubectl diff` to review changes before applying. Use `--dry-run=server` to catch server-side validation errors. For critical changes, ask a colleague to review the manifest. **How to recover:** If the change was bad and caused a rollout failure, use `kubectl rollout undo` or reapply the previous manifest. Keep manifests in version control so you can revert to a known good state. ### Deleting Resources with the Wrong Selector **Why it happens:** You use `kubectl delete -f` or `kubectl delete pod --all` without verifying the selector, accidentally deleting more than intended. **How to avoid:** Use labels and selectors carefully. Before deleting, list what would be deleted with `kubectl get -l `. For example, `kubectl get pods -l app=myapp` shows only pods with that label. **How to recover:** If you deleted pods that are part of a ReplicaSet, they may be recreated automatically. If not, you may need to apply the deployment manifest again to recreate them. For stateful sets, ensure data is not lost; use volume snapshots if available. ### Ignoring Pod Lifecycle and Probes **Why it happens:** You configure a deployment without proper readiness and liveness probes, so the service routes traffic to pods that are not ready, or the pod is killed prematurely. **How to avoid:** Define appropriate probes in your pod spec. For example, a readiness probe that checks an HTTP endpoint: ```yaml readinessProbe: httpGet: path: /healthz port: 8080 initialDelaySeconds: 5 periodSeconds: 10 ``` A liveness probe that checks a command: ```yaml livenessProbe: exec: command: - cat - /tmp/healthy initialDelaySeconds: 5 periodSeconds: 10 ``` **How to recover:** If a pod is stuck in `Running` but not `Ready`, check the readiness probe configuration. If a pod is being restarted frequently, check the liveness probe; it may be too aggressive. ### Neglecting Resource Limits **Why it happens:** You don't set resource requests and limits, causing pods to consume too much CPU or memory, starving other pods and leading to node instability. **How to avoid:** Always set resource requests and limits for your containers. For example: ```yaml resources: requests: memory: "64Mi" cpu: "250m" limits: memory: "128Mi" cpu: "500m" ``` Use resource quotas and limit ranges at the namespace level to enforce these. **How to recover:** If a pod is OOMKilled, check its memory limit. Adjust the limit or optimize the application's memory usage. Use `kubectl top pods` to see actual resource consumption. ## Conclusion Kubectl advanced concepts are essential for safe and efficient Kubernetes operations. By following the principles of observation before intervention, limiting blast radius, verifying results, and documenting recovery, you can avoid common pitfalls and maintain a reliable cluster. This article has provided practical commands and examples for version inventory, safe configuration, verification, diagnostics, failure recovery, and an operations checklist. These techniques are useful only when each recommendation is version-scoped, observable, and reversible where possible. Copying a command without checking prerequisites and expected output is not an operations procedure. As a next step, choose one low-risk verification from this guide, record the current state of your cluster, run the documented check, compare the result with the expected signal, and review dependencies such as namespaces, pods, and deployments. A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. By integrating these practices into your daily work, you will build a robust and resilient Kubernetes environment.