Kubectl Production Operations Checklist with Practical Examples

Intro

Kubectl is the primary command-line tool for interacting with Kubernetes clusters in production. Whether you are a developer deploying applications, a DevOps consultant managing client infrastructure, or part of a technical startup team, having a structured set of operational procedures for kubectl is essential. A production operations checklist helps ensure consistency, reduce human error, and speed up troubleshooting.

This article provides a practical checklist for kubectl production operations, with concrete examples, commands, and expected outputs. We will cover version and environment inventory, safe configuration practices, verification and diagnostics, failure modes and recovery, common pitfalls, and a repeatable operations checklist. By the end, you will have a solid foundation to manage your Kubernetes clusters confidently and safely.

The value of a checklist is twofold: it encodes best practices so that anyone on the team can perform tasks correctly, and it provides a baseline for automation and auditing. As your environment grows, the checklist can be extended and adapted to your specific needs.

Version and Environment Inventory

Before making any changes, you must understand your environment. This section covers how to gather version information, cluster topology, and prerequisites for safe kubectl operations.

Check kubectl version and cluster version

First, verify that your local kubectl version is compatible with the cluster's Kubernetes version. The general rule is that kubectl should be within one minor version of the cluster. Use the following commands:

kubectl version --short

Expected output (example):

Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.27.3

If the output shows a large discrepancy between client and server versions, consider updating kubectl to match the cluster's minor version. For example, if the server is at 1.27, using kubectl 1.28 is acceptable, but kubectl 1.30 might not be fully tested.

List nodes and their status

Understanding your cluster nodes is critical. Use:

kubectl get nodes -o wide

Expected output (example):

NAME     STATUS   ROLES           AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION       CONTAINER-RUNTIME
node-1   Ready    control-plane   10d   v1.27.3   192.168.1.10   <none>        Ubuntu 22.04.2 LTS   5.15.0-76-generic   containerd://1.6.21
node-2   Ready    <none>          10d   v1.27.3   192.168.1.11   <none>        Ubuntu 22.04.2 LTS   5.15.0-76-generic   containerd://1.6.21
node-3   Ready    <none>          10d   v1.27.3   192.168.1.12   <none>        Ubuntu 22.04.2 LTS   5.15.0-76-generic   containerd://1.6.21

Check for any nodes that are NotReady or have other abnormal statuses. If a node is NotReady, investigate with kubectl describe node <node-name>.

Identify namespaces and workloads

List all namespaces and the resources within them to understand what is running:

kubectl get namespaces
kubectl get all --all-namespaces

The second command shows pods, services, deployments, and more across all namespaces. For large clusters, you may want to focus on specific namespaces. For example, to see all resources in the production namespace:

kubectl get all -n production

Check cluster events

Recent cluster events can reveal issues. Use:

kubectl get events --all-namespaces --sort-by='.lastTimestamp'

Expected output snippet:

NAMESPACE   LAST SEEN   TYPE     REASON    OBJECT                          MESSAGE
default     2m          Normal   Scheduled pod/myapp-7c9f8b6d-abcde       Successfully assigned default/myapp-7c9f8b6d-abcde to node-2
default     2m          Normal   Pulling   pod/myapp-7c9f8b6d-abcde       Pulling image "nginx:1.25"
default     1m          Normal   Pulled    pod/myapp-7c9f8b6d-abcde       Successfully pulled image "nginx:1.25"
default     1m          Normal   Created   pod/myapp-7c9f8b6d-abcde       Created container nginx
default     1m          Normal   Started   pod/myapp-7c9f8b6d-abcde       Started container nginx

Look for Warning type events, which may indicate problems. You can filter for warnings using:

kubectl events --all-namespaces --types=Warning

Prerequisites checklist

Before performing any operational task, ensure the following:

• Access: You have the necessary RBAC permissions to perform the intended actions. Test with a dry-run or kubectl auth can-i.
• Context: You are using the correct kubeconfig context. Use kubectl config current-context to verify.
• Backup: For any destructive operation, ensure you have a backup or a rollback plan (see Failure Modes and Recovery).
• Maintenance window: For changes that may cause disruption, schedule during a maintenance window and inform stakeholders.

Example permission check:

kubectl auth can-i create deployments.apps --namespace production

Expected output:

yes

If the output is no, you may need to request additional permissions or use a different account.

Safe Configuration Path

Making configuration changes in production requires care. This section outlines a safe approach: use dry-run, apply changes incrementally, and leverage version control for manifests.

Use dry-run to validate changes

Always test changes with --dry-run=client or --dry-run=server before applying. For example, to test creating a deployment:

kubectl create deployment test-nginx --image=nginx:1.25 --dry-run=client -o yaml

Expected output:

apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: test-nginx
  name: test-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test-nginx
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: test-nginx
    spec:
      containers:
      - image: nginx:1.25
        name: nginx
        resources: {}
status: {}

This shows the generated YAML without actually creating the resource. For changes to existing resources, use kubectl apply --dry-run=client -f manifest.yaml to see what would change.

Use apply with diff

Before applying, use kubectl diff to see the exact changes that will be made:

kubectl diff -f deployment.yaml

Expected output example:

--- LIVE
+++ MERGED
@@ -6,6 +6,6 @@
     app: myapp
     name: myapp
 spec:
-  replicas: 3
+  replicas: 4

Review the diff to ensure only intended changes are made. If the diff is empty, no changes will be applied.

Use namespaces and labels for organization

Apply changes within the correct namespace. If not specified, kubectl defaults to the default namespace. Always specify the namespace explicitly to avoid mistakes:

kubectl apply -f deployment.yaml -n production

Use labels to organize resources and enable targeted operations. For example, to update only pods with a specific label:

kubectl set image deployment/myapp myapp=nginx:1.25 -n production --record

Expected output:

deployment.apps/myapp image updated

Use version control for manifests

Store all Kubernetes manifests in a version control system (e.g., Git). This provides an audit trail, enables code review, and allows easy rollback to a previous known-good state. Before applying, ensure the manifests are up to date and reviewed by a peer. Use a pull request workflow, and consider using a tool like kubectl apply --prune with a carefully managed set of labels to clean up removed resources.

Example: Update deployment image with canary

For critical updates, use a canary deployment strategy. Instead of changing the main deployment directly, create a new deployment with the new image and a small replica count, and use a service selector to split traffic. Here is a simplified example:

• Create a canary deployment:

kubectl create deployment myapp-canary --image=myapp:2.0 -n production --dry-run=client -o yaml > myapp-canary.yaml

• Edit the YAML to add a different label, e.g., track: canary, and set replicas to 1. Here is an example manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-canary
  namespace: production
  labels:
    app: myapp
    track: canary
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myapp
      track: canary
  template:
    metadata:
      labels:
        app: myapp
        track: canary
    spec:
      containers:
      - name: myapp
        image: myapp:2.0
        ports:
        - containerPort: 80

• Apply it:

kubectl apply -f myapp-canary.yaml -n production

Expected output:

deployment.apps/myapp-canary created

• If the canary works well, you can gradually increase its replicas or promote it to the main deployment by updating the service selector. For example, to route 10% of traffic to the canary, adjust the service selector to include both versions or use a service mesh like Istio for finer control.

Verification and Diagnostics

After making changes, verify that everything works as expected and diagnose issues effectively. This section provides commands for checking pod health, logs, and resource usage.

Check pod status and readiness

Use kubectl get pods to see the status of pods in a namespace:

kubectl get pods -n production

Expected output:

NAME                               READY   STATUS    RESTARTS   AGE
myapp-7c9f8b6d-abcde               1/1     Running   0          1m
myapp-7c9f8b6d-xyz12               1/1     Running   0          1m
myapp-canary-5d7f8b-1234           1/1     Running   0          30s

Look for pods that are not in Running state or have high restart counts. High restarts often indicate crash loops. Check with:

kubectl get pods -n production --field-selector=status.phase!=Running

This shows all pods not in a Running phase.

Describe pods for detailed information

Use kubectl describe pod to get detailed information about a pod, including events:

kubectl describe pod myapp-7c9f8b6d-abcde -n production

Expected output snippet:

Name:         myapp-7c9f8b6d-abcde
Namespace:    production
...
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  2m    default-scheduler  Successfully assigned production/myapp-7c9f8b6d-abcde to node-2
  Normal  Pulled     2m    kubelet            Container image "myapp:1.0" already present on machine
  Normal  Created    2m    kubelet            Created container myapp
  Normal  Started    2m    kubelet            Started container myapp

Check logs

To view logs from a pod:

kubectl logs myapp-7c9f8b6d-abcde -n production

For pods with multiple containers, specify the container name with -c:

kubectl logs myapp-7c9f8b6d-abcde -n production -c myapp

For logs from all pods matching a label, you can use a tool like stern or a simple loop:

kubectl logs -l app=myapp -n production --tail=20

Expected output example:

192.168.1.25 - - [10/Oct/2024:12:00:00 +0000] "GET /healthz HTTP/1.1" 200 2 "-" "kube-probe/1.27"
192.168.1.26 - - [10/Oct/2024:12:00:00 +0000] "GET /api/data HTTP/1.1" 200 1234 "-" "Mozilla/5.0"

Monitor resource usage

Use kubectl top to see resource usage for nodes and pods (requires metrics-server):

kubectl top nodes
kubectl top pods -n production

Expected output example for nodes:

NAME     CPU(cores)   MEMORY(bytes)
node-1   120m         1.5Gi
node-2   90m          1.2Gi
node-3   80m          1.1Gi

For pods:

NAME                            CPU(cores)   MEMORY(bytes)
myapp-7c9f8b6d-abcde            10m          50Mi
myapp-7c9f8b6d-xyz12            12m          55Mi

Run diagnostic commands inside a container

If you need to test connectivity or run other checks inside a container, use kubectl exec:

kubectl exec myapp-7c9f8b6d-abcde -n production -- curl -s http://localhost:8080/healthz

Expected output:

ok

Perform a connectivity test from outside

Use kubectl port-forward to access a service locally:

kubectl port-forward svc/myapp-service 8080:80 -n production

Expected output:

Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80

Then open a browser or use curl to test:

curl http://localhost:8080

Check service endpoints

Ensure that services have endpoints and they are correct:

kubectl get endpoints -n production

Expected output:

NAME           ENDPOINTS                                         AGE
myapp-service  192.168.1.11:8080,192.168.1.12:8080               2d

If the ENDPOINTS column is empty, the service selector may not match any pods.

Failure Modes and Recovery

Even with careful planning, failures can occur. This section covers common failure modes and how to recover using kubectl.

Common failure modes

• Image pull errors: Pods fail to start because the image is missing or the registry is unreachable.
• Configuration errors: Invalid YAML, missing ConfigMaps or Secrets, or incorrect environment variables.
• Resource exhaustion: Insufficient CPU or memory leading to pod evictions or OOM kills.
• Network issues: Service discovery failures, DNS problems, or network policy misconfigurations.
• Node failures: Node becomes NotReady due to hardware or OS issues, causing pods to be rescheduled.

Diagnose image pull errors

If a pod is in ImagePullBackOff or ErrImagePull, check the events:

kubectl describe pod myapp-7c9f8b6d-xyz12 -n production

Expected output snippet:

Events:
  Type     Reason     Age   From               Message
  ----     ------     ----  ----               -------
  Normal   Scheduled  1m    default-scheduler  Successfully assigned production/myapp-7c9f8b6d-xyz12 to node-2
  Warning  Failed     1m    kubelet            Failed to pull image "myapp:2.0": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/library/myapp:2.0": failed to resolve reference "docker.io/library/myapp:2.0": not found
  Warning  Failed     1m    kubelet            Error: ErrImagePull
  Normal   BackOff    1m    kubelet            Back-off pulling image "myapp:2.0"

In this case, the image tag 2.0 does not exist. Fix by correcting the image tag in the deployment and applying the change:

kubectl set image deployment/myapp myapp=myapp:1.0 -n production

Rollback a deployment

If a new deployment version causes issues, you can rollback to a previous revision. First, check the rollout history:

kubectl rollout history deployment/myapp -n production

Expected output:

deployment.apps/myapp
REVISION  CHANGE-CAUSE
1         <none>
2         <none>

To rollback to the previous revision:

kubectl rollout undo deployment/myapp -n production

Expected output:

deployment.apps/myapp rolled back

To rollback to a specific revision:

kubectl rollout undo deployment/myapp --to-revision=1 -n production

Restart a deployment

Sometimes restarting pods can resolve temporary issues. Use the following command:

kubectl rollout restart deployment/myapp -n production

Expected output:

deployment.apps/myapp restarted

Delete and recreate a stuck pod

If a pod is stuck in Terminating or CrashLoopBackOff, you can force delete it:

kubectl delete pod myapp-7c9f8b6d-xyz12 -n production --grace-period=0 --force

Expected output:

warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
pod "myapp-7c9f8b6d-xyz12" force deleted

Recover from resource exhaustion

If pods are being evicted or killed due to lack of resources, check node resource pressure:

kubectl describe node node-1 | grep -A5 "Conditions:"

Expected output:

Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Thu, 10 Oct 2024 12:00:00 +0000   Wed, 09 Oct 2024 12:00:00 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Thu, 10 Oct 2024 12:00:00 +0000   Wed, 09 Oct 2024 12:00:00 +0000   KubeletHasNoDiskPressure     kubelet has sufficient disk space available
  PIDPressure      False   Thu, 10 Oct 2024 12:00:00 +0000   Wed, 09 Oct 2024 12:00:00 +0000   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Thu, 10 Oct 2024 12:00:00 +0000   Wed, 09 Oct 2024 12:00:00 +0000   KubeletReady                 kubelet is posting ready status

If there is pressure, you may need to scale down or add nodes. To check pod resource usage and limits:

kubectl top pods -n production --containers

And review resource requests and limits in pod specs.

Backup and restore etcd

For disaster recovery, you should regularly back up the etcd database if you manage the control plane. This is beyond kubectl scope but essential. For managed clusters (e.g., EKS, GKE), use the cloud provider's snapshot features. For self-managed, use etcdctl snapshot.

Example etcd snapshot command (on a control plane node):

ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-$(date +%Y%m%d).db --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key

To restore, use etcdctl snapshot restore and follow your cluster restore procedure.

Common Pitfalls and How to Avoid Them

In day-to-day kubectl operations, certain mistakes recur. Recognizing these pitfalls can save hours of debugging.

Operating on the wrong cluster or namespace

Why it happens: Multiple kubeconfig contexts with similar names, or relying on the default namespace.

How to avoid: Always explicitly specify namespace and verify context before applying. Use kubectl config current-context and kubectl config get-contexts. Consider using tools like kubectx and kubens to switch contexts and namespaces quickly, and configure your shell prompt to show the current context.

Forgetting to dry-run or diff

Why it happens: Time pressure or overconfidence in manifests.

How to avoid: Make --dry-run=client and kubectl diff mandatory steps in your change process. You can even automate these checks in CI/CD pipelines to reject unauthorized direct applies.

Not using labels consistently

Why it happens: Labels are added ad hoc without a naming convention.

How to avoid: Define a label schema for your organization (e.g., app, tier, environment, owner) and enforce it with policies. Tools like OPA Gatekeeper or Kyverno can validate labels at admission time.

Ignoring events and logs until after a failure

Why it happens: Lack of proactive monitoring or alerting.

How to avoid: Set up monitoring and alerting for cluster events and pod logs. Use kubectl get events --watch during changes, and integrate with monitoring systems like Prometheus and Alertmanager.

Applying changes without a rollback plan

Why it happens: Assuming a change is low-risk.

How to avoid: For any deployment change, know the rollback command and test it in staging. Use kubectl rollout history to understand available revisions. For configuration changes, keep a copy of the previous manifest in Git.

Forgetting to clean up resources

Why it happens: Old resources accumulate and can cause confusion or resource leaks.

How to avoid: Regularly review unused resources with tools like kubectl get all --all-namespaces --show-labels, and delete dangling ReplicaSets, completed Jobs, or unused ConfigMaps. Consider using kubectl apply --prune with a label selector to remove resources not in the manifest, but use it carefully.

Operations Checklist

This section provides a repeatable checklist for common kubectl operations. Use it as a reference before, during, and after performing tasks. Assign an owner for each major item to ensure accountability. For example, designate a Kubernetes Platform Owner (a specific individual, such as "Priya Shah, Engineering Lead") who reviews cluster health weekly and owns the change management process.

Daily checks

• [ ] Verify cluster health: kubectl get nodes shows all nodes Ready. Owner: on-call engineer, reviewed every morning.
• [ ] Check for abnormal events: kubectl get events --all-namespaces --sort-by='.lastTimestamp' | grep -i warning. Owner: on-call engineer.
• [ ] Monitor resource usage: kubectl top nodes and kubectl top pods in critical namespaces. Owner: on-call engineer.
• [ ] Review pod restarts: kubectl get pods --all-namespaces | awk '$4>0' to find pods with restarts. Owner: on-call engineer.

Before applying changes

• [ ] Confirm current context: kubectl config current-context. Owner: developer making the change.
• [ ] Dry-run the change: kubectl apply --dry-run=client -f manifest.yaml. Owner: developer.
• [ ] Diff the change: kubectl diff -f manifest.yaml. Owner: developer.
• [ ] Ensure backup exists if change is destructive. Owner: developer and platform owner.
• [ ] Schedule a maintenance window if needed. Owner: platform owner, approved at least 24 hours in advance.

During change

• [ ] Apply the change: kubectl apply -f manifest.yaml -n <namespace>. Owner: developer.
• [ ] Monitor rollout status for deployments: kubectl rollout status deployment/myapp -n <namespace>. Owner: developer.
• [ ] Watch for new events: kubectl get events -n <namespace> --watch. Owner: developer.

After change

• [ ] Verify pod health: kubectl get pods -n <namespace>. Owner: developer.
• [ ] Check logs for errors: kubectl logs -l app=myapp -n <namespace> --tail=50. Owner: developer.
• [ ] Validate service endpoints: kubectl get endpoints -n <namespace>. Owner: developer.
• [ ] Run smoke tests (e.g., curl the service via port-forward). Owner: QA engineer or developer.

Periodic maintenance

• [ ] Update images to latest stable versions after testing. Owner: platform owner, monthly.
• [ ] Clean up unused resources (e.g., dangling deployments, old ReplicaSets). Owner: platform owner, monthly.
• [ ] Review and rotate secrets and certificates. Owner: security engineer, quarterly or as needed.
• [ ] Perform etcd backups if self-managed. Owner: platform owner, daily automated, verified weekly.
• [ ] Test disaster recovery procedures. Owner: platform owner, quarterly.

Useful kubectl aliases

To save time, consider adding these aliases to your shell profile:

alias k='kubectl'
alias kgp='kubectl get pods'
alias kgn='kubectl get nodes'
alias kgd='kubectl get deployments'
alias kd='kubectl describe'
alias kl='kubectl logs'

Conclusion

In this article, we have covered a comprehensive kubectl production operations checklist with practical examples. We started with gathering version and environment information, then discussed safe configuration paths using dry-run, diff, and canary deployments. We provided verification and diagnostic commands to ensure changes are working correctly, and we explored common failure modes and recovery steps. We also highlighted common pitfalls and how to avoid them, and presented a repeatable operations checklist for daily and periodic tasks.

By following this checklist, you can reduce the risk of production incidents and improve your team's operational maturity. Remember to always test changes in a staging environment first, use version control for manifests, and document any deviations from the checklist. As you gain more experience, you can adapt the checklist to your specific use cases and automate repetitive tasks.

Start with a narrow, measurable pilot, as suggested in the research: pick a small change, apply the safe configuration path, and verify with diagnostics before rolling out more broadly. This approach builds confidence and helps you refine your processes.

Happy kubectl-ing!