Kubelet Production Operations: A Practical Checklist with Real-World Examples

Intro

Kubelet is the node-level agent that turns container specifications into running workloads on every Kubernetes node. When kubelet fails, pods stop, node status degrades, and cluster capacity silently erodes. This article provides a production-focused operations checklist for kubelet, built around concrete commands, expected outputs, and recovery steps. It is written for developers, DevOps consultants, and technical startup teams who need to move from an observed symptom to a verified fix without guesswork.

The checklist is structured around operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached. Every section includes a read-only observation command, a minimum viable change, a verification signal, and a recovery path. We will cover version and environment inventory, safe configuration paths, diagnostics, failure modes, and a condensed operations checklist you can adapt to your own runbooks.

By the end, you will have a repeatable process for keeping kubelet healthy in production, whether you manage five nodes or five hundred.

Version and Environment Inventory

Before touching kubelet, you must know what you are running and where. Start with a read-only inventory of the kubelet version and the node's operating environment. This prevents mismatched flags, unsupported configurations, and accidental upgrades.

Read-only observation

On any node, run:

kubelet --version

Expected output on a recent Kubernetes release:

Kubernetes v1.29.2

Document this version for every node. If you are using a managed Kubernetes service like EKS, GKE, or AKS, the kubelet version may be managed for you. You can still query it via the node's status.nodeInfo.kubeletVersion field.

Cluster-wide version check

Use kubectl to list all nodes and their kubelet versions:

kubectl get nodes -o custom-columns='NAME:.metadata.name,KUBELET_VERSION:.status.nodeInfo.kubeletVersion'

Example output:

NAME           KUBELET_VERSION
node-1         v1.29.2
node-2         v1.28.5
node-3         v1.29.2

If versions differ significantly (more than one minor version skew from the control plane), schedule a rolling upgrade before making other kubelet changes. Version skew beyond the supported window causes API compatibility issues and is a common root cause of kubelet crashes.

Deployment topology

Identify how kubelet is deployed on the node: as a systemd service, a static binary managed by kubeadm, or a container in a kubelet-in-docker setup. The service name is usually kubelet. Check its status:

systemctl status kubelet --no-pager

Expected output when healthy:

● kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2024-05-14 10:22:31 UTC; 2 days ago
       Docs: https://kubernetes.io/docs/
   Main PID: 1337 (kubelet)
      Tasks: 15 (limit: 1152)
     Memory: 120.5M
     CGroup: /system.slice/kubelet.service

Note the Active: active (running) line and the uptime. If the service is not running or is in a crash loop, proceed to Failure Modes and Recovery.

Configuration file location

Kubelet reads configuration from a file, usually /var/lib/kubelet/config.yaml, and command-line flags from /var/lib/kubelet/kubeadm-flags.env (on kubeadm clusters). Verify the file exists and check its content:

ls -l /var/lib/kubelet/config.yaml
cat /var/lib/kubelet/config.yaml

A minimal production config.yaml might look like this:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
maxPods: 110

Never edit this file directly without a backup. Use the procedure in Safe Configuration Path.

Prerequisites

Before any change, confirm you have:

• Shell access to the node with sudo or root privileges.
• A recent backup of /var/lib/kubelet/ (including config.yaml, any kubelet certificates, and the kubeadm-flags.env file).
• The exact kubelet version you are running, so you can consult matching documentation.
• A maintenance window or cordon/drain strategy, because restarting kubelet can briefly disrupt node heartbeats.

Blast radius and recovery

All inventory commands are read-only and have negligible blast radius. If you accidentally modify a file, restore from backup and restart kubelet with:

sudo systemctl restart kubelet

Then verify with:

systemctl status kubelet --no-pager

Safe Configuration Path

Kubelet configuration changes are a leading cause of node instability. Use a controlled process: observe, modify one scoped item, validate, and document recovery.

Observe current effective configuration

Kubelet actively uses a combination of flags and the config file. You can dump the effective configuration via the kubelet's /configz endpoint (requires API server access and --enable-debugging-handlers=true). From a machine with kubectl and network access to the node's kubelet port (10250 by default), run:

kubectl proxy --port=8001 &
curl -s http://localhost:8001/api/v1/nodes/<NODE_NAME>/proxy/configz | jq .

Replace <NODE_NAME> with your actual node name, e.g., node-1. This returns the full effective configuration in JSON. Document the current values before any change, especially cgroupDriver, maxPods, evictionHard, and systemReserved.

Minimum viable change: example - adjust eviction thresholds

Suppose you want to set a custom eviction threshold to prevent node pressure. Edit /var/lib/kubelet/config.yaml (after backup) to include:

evictionHard:
  memory.available: "500Mi"
  nodefs.available: "10%"
  nodefs.inodesFree: "5%"

Then restart kubelet:

sudo systemctl restart kubelet

Verification

Check that kubelet started without errors and the new configuration is loaded:

journalctl -u kubelet -n 50 --no-pager | grep -i eviction

Expect to see the eviction settings in the startup log, or use the /configz endpoint again to confirm the values.

Rollback

If kubelet fails to start or the node becomes NotReady, revert the change:

sudo cp /var/lib/kubelet/config.yaml.bak /var/lib/kubelet/config.yaml
sudo systemctl restart kubelet

Always keep a timestamped backup:

sudo cp /var/lib/kubelet/config.yaml /var/lib/kubelet/config.yaml.bak.$(date +%Y%m%d)

Placeholder discipline

In any configuration example, replace real values with explicit placeholders in documentation. For instance, do not write a real IP for clusterDNS; use 10.96.0.10 only as an example and tell readers to substitute their own service CIDR DNS IP. The same applies to certificates, tokens, and cloud provider credentials.

Verification and Diagnostics

Effective diagnostics rely on log examination, node status checks, and pod-level observation. This section gives you concrete commands and expected signals to isolate kubelet issues.

Check node status

Run:

kubectl get nodes

If any node shows NotReady, inspect its conditions:

kubectl describe node <NODE_NAME>

Look for KubeletNotReady events and the Ready condition message. A common message is PLEG is not healthy, indicating a problem with the Pod Lifecycle Event Generator, often due to container runtime issues.

Inspect kubelet logs

If the node is NotReady or pods are failing, read kubelet logs with:

journalctl -u kubelet --since "10 minutes ago" --no-pager

Filter for errors:

journalctl -u kubelet --since "1 hour ago" --no-pager | grep -i 'error\|failed'

Expected output when healthy: minimal or no errors. Look for recurring patterns like:

• Failed to start ContainerManager - may indicate cgroup driver misconfiguration.
• PLEG is not healthy - often caused by Docker or containerd issues.
• Failed to get system container stats - check cAdvisor and resource limits.

Verify container runtime health

Kubelet depends on the container runtime (containerd, CRI-O, or Docker). Check the runtime's status. For containerd:

sudo crictl info

Expected output includes runtime version and status. If the runtime is unresponsive, kubelet loses the ability to manage pods.

Pod-level diagnostics

If pods are stuck in ContainerCreating or CrashLoopBackOff, describe the pod:

kubectl describe pod <POD_NAME> -n <NAMESPACE>

Look at the Events section. Example event:

Warning  FailedCreatePodSandBox  2m    kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox

This points to a CNI plugin failure, not kubelet itself, but it is often reported as a kubelet issue.

Use crictl for runtime introspection

List running containers via the runtime:

sudo crictl ps

List pods known to the runtime:

sudo crictl pods

Compare this list with kubectl get pods --all-namespaces to spot orphaned containers or missing sandboxes.

Failure Modes and Recovery

Kubelet failures fall into a few common categories. Each requires a different diagnostic path and recovery strategy.

Failure mode 1: kubelet service not running

Symptom: node shows NotReady, systemctl status kubelet shows inactive (dead) or failed.

Diagnostic:

systemctl status kubelet --no-pager
journalctl -u kubelet -n 100 --no-pager

Common causes and fixes:

• Configuration syntax error: validate /var/lib/kubelet/config.yaml with a YAML linter.
• Missing binary: check /usr/bin/kubelet exists and has execute permissions.
• Systemd unit misconfigured: verify /etc/systemd/system/kubelet.service matches your installation method.

Recovery: fix the underlying issue, then start kubelet:

sudo systemctl start kubelet

Failure mode 2: kubelet running but node NotReady

Symptom: kubelet process is up, but kubectl get nodes shows NotReady.

Diagnostic:

kubectl describe node <NODE_NAME>
journalctl -u kubelet --since "10 minutes ago" --no-pager

Common causes:

• Network plugin not ready: ensure CNI pods are running and network interfaces have IP addresses.
• Disk pressure: check df -h on the node; if root filesystem is above 85% usage, kubelet may evict pods and mark node NotReady. Fix by cleaning up disk or expanding storage.
• Certificate expired: kubelet client certificate may expire. Check certificate validity:

sudo openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -dates

If expired, kubelet cannot authenticate to the API server. Renew certificates (often via kubeadm certs renew or automatic rotation).

Failure mode 3: kubelet crash loop

Symptom: kubelet repeatedly restarts; systemctl status kubelet shows activating (auto-restart).

Diagnostic:

journalctl -u kubelet -n 200 --no-pager | grep -i 'panic\|fatal'

Common causes:

• Incompatible container runtime version: ensure the runtime's CRI version matches Kubernetes version support.
• Corrupted state files: files in /var/lib/kubelet/ can become corrupted. Move them aside (to a backup directory) and restart kubelet; kubelet will recreate them.
• Bug in kubelet version: check known issues for your Kubernetes version and consider upgrading.

Failure mode 4: PLEG is not healthy

Symptom: node periodically goes NotReady, logs show PLEG is not healthy.

This indicates kubelet cannot obtain pod lifecycle events from the runtime, often due to runtime slowness or bugs. Investigate the container runtime metrics and logs, and consider restarting the runtime:

sudo systemctl restart containerd

Then verify kubelet recovers.

Recovery verification

After any recovery action, confirm the node returns to Ready:

kubectl wait --for=condition=Ready node/<NODE_NAME> --timeout=60s

Then check pods are rescheduled and running.

Operations Checklist

Use this checklist during routine maintenance, incident response, or before/after cluster upgrades. Each item is concrete and includes the command to run.

Example workflow: pre-upgrade kubelet check

Before upgrading kubelet or the node OS, run this sequence:

1. Cordon the node to prevent new pods:

kubectl cordon <NODE_NAME>

1. Drain pods (respect PodDisruptionBudgets):

kubectl drain <NODE_NAME> --ignore-daemonsets --delete-emptydir-data

1. Backup kubelet configuration:

sudo cp -r /var/lib/kubelet /var/lib/kubelet.backup.$(date +%Y%m%d)

1. Perform upgrade according to your Kubernetes upgrade guide.

1. Uncordon the node:

kubectl uncordon <NODE_NAME>

1. Verify node is Ready:

kubectl get nodes

1. Check kubelet logs for errors post-upgrade:

journalctl -u kubelet --since "5 minutes ago" --no-pager

Sanity checks after configuration change

If you changed kubelet configuration, run these additional checks:

• For pod scheduling: kubectl run test-pod --image=busybox --restart=Never -- sleep 10 then verify it completes.
• For resource limits: kubectl describe node <NODE_NAME> | grep -A5 Allocated resources to see pressure.

Conclusion

Kubelet production operations require a methodical approach: know your version and environment, make one scoped change at a time, verify with concrete commands, and always have a tested recovery path. The checklist in this article gives you a foundation, but adapt it to your cluster's specifics, such as your container runtime, CNI plugin, and cloud provider.

A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. By applying these practices, you reduce downtime, prevent configuration drift, and maintain cluster health at scale.

As a next step, choose one low-risk verification from the checklist, run it on a test node, and record the results. Then implement the backup and rollback steps for kubelet configuration changes. Review the dependencies in your environment: Kube API Server version compatibility, Node OS limits, and kubeadm-managed certificates. With these habits, you will keep kubelet running smoothly and your workloads stable.