Kubernetes Container Runtimes Production Operations Checklist with Practical Examples

Intro

Operating Kubernetes container runtimes in production requires disciplined, verifiable steps rather than ad hoc troubleshooting. Container runtimes such as containerd and CRI-O sit directly beneath the kubelet and manage the full lifecycle of containers on every node. A misconfigured runtime can lead to pod scheduling failures, image pull errors, unstable networking, and security vulnerabilities that cascade across the cluster.

This article provides a production-focused checklist for Kubernetes container runtime operations. It covers version and environment inventory, safe configuration changes, verification and diagnostics, failure modes and recovery, and an operational checklist. Each section includes concrete commands, expected outputs, and examples using containerd (the default runtime for most managed and self-managed clusters) and CRI-O (common in OpenShift and some RHEL-based environments).

The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets when sharing commands, verify results, and document recovery paths before an incident forces rapid action.

Version and Environment Inventory

Before changing a container runtime, know exactly what is installed, where it runs, and which Kubernetes version it must support. A runtime that is too old may lack required CRI features; one that is too new may not yet be validated with your Kubernetes release.

Step 1: Identify the runtime on each node

Use kubectl to inspect node status and runtime information. The kubectl get nodes -o wide output shows the container runtime version reported by the kubelet.

kubectl get nodes -o wide

Example output (abbreviated):

NAME           STATUS   ROLES           AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
node-1         Ready    control-plane   30d   v1.28.5   10.0.0.11     <none>        Ubuntu 22.04.3 LTS   5.15.0-91-generic   containerd://1.7.11
node-2         Ready    <none>          30d   v1.28.5   10.0.0.12     <none>        Ubuntu 22.04.3 LTS   5.15.0-91-generic   containerd://1.7.11

If the CONTAINER-RUNTIME column is missing, use kubectl describe node <node-name> and look for the Container Runtime Version field in the System Info section.

For a more direct check on a node, SSH into it and run:

# For containerd
containerd --version
# Expected output: containerd github.com/containerd/containerd 1.7.11

# For CRI-O
crio --version
# Expected output: crio version 1.28.2

Step 2: Verify the CRI plugin status

The kubelet communicates with the runtime via the Container Runtime Interface (CRI) plugin. For containerd, check that the CRI plugin is enabled and the correct socket exists.

# On the node
sudo crictl info

Example output snippet:

{
  "status": {
    "conditions": [
      {
        "type": "RuntimeReady",
        "status": true,
        "reason": "",
        "message": ""
      },
      {
        "type": "NetworkReady",
        "status": true,
        "reason": "",
        "message": ""
      }
    ]
  }
}

If RuntimeReady is false, the runtime may be crashed or misconfigured. Check the runtime service status:

sudo systemctl status containerd
# or
sudo systemctl status crio

Step 3: Confirm Kubernetes compatibility

Kubernetes publishes a compatibility matrix for container runtimes. For Kubernetes 1.28, containerd 1.7.x and CRI-O 1.28.x are supported. Running a mismatched version can cause subtle failures, such as pods not starting after an upgrade.

Record the following in your inventory:

• Kubernetes version: kubectl version --short or kubectl version
• Runtime name and version: from kubectl get nodes -o wide or on-node commands
• CRI socket path: typically /run/containerd/containerd.sock for containerd, /var/run/crio/crio.sock for CRI-O
• Kubelet configuration: cat /var/lib/kubelet/config.yaml on the node (path may vary)

Practical example: Inventory on a single node

kubectl get nodes -o wide

Expected output:

containerd://1.7.11

If the output is empty or docker:// (legacy), you may need to migrate or investigate. Only proceed to configuration changes after this inventory is complete and consistent across nodes.

Safe Configuration Path

Changing a runtime configuration can affect every pod on that node. Follow a safe path: back up current configuration, make one scoped change, restart the runtime, and verify that the node remains Ready and that a test pod can start.

Step 1: Back up the current configuration

For containerd, the main configuration file is typically /etc/containerd/config.toml on Linux. Before editing, create a backup with a timestamp.

sudo cp /etc/containerd/config.toml /etc/containerd/config.toml.bak.$(date +%Y%m%d%H%M%S)

For CRI-O, the configuration is often in /etc/crio/crio.conf or drop-in files under /etc/crio/crio.conf.d/. Back up similarly.

Step 2: Make a minimal, documented change

Suppose you need to change the runtime's log level from info to debug for troubleshooting. For containerd, edit config.toml and set:

[debug]
  level = "debug"

For CRI-O, in crio.conf:

[crio.runtime]
log_level = "debug"

Avoid changing multiple settings at once. If a later issue arises, you can isolate the cause.

Step 3: Restart the runtime service

sudo systemctl restart containerd
# or
sudo systemctl restart crio

Check that the service is active and has no errors:

sudo systemctl status containerd --no-pager -l

Expected output (partial):

● containerd.service - containerd container runtime
     Loaded: loaded (/lib/systemd/system/containerd.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2024-02-05 10:15:22 UTC; 5s ago

Step 4: Verify node health and run a test pod

After the runtime restarts, the kubelet should reconnect. Confirm the node remains Ready:

kubectl get node node-1

Expected output:

NAME     STATUS   ROLES           AGE   VERSION
node-1   Ready    control-plane   30d   v1.28.5

If the node briefly goes NotReady and returns, that is normal during a runtime restart. If it stays NotReady, investigate kubelet logs and the runtime socket.

Run a minimal test pod on that node using a node selector (if you need to target a specific node):

apiVersion: v1
kind: Pod
metadata:
  name: runtime-test
spec:
  nodeName: node-1
  containers:
  - name: busybox
    image: busybox:1.36
    command: ["sleep", "3600"]

kubectl apply -f runtime-test.yaml
kubectl get pod runtime-test -o wide

Expected output:

NAME           READY   STATUS    RESTARTS   AGE   IP           NODE     NOMINATED NODE   READINESS GATES
runtime-test   1/1     Running   0          10s   10.244.1.5   node-1   <none>           <none>

If the pod fails to start, check events and logs:

kubectl describe pod runtime-test
kubectl logs runtime-test

Once verified, delete the test pod.

Common configuration changes and safety notes

• Changing the runtime binary path: Ensure the kubelet has the correct --container-runtime-endpoint flag. For containerd, it is unix:///run/containerd/containerd.sock; for CRI-O, unix:///var/run/crio/crio.sock.
• Image registry mirrors: Configure in containerd's config.toml under [plugins."io.containerd.grpc.v1.cri".registry.mirrors]. For CRI-O, use registries.conf. Misconfiguration can cause image pull failures.
• Changing the CRI socket owner/permissions: The kubelet runs as root typically, but if you change permissions, ensure the kubelet can still connect. Test with sudo crictl --runtime-endpoint unix:///run/containerd/containerd.sock ps.

Verification and Diagnostics

Once a change is made, verify that the runtime is healthy and that expected behavior is observed. This section provides systematic diagnostics using crictl, kubectl, and runtime-specific tools.

1. Check runtime health via crictl

crictl is a CLI for CRI-compatible runtimes. It lets you inspect containers, images, and sandboxes directly, bypassing the kubelet.

List running containers on a node:

sudo crictl ps

Expected output (abbreviated):

CONTAINER ID        IMAGE               CREATED             STATE               NAME                ATTEMPT             POD ID
1a2b3c4d5e6f7       nginx:1.25.3        2 hours ago         Running             nginx               0                   0a1b2c3d4e5f6

List images:

sudo crictl images

Inspect a specific container:

sudo crictl inspect 1a2b3c4d5e6f7

This output includes runtime information, resource limits, and mounts.

2. Check kubelet logs for CRI errors

If pods are not starting, the kubelet logs often contain CRI errors.

journalctl -u kubelet -n 100 --no-pager | grep -i cri

Look for messages like Failed to create pod sandbox, rpc error: code = Unavailable desc = connection error, or failed to start container.

Example problematic log:

Feb 05 10:20:15 node-1 kubelet[1234]: E0205 10:20:15.123456    1234 remote_runtime.go:176] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to setup network for sandbox ..."

3. Debug image pull failures

Image pull failures are common runtime issues. Use kubectl describe pod to see events:

kubectl describe pod <pod-name> | grep -A5 Events

Look for Failed to pull image, ErrImagePull, or ImagePullBackOff.

On the node, manually pull the image using the runtime's tool:

For containerd:

sudo ctr image pull docker.io/library/nginx:1.25.3

For CRI-O (using crictl):

sudo crictl pull docker.io/library/nginx:1.25.3

If the pull fails, check network connectivity, registry credentials, and runtime registry configuration.

4. Verify container logs

Use crictl logs to view runtime-level logs for a container, which may differ from kubectl logs.

sudo crictl logs 1a2b3c4d5e6f7

For a pod that is crash-looping, get the previous container logs with kubectl:

kubectl logs <pod-name> --previous

Example output for a crash loop:

Error: unable to open database file: /data/app.db

This points to a volume or permission issue rather than a runtime problem, but it is important to distinguish.

Failure Modes and Recovery

Container runtimes can fail in several ways. Knowing common failure modes and their recovery steps reduces downtime.

Failure mode 1: Runtime service crash

Symptom: Node becomes NotReady, pods cannot start or stop, kubectl describe node shows ContainerRuntimeUnhealthy.

Diagnosis (on the node):

sudo systemctl status containerd

May show Active: failed or inactive (dead).

Check logs:

journalctl -u containerd -n 50 --no-pager

Common causes: corrupted state, disk full, configuration error.

Recovery:

1. Ensure disk space: df -h / and df -h /var/lib/containerd (for containerd).
2. Fix configuration if recently changed (restore backup).
3. Restart service: sudo systemctl restart containerd.
4. Watch status with sudo systemctl status containerd and journalctl -u containerd -f.
5. After restart, check node becomes Ready: kubectl get nodes.

If the runtime repeatedly crashes, consider resetting the runtime state (but this deletes all containers on the node):

For containerd:

sudo systemctl stop containerd
sudo rm -rf /var/lib/containerd/*
sudo systemctl start containerd

WARNING: This is destructive. Only do this if you can reschedule workloads and have backups.

Failure mode 2: CNI plugin failure (network not ready)

Symptom: Pods stuck in ContainerCreating, events show failed to setup network or network plugin is not ready.

Diagnosis:

• Check CNI plugin status: kubectl get pods -n kube-system to ensure CNI pods (e.g., Calico, Cilium, Flannel) are running.
• On the node, check CNI config: ls /etc/cni/net.d/ and cat /etc/cni/net.d/*.conf.
• Runtime logs may show errors from CNI plugin.

Recovery:

• If CNI pods are crashing, restart them: kubectl rollout restart daemonset <cni-daemonset> -n kube-system.
• Ensure the CNI binary and config are present and correct on the node.
• If missing, reinstall the CNI plugin or copy from a working node.

Failure mode 3: Image pull backoff

Symptom: Pods show ImagePullBackOff or ErrImagePull.

Diagnosis:

kubectl describe pod <pod-name> | tail -20

Look for the specific error: authentication failed, timeout, not found.

Recovery:

• If authentication failed, check image pull secrets: kubectl get secret <secret-name> -o yaml and ensure credentials are correct.
• If timeout, check node network access to registry: curl -v https://registry-1.docker.io/v2/ from the node.
• If not found, verify image name and tag.
• For private registries, ensure the runtime is configured with the registry mirror and credentials. For containerd, check config.toml for [plugins."io.containerd.grpc.v1.cri".registry.configs]. For CRI-O, check /etc/containers/registries.conf and auth.json.

Failure mode 4: Disk pressure due to runtime artifacts

Symptom: Node shows DiskPressure, pods evicted, container images or logs filling disk.

Diagnosis:

sudo du -sh /var/lib/containerd /var/log/containers /var/log/pods

Recovery:

• Remove unused images: sudo crictl rmi --prune for CRI-O or sudo ctr images prune for containerd (containerd may require additional flags).
• Clean up exited containers: sudo crictl rm $(sudo crictl ps -a -q --state exited).
• Adjust log rotation settings in the runtime or kubelet configuration (e.g., containerLogMaxSize and containerLogMaxFiles in kubelet config).
• Monitor disk usage going forward.

Recovery verification

After any recovery, verify:

1. Node is Ready: kubectl get nodes
2. Runtime service is active: sudo systemctl is-active containerd
3. Test pod can start: apply a simple pod manifest and check kubectl get pod.
4. crictl info shows RuntimeReady: true and NetworkReady: true.

Operations Checklist

Use the following checklist for routine operations and when making changes. Each item includes a concrete command or check.

Example rollback procedure

Suppose you changed containerd log level to debug and want to revert.

1. Restore backup:

   sudo cp /etc/containerd/config.toml.bak.$(date +%s -d '1 hour ago') /etc/containerd/config.toml

(Use the actual backup filename.)

1. Restart containerd:

   sudo systemctl restart containerd

1. Verify service is active and node is Ready:

   sudo systemctl status containerd
   kubectl get node node-1

1. Remove test pods if any.

Automation suggestions

• Schedule a weekly cron job on each node to run crictl info and alert if runtime is not ready. For example:

  # /etc/cron.d/runtime-check
  0 3 * * * root /usr/local/bin/check-runtime.sh

Script content:

  #!/bin/bash
  if ! sudo crictl info | grep -q '"RuntimeReady": true'; then
    echo "Runtime not ready on $(hostname)" | mail -s "Runtime alert" [email protected]
  fi

• Use a DaemonSet that checks runtime health and reports as a Prometheus metric. Many monitoring stacks include node-exporter with textfile collector; you can write a small script to export crictl info status.

Conclusion

Operating Kubernetes container runtimes in production demands rigorous, repeatable procedures. The checklist and examples in this article provide a foundation for version inventory, safe configuration changes, verification, failure recovery, and routine operations. Always observe before changing, back up configurations, limit changes to one scoped item at a time, and verify with concrete commands and expected outputs.

As a next step, choose a low-risk verification from the Operations Checklist on one node in a staging environment. Record the current state, run the diagnostic, compare with expected results, and review dependencies such as kubelet, pod networking (CNI), and image registries. Then apply the same process to production nodes one at a time, monitoring cluster health after each node.

A reliable workflow makes failures visible, protects sensitive values, limits blast radius, and defines recovery verification before an incident forces a hasty decision. By integrating these practices into your operations, you reduce downtime and improve the stability of your Kubernetes platform.