Kubernetes Container Runtimes: Common Errors and Fixes with Practical Examples

Intro

Kubernetes abstracts away many infrastructure details, but container runtime failures can stop your cluster cold. The container runtime is the software that actually runs your containers: it pulls images, manages their lifecycle, and enforces isolation. The most common runtimes in production today are containerd and CRI-O, with Docker as a legacy option. When the runtime misbehaves, you will see errors like FailedCreatePodSandBox, ErrImagePull, or CreateContainerConfigError. These often surface as pod events, but the root cause lies in the runtime configuration or environment.

This guide walks through the most frequent container runtime errors, how to diagnose them, and how to fix them safely. You will learn how to verify your runtime setup, make scoped configuration changes, and recover from failed attempts. Each fix includes a concrete command with expected output, so you can apply it to your own cluster with confidence.

Version and Environment Inventory

Before changing anything, gather the exact versions and configuration of your runtime. This prevents mismatches and makes rollback easier. The following commands work on any Linux node.

Check the Active Runtime

Kubernetes uses the Container Runtime Interface (CRI) to talk to the runtime. Determine which runtime your node is using:

ps aux | grep -E 'containerd|crio|dockerd' | grep -v grep

Expected output on a node with containerd:

root 1234 0.1 2.3 1234567 234567 ? Ssl Jan01 12:34 /usr/bin/containerd

If you see crio instead, adjust the following commands accordingly. For Docker, note that it uses a shim (dockershim), which is deprecated.

Confirm Runtime Version

Every runtime provides a version command. For containerd:

containerd --version

Expected output:

containerd github.com/containerd/containerd v1.7.11

For CRI-O:

crictl version

Expected output includes both client and server versions. If crictl is not installed, install it from the Kubernetes release page; it is a critical diagnostic tool.

Check CRI Socket

The kubelet must connect to the runtime via a Unix socket. Verify the configured endpoint:

ps aux | grep kubelet | grep -o '\(container-runtime-endpoint=[^ ]*\)'

Expected output:

--container-runtime-endpoint=unix:///run/containerd/containerd.sock

If you see unix:///var/run/crio/crio.sock, that indicates CRI-O. Note the path for later.

Check Runtime Configuration Files

Runtime configuration files are often the source of errors. For containerd, the default config is /etc/containerd/config.toml. Check for recent changes:

stat /etc/containerd/config.toml

Look for the modification time to see if it changed before the incident. For CRI-O, the main config is /etc/crio/crio.conf.

Check Node Status

Use kubectl to see node conditions, which include runtime health:

kubectl describe node <node-name> | grep -A5 Conditions

Look for ContainerRuntimeIsReady or KubeletReady. If it is False or Unknown, the runtime is failing.

Check Runtime Status with systemctl

If the runtime service is not running, nothing else matters. Verify with:

systemctl status containerd

Expected output when healthy:

● containerd.service - containerd container runtime
   Loaded: loaded (/usr/lib/systemd/system/containerd.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2025-04-18 10:15:22 UTC; 1 day ago

If it is failed or inactive, check the logs with journalctl -u containerd -n 50 --no-pager.

Document Your Baseline

This inventory gives you a baseline. If a fix fails, you can compare against these values.

Safe Configuration Path

Container runtime configuration is sensitive. A small mistake can prevent all pods from starting. Follow a scoped approach: change one setting at a time, test in a non-production environment, and have a rollback plan.

Common Runtime Errors and Their Causes

Before making changes, understand the error categories:

We will address each with a concrete fix.

Fix 1: Configure the Correct cgroup Driver

A frequent issue is a mismatch between the cgroup driver used by the runtime and the kubelet. For systemd-based systems, both should use systemd. Check the kubelet driver:

ps aux | grep kubelet | grep -o 'cgroup-driver=[^ ]*'

If it returns --cgroup-driver=cgroupfs but your runtime uses systemd, you will see errors like:

failed to create container: cgroup manager: failed to create cgroup: cgroups: cgroup mountpoint does not exist

For containerd, edit /etc/containerd/config.toml:

sudo vi /etc/containerd/config.toml

Ensure the following section is present:

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
  SystemdCgroup = true

Then restart containerd:

sudo systemctl restart containerd

Restart kubelet:

sudo systemctl restart kubelet

Verify with:

sudo journalctl -u kubelet -n 20 | grep -i cgroup

You should see no cgroup errors. On CRI-O, set cgroup_manager = "systemd" in /etc/crio/crio.conf under [crio.runtime].

Fix 2: Configure Image Pull Secrets

Many ErrImagePull errors are due to missing credentials for private registries. Create a secret:

kubectl create secret docker-registry my-registry-secret \
  --docker-server=myregistry.example.com \
  --docker-username=myuser \
  --docker-password=mypassword \
  [email protected]

Then reference it in your pod spec:

spec:
  imagePullSecrets:
    - name: my-registry-secret
  containers:
    - name: app
      image: myregistry.example.com/app: v1

Apply and verify:

kubectl apply -f pod.yaml
kubectl get pod <pod-name> -o wide

If the pod is Running, the pull succeeded. Otherwise, check events: kubectl describe pod <pod-name> and look for Failed to pull image with a reason like 401 Unauthorized or dial tcp: lookup myregistry.example.com: no such host.

Fix 3: Configure CNI Network Plugin

A broken network plugin causes FailedCreatePodSandBox errors. Verify the CNI configuration directory exists and has a valid config:

ls /etc/cni/net.d/
cat /etc/cni/net.d/*.conf

A typical CNI config for Calico or Flannel includes a type field. If the file is missing, install a CNI plugin. For example, to install Calico:

kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

After installation, check that the CNI pods are running:

kubectl get pods -n kube-system | grep calico

Then test pod creation:

kubectl run test-pod --image=nginx --restart=Never
kubectl get pod test-pod

If it transitions to Running, the network is fixed.

Fix 4: Configure RuntimeClass (Optional)

For specialized workloads, you can define a RuntimeClass to select a different runtime. Create a RuntimeClass object:

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: gvisor
handler: runsc

Apply it and reference in a pod:

kubectl apply -f runtimeclass.yaml

In the pod spec:

spec:
  runtimeClassName: gvisor
  containers:
    - name: test
      image: nginx

If the handler is not installed, you will get an error like Failed to create pod sandbox: unknown runtime handler. Ensure the runtime handler is configured on each node.

Verification and Diagnostics

After applying a fix, verify that the runtime is healthy and pods can run. Use these commands to confirm.

Verify Runtime Service

Check that the runtime service is active and not restarting:

systemctl status containerd --no-pager -l

Look for Active: active (running) and no recent errors in logs:

journalctl -u containerd -n 50 --no-pager

Verify CRI Connectivity

Use crictl to talk directly to the runtime via the CRI. Check the runtime version:

crictl version

Expected output shows both client and server versions. Then list current pods from the runtime perspective:

crictl pods

You should see a list of pods. If it returns an error like connection refused, the socket path may be wrong. Check the kubelet configuration and adjust the crictl endpoint:

export CONTAINER_RUNTIME_ENDPOINT=unix:///run/containerd/containerd.sock
crictl pods

Verify Pod Lifecycle

Create a simple test pod and watch its status:

kubectl run runtime-test --image=nginx --restart=Never
kubectl get pod runtime-test -w

Wait until it is Running (or Completed if you override the command). If it stays in Pending or ContainerCreating, check events:

kubectl describe pod runtime-test

Look for events like FailedCreatePodSandBox or Failed to pull image. Once verified, delete the test pod:

kubectl delete pod runtime-test

Check Node Conditions

After a fix, wait a moment and check node conditions again:

kubectl describe node <node-name> | grep -A5 Conditions

ContainerRuntimeIsReady should be True, and the node should be Ready.

Diagnostics Commands Summary

Failure Modes and Recovery

Even with careful changes, things can go wrong. Here are common failure scenarios after a runtime configuration change and how to recover.

Failure: Runtime Fails to Start After Configuration Change

If containerd fails to start after editing config.toml, check the logs for syntax errors:

sudo journalctl -u containerd -n 50 --no-pager

Look for lines like failed to load config: near line 10: unexpected token. Rollback by restoring the previous config file:

sudo cp /etc/containerd/config.toml.backup /etc/containerd/config.toml
sudo systemctl restart containerd

Then restart kubelet:

sudo systemctl restart kubelet

Failure: Kubelet Cannot Connect to Runtime

If the CRI socket path changed or the runtime is down, kubelet will error. Check kubelet logs:

sudo journalctl -u kubelet -n 50 --no-pager

Look for failed to connect to containerd or dial unix /run/containerd/containerd.sock: connect: no such file or directory. Verify the socket exists:

ls -l /run/containerd/containerd.sock

If missing, ensure containerd is running. If the socket path is wrong in kubelet config, update /var/lib/kubelet/config.yaml or the kubelet drop-in. Then restart kubelet.

Failure: Pods Stuck in Terminating State

If a pod cannot be deleted because the runtime is unresponsive, force delete the pod:

kubectl delete pod <pod-name> --force --grace-period=0

If that does not work, restart the runtime:

sudo systemctl restart containerd

Then clean up any orphaned containers with crictl rm:

crictl ps -a | grep <pod-name>
crictl rm <container-id>

Rollback Planning

Always make a backup of runtime configs before editing:

sudo cp /etc/containerd/config.toml /etc/containerd/config.toml.backup

For CRI-O:

sudo cp /etc/crio/crio.conf /etc/crio/crio.conf.backup

Also note the exact changes made, so you can revert them manually if needed.

Operations Checklist

Use this checklist to regularly audit your container runtime and speed up troubleshooting.

Daily or Weekly Checks

• Verify runtime service is active: systemctl is-active containerd should return active.
• Check node readiness: kubectl get nodes should show all nodes Ready.
• Monitor runtime logs for repeated errors: journalctl -u containerd --since "24 hours ago" | grep -i error.
• Check disk space on /var/lib/containerd (image store) and /var/lib/kubelet (pod storage) with df -h.
• Ensure critical CNI pods are running: kubectl get pods -n kube-system | grep -E 'calico|flannel|weave|cilium'.

Before Making Configuration Changes

• Backup current runtime configuration files.
• Record current runtime version and kubelet version.
• Test changes in a staging environment first.
• Plan rollback steps.
• Schedule a maintenance window if needed.

After Making Configuration Changes

• Restart the runtime service and check status.
• Restart kubelet and check status.
• Verify CRI connectivity with crictl version.
• Run a test pod to confirm functionality.
• Check node conditions for runtime readiness.
• Monitor logs for new errors for at least 10 minutes.

Monthly Audit

• Review runtime configuration against best practices (e.g., cgroup driver, CNI settings).
• Check for runtime updates and apply in a controlled manner.
• Test pod creation, deletion, and exec operations.
• Validate image pull from all used registries.

Common Pitfalls and Misconceptions

Even experienced operators hit these recurring traps. Understanding them upfront can save hours of debugging.

Pitfall 1: Mixing cgroup Drivers

Why it happens: You install a runtime that defaults to cgroupfs on a systemd host, or you copy a Kubernetes configuration from an older deployment without updating the driver.

How to avoid: Always set both kubelet and runtime to systemd on OSes with systemd. Add a pre-deployment check:

# Check runtime driver
containerd config dump | grep -i systemd_cgroup
# Check kubelet driver
ps aux | grep kubelet | grep -o 'cgroup-driver=[^ ]*'

If they differ, fix before rolling out workloads.

Recovery: Apply Fix 1, then restart services. Verify by creating a test pod; if it still fails, inspect journalctl -u kubelet for cgroup errors.

Pitfall 2: Editing Runtime Config Without Backups

Why it happens: You need a quick fix during an incident and skip the backup step.

How to avoid: Make backups part of your change process. Use a timestamped copy:

sudo cp /etc/containerd/config.toml /etc/containerd/config.toml.$(date +%Y%m%d%H%M%S).bak

Recovery: If you didn't back up, check package manager for the original config: rpm -V containerd or dpkg -S containerd. For a running system, you can extract the current config with containerd config dump and save it.

Pitfall 3: Assuming CNI Is Always Pre-installed

Why it happens: Managed Kubernetes services often install CNI automatically, but self-managed clusters require manual setup.

How to avoid: After cluster bootstrap, immediately verify CNI pods are running and test with a sample pod before deploying production workloads.

Recovery: Install a CNI plugin as described in Fix 3. If pods are stuck in ContainerCreating, check events for sandbox errors.

Pitfall 4: Ignoring Runtime Log Rotation

Why it happens: The runtime writes logs to disk without rotation, filling the filesystem over time.

How to avoid: Configure log rotation in the runtime config. For containerd, set:

[plugins."io.containerd.grpc.v1.cri".containerd]
  max_container_log_line_size = 16384

And ensure systemd journald limits are appropriate. Monitor disk usage regularly.

Recovery: If disk is full, prune unused images (crictl rmi --prune) and remove old logs. Then configure rotation to prevent recurrence.

Pitfall 5: Running Containers as Root by Default

Why it happens: Many container images default to the root user, increasing the blast radius of a breakout.

How to avoid: Enforce non-root with pod security policies or admission controllers. In your pod spec:

securityContext:
  runAsNonRoot: true
  runAsUser: 1000

Recovery: If a container is running as root, update the image or security context. You can also set a default RuntimeClass with runAsNonRoot enforced.

Conclusion

Container runtime errors can be intimidating, but they are usually caused by a few common misconfigurations. By methodically checking cgroup drivers, image pull settings, CNI networking, and runtime health, you can resolve most issues quickly. Always work with a safe configuration path: make one change at a time, verify with concrete commands, and have a rollback plan. The checklist provided will help you maintain a healthy runtime and minimize downtime. With these practical examples and diagnostic commands, you can confidently troubleshoot and fix Kubernetes container runtime errors in your own environment.