## Intro

Kubernetes abstracts away many infrastructure details, but container runtime failures can stop your cluster cold. The container runtime is the software that actually runs your containers: it pulls images, manages their lifecycle, and enforces isolation. The most common runtimes in production today are containerd and CRI-O, with Docker as a legacy option. When the runtime misbehaves, you will see errors like `FailedCreatePodSandBox`, `ErrImagePull`, or `CreateContainerConfigError`. These often surface as pod events, but the root cause lies in the runtime configuration or environment.

This guide walks through the most frequent container runtime errors, how to diagnose them, and how to fix them safely. You will learn how to verify your runtime setup, make scoped configuration changes, and recover from failed attempts. Each fix includes a concrete command with expected output, so you can apply it to your own cluster with confidence.

## Version and Environment Inventory

Before changing anything, gather the exact versions and configuration of your runtime. This prevents mismatches and makes rollback easier. The following commands work on any Linux node.

### Check the Active Runtime

Kubernetes uses the Container Runtime Interface (CRI) to talk to the runtime. Determine which runtime your node is using:

```bash
ps aux | grep -E 'containerd|crio|dockerd' | grep -v grep
```

Expected output on a node with containerd:

```text
root 1234 0.1 2.3 1234567 234567 ? Ssl Jan01 12:34 /usr/bin/containerd
```

If you see `crio` instead, adjust the following commands accordingly. For Docker, note that it uses a shim (`dockershim`), which is deprecated.

### Confirm Runtime Version

Every runtime provides a version command. For containerd:

```bash
containerd --version
```

Expected output:

```text
containerd github.com/containerd/containerd v1.7.11
```

For CRI-O:

```bash
crictl version
```

Expected output includes both client and server versions. If `crictl` is not installed, install it from the Kubernetes release page; it is a critical diagnostic tool.

### Check CRI Socket

The kubelet must connect to the runtime via a Unix socket. Verify the configured endpoint:

```bash
ps aux | grep kubelet | grep -o '\(container-runtime-endpoint=[^ ]*\)'
```

Expected output:

```text
--container-runtime-endpoint=unix:///run/containerd/containerd.sock
```

If you see `unix:///var/run/crio/crio.sock`, that indicates CRI-O. Note the path for later.

### Check Runtime Configuration Files

Runtime configuration files are often the source of errors. For containerd, the default config is `/etc/containerd/config.toml`. Check for recent changes:

```bash
stat /etc/containerd/config.toml
```

Look for the modification time to see if it changed before the incident. For CRI-O, the main config is `/etc/crio/crio.conf`.

### Check Node Status

Use `kubectl` to see node conditions, which include runtime health:

```bash
kubectl describe node <node-name> | grep -A5 Conditions
```

Look for `ContainerRuntimeIsReady` or `KubeletReady`. If it is `False` or `Unknown`, the runtime is failing.

### Check Runtime Status with systemctl

If the runtime service is not running, nothing else matters. Verify with:

```bash
systemctl status containerd
```

Expected output when healthy:

```text
● containerd.service - containerd container runtime
   Loaded: loaded (/usr/lib/systemd/system/containerd.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2025-04-18 10:15:22 UTC; 1 day ago
```

If it is `failed` or `inactive`, check the logs with `journalctl -u containerd -n 50 --no-pager`.

### Document Your Baseline

| Component | Command Example | Typical Healthy Output |
|-----------|-----------------|------------------------|
| Runtime process | `ps aux | grep containerd` | Process listed with path |
| Runtime version | `containerd --version` | Version string, e.g., v1.7.11 |
| CRI socket | `ls -l /run/containerd/containerd.sock` | Socket file exists |
| Node readiness | `kubectl get node <node>` | `Ready` status |

This inventory gives you a baseline. If a fix fails, you can compare against these values.

## Safe Configuration Path

Container runtime configuration is sensitive. A small mistake can prevent all pods from starting. Follow a scoped approach: change one setting at a time, test in a non-production environment, and have a rollback plan.

### Common Runtime Errors and Their Causes

Before making changes, understand the error categories:

| Error Message | Likely Cause | Runtime Component |
|---------------|--------------|-------------------|
| `FailedCreatePodSandBox` | CNI plugin missing or misconfigured | Network plugin |
| `ErrImagePull` / `ImagePullBackOff` | Registry authentication or network issue | Image pull |
| `CreateContainerConfigError` | Invalid container spec, e.g., missing ConfigMap | Kubelet / runtime |
| `runc run failed: unable to start container process` | Seccomp, AppArmor, or capability issue | OCI runtime (runc) |
| `cgroups: cgroup mountpoint does not exist` | Systemd cgroup driver mismatch | cgroup configuration |

We will address each with a concrete fix.

### Fix 1: Configure the Correct cgroup Driver

A frequent issue is a mismatch between the cgroup driver used by the runtime and the kubelet. For systemd-based systems, both should use `systemd`. Check the kubelet driver:

```bash
ps aux | grep kubelet | grep -o 'cgroup-driver=[^ ]*'
```

If it returns `--cgroup-driver=cgroupfs` but your runtime uses `systemd`, you will see errors like:

```text
failed to create container: cgroup manager: failed to create cgroup: cgroups: cgroup mountpoint does not exist
```

For containerd, edit `/etc/containerd/config.toml`:

```bash
sudo vi /etc/containerd/config.toml
```

Ensure the following section is present:

```toml
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
  SystemdCgroup = true
```

Then restart containerd:

```bash
sudo systemctl restart containerd
```

Restart kubelet:

```bash
sudo systemctl restart kubelet
```

Verify with:

```bash
sudo journalctl -u kubelet -n 20 | grep -i cgroup
```

You should see no cgroup errors. On CRI-O, set `cgroup_manager = "systemd"` in `/etc/crio/crio.conf` under `[crio.runtime]`.

### Fix 2: Configure Image Pull Secrets

Many `ErrImagePull` errors are due to missing credentials for private registries. Create a secret:

```bash
kubectl create secret docker-registry my-registry-secret \
  --docker-server=myregistry.example.com \
  --docker-username=myuser \
  --docker-password=mypassword \
  --docker-email=my@example.com
```

Then reference it in your pod spec:

```yaml
spec:
  imagePullSecrets:
    - name: my-registry-secret
  containers:
    - name: app
      image: myregistry.example.com/app: v1
```

Apply and verify:

```bash
kubectl apply -f pod.yaml
kubectl get pod <pod-name> -o wide
```

If the pod is `Running`, the pull succeeded. Otherwise, check events: `kubectl describe pod <pod-name>` and look for `Failed to pull image` with a reason like `401 Unauthorized` or `dial tcp: lookup myregistry.example.com: no such host`.

### Fix 3: Configure CNI Network Plugin

A broken network plugin causes `FailedCreatePodSandBox` errors. Verify the CNI configuration directory exists and has a valid config:

```bash
ls /etc/cni/net.d/
cat /etc/cni/net.d/*.conf
```

A typical CNI config for Calico or Flannel includes a `type` field. If the file is missing, install a CNI plugin. For example, to install Calico:

```bash
kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml
```

After installation, check that the CNI pods are running:

```bash
kubectl get pods -n kube-system | grep calico
```

Then test pod creation:

```bash
kubectl run test-pod --image=nginx --restart=Never
kubectl get pod test-pod
```

If it transitions to `Running`, the network is fixed.

### Fix 4: Configure RuntimeClass (Optional)

For specialized workloads, you can define a RuntimeClass to select a different runtime. Create a RuntimeClass object:

```yaml
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: gvisor
handler: runsc
```

Apply it and reference in a pod:

```bash
kubectl apply -f runtimeclass.yaml
```

In the pod spec:

```yaml
spec:
  runtimeClassName: gvisor
  containers:
    - name: test
      image: nginx
```

If the handler is not installed, you will get an error like `Failed to create pod sandbox: unknown runtime handler`. Ensure the runtime handler is configured on each node.

## Verification and Diagnostics

After applying a fix, verify that the runtime is healthy and pods can run. Use these commands to confirm.

### Verify Runtime Service

Check that the runtime service is active and not restarting:

```bash
systemctl status containerd --no-pager -l
```

Look for `Active: active (running)` and no recent errors in logs:

```bash
journalctl -u containerd -n 50 --no-pager
```

### Verify CRI Connectivity

Use `crictl` to talk directly to the runtime via the CRI. Check the runtime version:

```bash
crictl version
```

Expected output shows both client and server versions. Then list current pods from the runtime perspective:

```bash
crictl pods
```

You should see a list of pods. If it returns an error like `connection refused`, the socket path may be wrong. Check the kubelet configuration and adjust the `crictl` endpoint:

```bash
export CONTAINER_RUNTIME_ENDPOINT=unix:///run/containerd/containerd.sock
crictl pods
```

### Verify Pod Lifecycle

Create a simple test pod and watch its status:

```bash
kubectl run runtime-test --image=nginx --restart=Never
kubectl get pod runtime-test -w
```

Wait until it is `Running` (or `Completed` if you override the command). If it stays in `Pending` or `ContainerCreating`, check events:

```bash
kubectl describe pod runtime-test
```

Look for events like `FailedCreatePodSandBox` or `Failed to pull image`. Once verified, delete the test pod:

```bash
kubectl delete pod runtime-test
```

### Check Node Conditions

After a fix, wait a moment and check node conditions again:

```bash
kubectl describe node <node-name> | grep -A5 Conditions
```

`ContainerRuntimeIsReady` should be `True`, and the node should be `Ready`.

### Diagnostics Commands Summary

| Check | Command | Expected Healthy Output |
|-------|---------|-------------------------|
| Runtime service status | `systemctl status containerd` | active (running) |
| CRI connectivity | `crictl version` | server version returned |
| Pod creation test | `kubectl run ... --image=nginx` | pod becomes Running |
| Node condition | `kubectl describe node` | RuntimeReady=True |

## Failure Modes and Recovery

Even with careful changes, things can go wrong. Here are common failure scenarios after a runtime configuration change and how to recover.

### Failure: Runtime Fails to Start After Configuration Change

If containerd fails to start after editing `config.toml`, check the logs for syntax errors:

```bash
sudo journalctl -u containerd -n 50 --no-pager
```

Look for lines like `failed to load config: near line 10: unexpected token`. Rollback by restoring the previous config file:

```bash
sudo cp /etc/containerd/config.toml.backup /etc/containerd/config.toml
sudo systemctl restart containerd
```

Then restart kubelet:

```bash
sudo systemctl restart kubelet
```

### Failure: Kubelet Cannot Connect to Runtime

If the CRI socket path changed or the runtime is down, kubelet will error. Check kubelet logs:

```bash
sudo journalctl -u kubelet -n 50 --no-pager
```

Look for `failed to connect to containerd` or `dial unix /run/containerd/containerd.sock: connect: no such file or directory`. Verify the socket exists:

```bash
ls -l /run/containerd/containerd.sock
```

If missing, ensure containerd is running. If the socket path is wrong in kubelet config, update `/var/lib/kubelet/config.yaml` or the kubelet drop-in. Then restart kubelet.

### Failure: Pods Stuck in Terminating State

If a pod cannot be deleted because the runtime is unresponsive, force delete the pod:

```bash
kubectl delete pod <pod-name> --force --grace-period=0
```

If that does not work, restart the runtime:

```bash
sudo systemctl restart containerd
```

Then clean up any orphaned containers with `crictl rm`:

```bash
crictl ps -a | grep <pod-name>
crictl rm <container-id>
```

### Rollback Planning

Always make a backup of runtime configs before editing:

```bash
sudo cp /etc/containerd/config.toml /etc/containerd/config.toml.backup
```

For CRI-O:

```bash
sudo cp /etc/crio/crio.conf /etc/crio/crio.conf.backup
```

Also note the exact changes made, so you can revert them manually if needed.

## Operations Checklist

Use this checklist to regularly audit your container runtime and speed up troubleshooting.

### Daily or Weekly Checks

- Verify runtime service is active: `systemctl is-active containerd` should return `active`.
- Check node readiness: `kubectl get nodes` should show all nodes `Ready`.
- Monitor runtime logs for repeated errors: `journalctl -u containerd --since "24 hours ago" | grep -i error`.
- Check disk space on `/var/lib/containerd` (image store) and `/var/lib/kubelet` (pod storage) with `df -h`.
- Ensure critical CNI pods are running: `kubectl get pods -n kube-system | grep -E 'calico|flannel|weave|cilium'`.

### Before Making Configuration Changes

- Backup current runtime configuration files.
- Record current runtime version and kubelet version.
- Test changes in a staging environment first.
- Plan rollback steps.
- Schedule a maintenance window if needed.

### After Making Configuration Changes

- Restart the runtime service and check status.
- Restart kubelet and check status.
- Verify CRI connectivity with `crictl version`.
- Run a test pod to confirm functionality.
- Check node conditions for runtime readiness.
- Monitor logs for new errors for at least 10 minutes.

### Monthly Audit

- Review runtime configuration against best practices (e.g., cgroup driver, CNI settings).
- Check for runtime updates and apply in a controlled manner.
- Test pod creation, deletion, and exec operations.
- Validate image pull from all used registries.

## Common Pitfalls and Misconceptions

Even experienced operators hit these recurring traps. Understanding them upfront can save hours of debugging.

### Pitfall 1: Mixing cgroup Drivers

**Why it happens:** You install a runtime that defaults to `cgroupfs` on a systemd host, or you copy a Kubernetes configuration from an older deployment without updating the driver.

**How to avoid:** Always set both kubelet and runtime to `systemd` on OSes with systemd. Add a pre-deployment check:

```bash
# Check runtime driver
containerd config dump | grep -i systemd_cgroup
# Check kubelet driver
ps aux | grep kubelet | grep -o 'cgroup-driver=[^ ]*'
```

If they differ, fix before rolling out workloads.

**Recovery:** Apply Fix 1, then restart services. Verify by creating a test pod; if it still fails, inspect `journalctl -u kubelet` for cgroup errors.

### Pitfall 2: Editing Runtime Config Without Backups

**Why it happens:** You need a quick fix during an incident and skip the backup step.

**How to avoid:** Make backups part of your change process. Use a timestamped copy:

```bash
sudo cp /etc/containerd/config.toml /etc/containerd/config.toml.$(date +%Y%m%d%H%M%S).bak
```

**Recovery:** If you didn't back up, check package manager for the original config: `rpm -V containerd` or `dpkg -S containerd`. For a running system, you can extract the current config with `containerd config dump` and save it.

### Pitfall 3: Assuming CNI Is Always Pre-installed

**Why it happens:** Managed Kubernetes services often install CNI automatically, but self-managed clusters require manual setup.

**How to avoid:** After cluster bootstrap, immediately verify CNI pods are running and test with a sample pod before deploying production workloads.

**Recovery:** Install a CNI plugin as described in Fix 3. If pods are stuck in `ContainerCreating`, check events for sandbox errors.

### Pitfall 4: Ignoring Runtime Log Rotation

**Why it happens:** The runtime writes logs to disk without rotation, filling the filesystem over time.

**How to avoid:** Configure log rotation in the runtime config. For containerd, set:

```toml
[plugins."io.containerd.grpc.v1.cri".containerd]
  max_container_log_line_size = 16384
```

And ensure systemd journald limits are appropriate. Monitor disk usage regularly.

**Recovery:** If disk is full, prune unused images (`crictl rmi --prune`) and remove old logs. Then configure rotation to prevent recurrence.

### Pitfall 5: Running Containers as Root by Default

**Why it happens:** Many container images default to the root user, increasing the blast radius of a breakout.

**How to avoid:** Enforce non-root with pod security policies or admission controllers. In your pod spec:

```yaml
securityContext:
  runAsNonRoot: true
  runAsUser: 1000
```

**Recovery:** If a container is running as root, update the image or security context. You can also set a default RuntimeClass with `runAsNonRoot` enforced.

## Conclusion

Container runtime errors can be intimidating, but they are usually caused by a few common misconfigurations. By methodically checking cgroup drivers, image pull settings, CNI networking, and runtime health, you can resolve most issues quickly. Always work with a safe configuration path: make one change at a time, verify with concrete commands, and have a rollback plan. The checklist provided will help you maintain a healthy runtime and minimize downtime. With these practical examples and diagnostic commands, you can confidently troubleshoot and fix Kubernetes container runtime errors in your own environment.