## Intro

Container runtimes are the software that actually run your containers on each Kubernetes node. If a runtime is misconfigured, every workload on that node is at risk. Hardening the runtime is therefore a foundational security task for any Kubernetes cluster. This guide focuses on the two most common Kubernetes container runtimes today: containerd (used by Docker, GKE, AKS, EKS, and many others) and CRI-O (used by OpenShift and some RHEL-based distributions). We will walk through version and environment inventory, safe configuration changes, verification, failure recovery, and provide checklists and common pitfalls.

We will use concrete commands and configuration snippets that you can adapt to your environment. Always test in a staging cluster first and follow your change management process. The goal is to make changes that are observable, reversible, and scoped to the intended resource.

## Version and Environment Inventory

Before making any change, know exactly what runtime you are running, its version, and how it is configured. This information is essential because security features and configuration options vary by version and runtime type.

### Identify the runtime and version

On each node, check which runtime is active. You can do this via the Kubernetes API or directly on the node.

Using kubectl, list nodes and their container runtime version:

```bash
kubectl get nodes -o wide
```

Look at the `CONTAINER-RUNTIME` column. It will show something like `containerd://1.6.18` or `cri-o://1.26.0`.

Alternatively, SSH into the node and run the runtime's version command directly:

- For containerd: `containerd --version` and `ctr version`
- For CRI-O: `crio --version`

Record the exact version and the Kubernetes version (`kubectl version --short`). Check the runtime's release notes and security advisories for your version. For containerd, known vulnerabilities are tracked on its GitHub security page; for CRI-O, check the Red Hat or upstream advisories.

### Inspect current configuration

Both runtimes use a configuration file. The default locations are:

- containerd: `/etc/containerd/config.toml`
- CRI-O: `/etc/crio/crio.conf` (and drop-in files in `/etc/crio/crio.conf.d/`)

View the current configuration (read-only) to understand what is already set. For containerd, you can print the default configuration (which may differ from what's active) with:

```bash
containerd config default > /tmp/containerd-default.toml
diff /tmp/containerd-default.toml /etc/containerd/config.toml
```

This shows you if any customizations have been made. For CRI-O, use `crio --config /etc/crio/crio.conf config` to dump the effective configuration.

Capture the current state and a timestamp. This serves as a baseline for rollback and troubleshooting. Store this information securely, as configuration may contain sensitive data (like registry credentials or certificate paths).

### Check runtime dependencies

The runtime depends on the kernel, container images, and the kubelet. Ensure the operating system and kernel are up to date with security patches. For containerd, check that the CNI plugins are installed and compatible. For CRI-O, verify that `conmon` and `runc` (or `crun`) versions are supported.

Use the following commands to check key binaries:

```bash
runc --version
conmon --version   # CRI-O only
cni-plugins --version  # if available
```

### Blast radius assessment

Before changing runtime configuration, understand which workloads run on the node. A runtime restart will affect all pods on that node. Use:

```bash
kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=<node-name>
```

Plan for a maintenance window if the node hosts critical workloads. Ensure you can drain the node safely if needed.

## Safe Configuration Path

Now that you have a baseline, plan and execute configuration changes carefully. The following subsections cover key hardening areas for both runtimes.

### General hardening principles

- **Use a dedicated user for the runtime**: Run the runtime as a non-root user (e.g., `containerd` or `crio`). Check the service file (`/etc/systemd/system/containerd.service` or `/etc/systemd/system/crio.service`) for the `User=` directive. If it is missing or set to root, add it.
- **Restrict runtime socket permissions**: The runtime's gRPC socket (e.g., `/run/containerd/containerd.sock`) should be accessible only to root and the kubelet. Check permissions with `ls -l /run/containerd/containerd.sock`; it should be `srw-------` owned by root. If not, adjust the socket path or permissions.
- **Enable seccomp**: Seccomp restricts the system calls a container can make. Ensure the runtime is configured to use a default seccomp profile. For containerd, in the `plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options` section, you can set `NoNewPrivileges = true` and ensure the default seccomp profile is applied. For CRI-O, set `seccomp_profile = "/path/to/profile.json"` in the `crio.runtime` section, or ensure the default profile is used.
- **Use appropriate capabilities**: Limit capabilities granted to containers. By default, runtimes drop many capabilities already; do not add back unnecessary ones. Verify the default set with `capsh --print` inside a test container.

### containerd-specific configuration

Edit `/etc/containerd/config.toml`. After making changes, validate and restart the service.

**Example: enable CRI plugin and set default runtime.**

In the config, ensure the CRI plugin is enabled:

```toml
version = 2

[plugins."io.containerd.grpc.v1.cri"]
  sandbox_image = "registry.k8s.io/pause:3.9"

[plugins."io.containerd.grpc.v1.cri".containerd]
  default_runtime_name = "runc"
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
    runtime_type = "io.containerd.runc.v2"
    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
      NoNewPrivileges = true
      SystemdCgroup = true
```

By default, `NoNewPrivileges` is false. Setting it to true ensures containers cannot gain additional privileges via setuid binaries. This is a recommended hardening step.

**Example: set seccomp profile for containerd**

Containerd CRI plugin applies the Kubernetes default seccomp profile automatically when requested via the pod security context, but you can also set a default at the runtime level. In the `runc.options` section, add:

```toml
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
  NoNewPrivileges = true
  SystemdCgroup = true
  # Seccomp profile can be specified via annotation or pod spec.
  # To enforce a default, you could use a mutating webhook.
```

For more advanced seccomp defaults, consider using Kubernetes pod security standards or a policy engine.

After editing, validate:

```bash
containerd config dump > /tmp/containerd-new.toml
containerd config verify < /tmp/containerd-new.toml
```

If no errors, restart containerd:

```bash
sudo systemctl restart containerd
```

Check the service status:

```bash
sudo systemctl status containerd
```

### CRI-O-specific configuration

Edit `/etc/crio/crio.conf` or add a drop-in file under `/etc/crio/crio.conf.d/`.

**Example: Set default capabilities drop and seccomp**

In the `crio.runtime` section:

```toml
[crio.runtime]
default_capabilities = [
  "CHOWN",
  "DAC_OVERRIDE",
  "FSETID",
  "FOWNER",
  "SETGID",
  "SETUID",
  "SETPCAP",
  "NET_BIND_SERVICE",
  "KILL",
]
seccomp_profile = "/etc/crio/seccomp.json"
```

Ensure the `seccomp_profile` points to a valid seccomp profile file. You can copy the default from Kubernetes or create a custom one.

Also, ensure `conmon` and `runc` are secure:

```toml
[crio.runtime]
conmon = "/usr/bin/conmon"
conmon_cgroup = "pod"
```

After changes, validate:

```bash
crio config validate
```

Restart CRI-O:

```bash
sudo systemctl restart crio
```

### Applying configuration changes safely

Always test on a single node first. Drain the node to avoid disruption:

```bash
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data
```

Make your configuration change, restart the runtime, and uncordon the node:

```bash
kubectl uncordon <node-name>
```

Monitor pod health on that node before proceeding to other nodes.

## Verification and Diagnostics

After any configuration change, verify that the runtime is functioning correctly and that security controls are enforced.

### Verify runtime is running

Check the service status:

```bash
systemctl status containerd   # or crio
```

Look for errors in the logs:

```bash
journalctl -u containerd -n 100 --no-pager
```

For CRI-O:

```bash
journalctl -u crio -n 100 --no-pager
```

### Test container operations

Run a simple pod on the node (or cluster) and ensure it starts:

```bash
kubectl run test-pod --image=busybox --restart=Never --command -- sleep 3600
kubectl get pod test-pod
```

Check that the pod is running. Then exec into it to test security features:

```bash
kubectl exec -it test-pod -- sh
```

Inside the container, test that `setuid` binaries are restricted if `NoNewPrivileges` is set:

```bash
# Try to gain root
su -
```

This should fail if the container is running as non-root and `NoNewPrivileges` is true. If you are root, test whether you can change user to a higher privilege (you can't).

Test seccomp: attempt to use a syscall not in the default profile, e.g., `reboot` or `mount` with unusual arguments. In a secure runtime, these should be denied.

Example to test seccomp with a custom profile: create a pod with a restrictive seccomp profile and try to run a disallowed syscall.

### Check pod security context

Ensure pods are running with restricted security contexts. Use:

```bash
kubectl get pod test-pod -o yaml | grep -A 10 securityContext
```

Look for `runAsNonRoot: true`, `allowPrivilegeEscalation: false`, `seccompProfile: RuntimeDefault`, etc. These should be set by your pod policies or admission controllers.

### Monitor runtime metrics

Both runtimes expose metrics. For containerd, you can query the metrics endpoint if enabled (usually at `http://localhost:1338/metrics`). For CRI-O, metrics are at `http://localhost:9090/metrics`. Use `curl` to check metrics related to containers and errors.

Example:

```bash
curl -s http://localhost:1338/metrics | grep -E 'container_|runtime_'
```

If metrics are not enabled, consider enabling them for monitoring in production.

## Failure Modes and Recovery

Despite careful planning, failures can occur. This section covers common failure modes and how to recover.

### Runtime fails to start after configuration change

**Symptom**: `systemctl status containerd` shows failed, and `journalctl -u containerd` shows errors like "invalid config" or "failed to load config".

**Cause**: Syntax error in config file, unsupported option, or file permissions issue.

**Recovery**: Roll back to the previous configuration.

If you made a backup (you should), restore it:

```bash
sudo cp /etc/containerd/config.toml.bak /etc/containerd/config.toml
sudo systemctl restart containerd
```

If no backup, you can use the default config generated earlier:

```bash
containerd config default > /etc/containerd/config.toml
sudo systemctl restart containerd
```

Then reapply changes one by one, validating after each.

### Pods cannot start after runtime restart

**Symptom**: New pods stay in `ContainerCreating` state, and `kubectl describe pod` shows errors like "failed to create containerd task" or "rpc error: code = Unknown desc = failed to start container".

**Cause**: Could be missing CNI plugins, incorrect sandbox image, or seccomp profile path incorrect.

**Recovery**: Check runtime logs for detailed error. If CNI issue, verify network plugin config (`/etc/cni/net.d/`). If sandbox image issue, ensure the pause image is correct and pullable (`registry.k8s.io/pause:3.9`).

If seccomp profile path is wrong, fix it or remove the seccomp setting and restart.

### Node becomes NotReady

**Symptom**: `kubectl get nodes` shows node as `NotReady` after runtime restart.

**Cause**: Kubelet lost connection to runtime, perhaps due to socket permission change.

**Recovery**: Check kubelet logs:

```bash
journalctl -u kubelet -n 100
```

If socket permission was changed incorrectly, restore original permissions (e.g., `chmod 660` for containerd socket, owner root:kubelet). Then restart kubelet.

### Rollback strategy

Always maintain versioned backups of configuration files. Use `etckeeper` or manual backups with timestamps. Before a change, copy the config:

```bash
sudo cp /etc/containerd/config.toml /etc/containerd/config.toml.$(date +%Y%m%d%H%M%S)
```

If all else fails, you can reinstall the runtime package (this will restore default configs but you will lose custom settings) and then reapply your hardening from documentation.

## Operations Checklist

Use this checklist before and after any runtime hardening change.

### Pre-change checklist

- [ ] Identify runtime type and version: `kubectl get nodes -o wide` and record.
- [ ] Read the runtime's security advisories and release notes for your version.
- [ ] Back up current configuration files with timestamps.
- [ ] Document current state (config dump, runtime status, metrics baseline).
- [ ] Identify workloads on target node(s) and plan maintenance window.
- [ ] Drain node(s) if required: `kubectl drain <node> --ignore-daemonsets`.
- [ ] Prepare rollback plan and rollback commands.
- [ ] Ensure monitoring and alerting are active.
- [ ] Designate an owner for this change: **Priya Shah, Engineering Lead** (or your equivalent).

### Post-change verification checklist

- [ ] Runtime service is active: `systemctl status containerd` (or `crio`).
- [ ] Runtime logs show no fatal errors: `journalctl -u containerd -n 50`.
- [ ] Test pod starts and runs: `kubectl run test-pod --image=busybox --command -- sleep 60`.
- [ ] Test pod respects security context: check `kubectl get pod test-pod -o yaml | grep securityContext`.
- [ ] Node is Ready: `kubectl get nodes`.
- [ ] Existing pods on node are still healthy (if not drained).
- [ ] If drained, uncordon node: `kubectl uncordon <node>`.
- [ ] Clean up test pod: `kubectl delete pod test-pod`.
- [ ] Update documentation and configuration management repository.
- [ ] Schedule a post-change review within one week to catch delayed issues.

### Recurring maintenance

- **Monthly**: Review runtime versions against security advisories and plan upgrades.
- **Quarterly**: Audit runtime configuration for compliance with hardening benchmarks (CIS, STIG).
- **Continuously**: Monitor runtime metrics and logs for anomalies.

**Owner for recurring checks**: **Alex Chen, Platform Security Engineer** (or your equivalent). The owner reviews and updates this checklist quarterly.

## Common Pitfalls

Avoid these common mistakes when hardening container runtimes.

### 1. Not testing on a single node first

**What happens**: You apply a configuration change across all nodes simultaneously, causing a cluster-wide outage.

**Why**: Overconfidence or time pressure.

**How to avoid**: Always start with one non-critical node. Drain it, make the change, verify, and then roll out gradually.

### 2. Ignoring version compatibility

**What happens**: You apply configuration options that do not exist in your runtime version, leading to startup failure.

**Why**: Copying config snippets from the internet without checking the version.

**How to avoid**: Always check the documentation for your specific runtime version. Use `containerd config default` or `crio config dump` to see the current supported options.

### 3. Over-restricting seccomp or capabilities

**What happens**: Applications fail because they need a syscall or capability that was removed.

**Why**: Assuming a generic profile fits all workloads.

**How to avoid**: Start with the default seccomp profile and only remove capabilities if justified. Use audit logs to identify missing syscalls, then customize profiles.

### 4. Forgetting to backup and test rollback

**What happens**: When a change fails, you cannot quickly revert, extending downtime.

**Why**: Not treating config as code with version control.

**How to avoid**: Always backup config files and store them in a version control system. Actually test the rollback procedure in a staging environment.

### 5. Changing socket permissions incorrectly

**What happens**: Kubelet cannot communicate with the runtime, breaking all pods.

**Why**: Trying to harden by restricting socket permissions without understanding required access.

**How to avoid**: Check the default permissions and required group (usually `kubelet` or `root`). If in doubt, leave as default and rely on filesystem permissions and SELinux/AppArmor.

### 6. Neglecting to update the kubelet configuration

**What happens**: Runtime changes may require corresponding kubelet flags (e.g., cgroup driver).

**Why**: Focusing only on the runtime config.

**How to avoid**: Coordinate changes across runtime and kubelet. Review kubelet logs after restart (`journalctl -u kubelet`).

## Conclusion

Hardening Kubernetes container runtimes is a critical step in securing your cluster. By methodically inventorying your environment, making scoped configuration changes, verifying results, and having recovery plans, you reduce the risk of security incidents and operational disruptions. Remember to test changes thoroughly, monitor continuously, and keep your runtime versions up to date.

Start with one low-risk hardening item from this guide, such as enabling `NoNewPrivileges` in containerd or setting a default seccomp profile in CRI-O. Record the current state, apply the change on a test node, and verify with the provided commands. Document everything and schedule regular reviews. Over time, these practices will become second nature, leading to a more secure and resilient Kubernetes environment.