Kubernetes Container Runtimes Security Hardening: A Practical Guide

Intro

Container runtimes are the software that actually run your containers on each Kubernetes node. If a runtime is misconfigured, every workload on that node is at risk. Hardening the runtime is therefore a foundational security task for any Kubernetes cluster. This guide focuses on the two most common Kubernetes container runtimes today: containerd (used by Docker, GKE, AKS, EKS, and many others) and CRI-O (used by OpenShift and some RHEL-based distributions). We will walk through version and environment inventory, safe configuration changes, verification, failure recovery, and provide checklists and common pitfalls.

We will use concrete commands and configuration snippets that you can adapt to your environment. Always test in a staging cluster first and follow your change management process. The goal is to make changes that are observable, reversible, and scoped to the intended resource.

Version and Environment Inventory

Before making any change, know exactly what runtime you are running, its version, and how it is configured. This information is essential because security features and configuration options vary by version and runtime type.

Identify the runtime and version

On each node, check which runtime is active. You can do this via the Kubernetes API or directly on the node.

Using kubectl, list nodes and their container runtime version:

kubectl get nodes -o wide

Look at the CONTAINER-RUNTIME column. It will show something like containerd://1.6.18 or cri-o://1.26.0.

Alternatively, SSH into the node and run the runtime's version command directly:

• For containerd: containerd --version and ctr version
• For CRI-O: crio --version

Record the exact version and the Kubernetes version (kubectl version --short). Check the runtime's release notes and security advisories for your version. For containerd, known vulnerabilities are tracked on its GitHub security page; for CRI-O, check the Red Hat or upstream advisories.

Inspect current configuration

Both runtimes use a configuration file. The default locations are:

• containerd: /etc/containerd/config.toml
• CRI-O: /etc/crio/crio.conf (and drop-in files in /etc/crio/crio.conf.d/)

View the current configuration (read-only) to understand what is already set. For containerd, you can print the default configuration (which may differ from what's active) with:

containerd config default > /tmp/containerd-default.toml
diff /tmp/containerd-default.toml /etc/containerd/config.toml

This shows you if any customizations have been made. For CRI-O, use crio --config /etc/crio/crio.conf config to dump the effective configuration.

Capture the current state and a timestamp. This serves as a baseline for rollback and troubleshooting. Store this information securely, as configuration may contain sensitive data (like registry credentials or certificate paths).

Check runtime dependencies

The runtime depends on the kernel, container images, and the kubelet. Ensure the operating system and kernel are up to date with security patches. For containerd, check that the CNI plugins are installed and compatible. For CRI-O, verify that conmon and runc (or crun) versions are supported.

Use the following commands to check key binaries:

runc --version
conmon --version   # CRI-O only
cni-plugins --version  # if available

Blast radius assessment

Before changing runtime configuration, understand which workloads run on the node. A runtime restart will affect all pods on that node. Use:

kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=<node-name>

Plan for a maintenance window if the node hosts critical workloads. Ensure you can drain the node safely if needed.

Safe Configuration Path

Now that you have a baseline, plan and execute configuration changes carefully. The following subsections cover key hardening areas for both runtimes.

General hardening principles

• Use a dedicated user for the runtime: Run the runtime as a non-root user (e.g., containerd or crio). Check the service file (/etc/systemd/system/containerd.service or /etc/systemd/system/crio.service) for the User= directive. If it is missing or set to root, add it.
• Restrict runtime socket permissions: The runtime's gRPC socket (e.g., /run/containerd/containerd.sock) should be accessible only to root and the kubelet. Check permissions with ls -l /run/containerd/containerd.sock; it should be srw------- owned by root. If not, adjust the socket path or permissions.
• Enable seccomp: Seccomp restricts the system calls a container can make. Ensure the runtime is configured to use a default seccomp profile. For containerd, in the plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options section, you can set NoNewPrivileges = true and ensure the default seccomp profile is applied. For CRI-O, set seccomp_profile = "/path/to/profile.json" in the crio.runtime section, or ensure the default profile is used.
• Use appropriate capabilities: Limit capabilities granted to containers. By default, runtimes drop many capabilities already; do not add back unnecessary ones. Verify the default set with capsh --print inside a test container.

containerd-specific configuration

Edit /etc/containerd/config.toml. After making changes, validate and restart the service.

Example: enable CRI plugin and set default runtime.

In the config, ensure the CRI plugin is enabled:

version = 2

[plugins."io.containerd.grpc.v1.cri"]
  sandbox_image = "registry.k8s.io/pause:3.9"

[plugins."io.containerd.grpc.v1.cri".containerd]
  default_runtime_name = "runc"
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
    runtime_type = "io.containerd.runc.v2"
    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
      NoNewPrivileges = true
      SystemdCgroup = true

By default, NoNewPrivileges is false. Setting it to true ensures containers cannot gain additional privileges via setuid binaries. This is a recommended hardening step.

Example: set seccomp profile for containerd

Containerd CRI plugin applies the Kubernetes default seccomp profile automatically when requested via the pod security context, but you can also set a default at the runtime level. In the runc.options section, add:

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
  NoNewPrivileges = true
  SystemdCgroup = true
  # Seccomp profile can be specified via annotation or pod spec.
  # To enforce a default, you could use a mutating webhook.

For more advanced seccomp defaults, consider using Kubernetes pod security standards or a policy engine.

After editing, validate:

containerd config dump > /tmp/containerd-new.toml
containerd config verify < /tmp/containerd-new.toml

If no errors, restart containerd:

sudo systemctl restart containerd

Check the service status:

sudo systemctl status containerd

CRI-O-specific configuration

Edit /etc/crio/crio.conf or add a drop-in file under /etc/crio/crio.conf.d/.

Example: Set default capabilities drop and seccomp

In the crio.runtime section:

[crio.runtime]
default_capabilities = [
  "CHOWN",
  "DAC_OVERRIDE",
  "FSETID",
  "FOWNER",
  "SETGID",
  "SETUID",
  "SETPCAP",
  "NET_BIND_SERVICE",
  "KILL",
]
seccomp_profile = "/etc/crio/seccomp.json"

Ensure the seccomp_profile points to a valid seccomp profile file. You can copy the default from Kubernetes or create a custom one.

Also, ensure conmon and runc are secure:

[crio.runtime]
conmon = "/usr/bin/conmon"
conmon_cgroup = "pod"

After changes, validate:

crio config validate

Restart CRI-O:

sudo systemctl restart crio

Applying configuration changes safely

Always test on a single node first. Drain the node to avoid disruption:

kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

Make your configuration change, restart the runtime, and uncordon the node:

kubectl uncordon <node-name>

Monitor pod health on that node before proceeding to other nodes.

Verification and Diagnostics

After any configuration change, verify that the runtime is functioning correctly and that security controls are enforced.

Verify runtime is running

Check the service status:

systemctl status containerd   # or crio

Look for errors in the logs:

journalctl -u containerd -n 100 --no-pager

For CRI-O:

journalctl -u crio -n 100 --no-pager

Test container operations

Run a simple pod on the node (or cluster) and ensure it starts:

kubectl run test-pod --image=busybox --restart=Never --command -- sleep 3600
kubectl get pod test-pod

Check that the pod is running. Then exec into it to test security features:

kubectl exec -it test-pod -- sh

Inside the container, test that setuid binaries are restricted if NoNewPrivileges is set:

# Try to gain root
su -

This should fail if the container is running as non-root and NoNewPrivileges is true. If you are root, test whether you can change user to a higher privilege (you can't).

Test seccomp: attempt to use a syscall not in the default profile, e.g., reboot or mount with unusual arguments. In a secure runtime, these should be denied.

Example to test seccomp with a custom profile: create a pod with a restrictive seccomp profile and try to run a disallowed syscall.

Check pod security context

Ensure pods are running with restricted security contexts. Use:

kubectl get pod test-pod -o yaml | grep -A 10 securityContext

Look for runAsNonRoot: true, allowPrivilegeEscalation: false, seccompProfile: RuntimeDefault, etc. These should be set by your pod policies or admission controllers.

Monitor runtime metrics

Both runtimes expose metrics. For containerd, you can query the metrics endpoint if enabled (usually at http://localhost:1338/metrics). For CRI-O, metrics are at http://localhost:9090/metrics. Use curl to check metrics related to containers and errors.

Example:

curl -s http://localhost:1338/metrics | grep -E 'container_|runtime_'

If metrics are not enabled, consider enabling them for monitoring in production.

Failure Modes and Recovery

Despite careful planning, failures can occur. This section covers common failure modes and how to recover.

Runtime fails to start after configuration change

Symptom: systemctl status containerd shows failed, and journalctl -u containerd shows errors like "invalid config" or "failed to load config".

Cause: Syntax error in config file, unsupported option, or file permissions issue.

Recovery: Roll back to the previous configuration.

If you made a backup (you should), restore it:

sudo cp /etc/containerd/config.toml.bak /etc/containerd/config.toml
sudo systemctl restart containerd

If no backup, you can use the default config generated earlier:

containerd config default > /etc/containerd/config.toml
sudo systemctl restart containerd

Then reapply changes one by one, validating after each.

Pods cannot start after runtime restart

Symptom: New pods stay in ContainerCreating state, and kubectl describe pod shows errors like "failed to create containerd task" or "rpc error: code = Unknown desc = failed to start container".

Cause: Could be missing CNI plugins, incorrect sandbox image, or seccomp profile path incorrect.

Recovery: Check runtime logs for detailed error. If CNI issue, verify network plugin config (/etc/cni/net.d/). If sandbox image issue, ensure the pause image is correct and pullable (registry.k8s.io/pause:3.9).

If seccomp profile path is wrong, fix it or remove the seccomp setting and restart.

Node becomes NotReady

Symptom: kubectl get nodes shows node as NotReady after runtime restart.

Cause: Kubelet lost connection to runtime, perhaps due to socket permission change.

Recovery: Check kubelet logs:

journalctl -u kubelet -n 100

If socket permission was changed incorrectly, restore original permissions (e.g., chmod 660 for containerd socket, owner root:kubelet). Then restart kubelet.

Rollback strategy

Always maintain versioned backups of configuration files. Use etckeeper or manual backups with timestamps. Before a change, copy the config:

sudo cp /etc/containerd/config.toml /etc/containerd/config.toml.$(date +%Y%m%d%H%M%S)

If all else fails, you can reinstall the runtime package (this will restore default configs but you will lose custom settings) and then reapply your hardening from documentation.

Operations Checklist

Use this checklist before and after any runtime hardening change.

Pre-change checklist

• [ ] Identify runtime type and version: kubectl get nodes -o wide and record.
• [ ] Read the runtime's security advisories and release notes for your version.
• [ ] Back up current configuration files with timestamps.
• [ ] Document current state (config dump, runtime status, metrics baseline).
• [ ] Identify workloads on target node(s) and plan maintenance window.
• [ ] Drain node(s) if required: kubectl drain <node> --ignore-daemonsets.
• [ ] Prepare rollback plan and rollback commands.
• [ ] Ensure monitoring and alerting are active.
• [ ] Designate an owner for this change: Priya Shah, Engineering Lead (or your equivalent).

Post-change verification checklist

• [ ] Runtime service is active: systemctl status containerd (or crio).
• [ ] Runtime logs show no fatal errors: journalctl -u containerd -n 50.
• [ ] Test pod starts and runs: kubectl run test-pod --image=busybox --command -- sleep 60.
• [ ] Test pod respects security context: check kubectl get pod test-pod -o yaml | grep securityContext.
• [ ] Node is Ready: kubectl get nodes.
• [ ] Existing pods on node are still healthy (if not drained).
• [ ] If drained, uncordon node: kubectl uncordon <node>.
• [ ] Clean up test pod: kubectl delete pod test-pod.
• [ ] Update documentation and configuration management repository.
• [ ] Schedule a post-change review within one week to catch delayed issues.

Recurring maintenance

• Monthly: Review runtime versions against security advisories and plan upgrades.
• Quarterly: Audit runtime configuration for compliance with hardening benchmarks (CIS, STIG).
• Continuously: Monitor runtime metrics and logs for anomalies.

Owner for recurring checks: Alex Chen, Platform Security Engineer (or your equivalent). The owner reviews and updates this checklist quarterly.

Common Pitfalls

Avoid these common mistakes when hardening container runtimes.

1. Not testing on a single node first

What happens: You apply a configuration change across all nodes simultaneously, causing a cluster-wide outage.

Why: Overconfidence or time pressure.

How to avoid: Always start with one non-critical node. Drain it, make the change, verify, and then roll out gradually.

2. Ignoring version compatibility

What happens: You apply configuration options that do not exist in your runtime version, leading to startup failure.

Why: Copying config snippets from the internet without checking the version.

How to avoid: Always check the documentation for your specific runtime version. Use containerd config default or crio config dump to see the current supported options.

3. Over-restricting seccomp or capabilities

What happens: Applications fail because they need a syscall or capability that was removed.

Why: Assuming a generic profile fits all workloads.

How to avoid: Start with the default seccomp profile and only remove capabilities if justified. Use audit logs to identify missing syscalls, then customize profiles.

4. Forgetting to backup and test rollback

What happens: When a change fails, you cannot quickly revert, extending downtime.

Why: Not treating config as code with version control.

How to avoid: Always backup config files and store them in a version control system. Actually test the rollback procedure in a staging environment.

5. Changing socket permissions incorrectly

What happens: Kubelet cannot communicate with the runtime, breaking all pods.

Why: Trying to harden by restricting socket permissions without understanding required access.

How to avoid: Check the default permissions and required group (usually kubelet or root). If in doubt, leave as default and rely on filesystem permissions and SELinux/AppArmor.

6. Neglecting to update the kubelet configuration

What happens: Runtime changes may require corresponding kubelet flags (e.g., cgroup driver).

Why: Focusing only on the runtime config.

How to avoid: Coordinate changes across runtime and kubelet. Review kubelet logs after restart (journalctl -u kubelet).

Conclusion

Hardening Kubernetes container runtimes is a critical step in securing your cluster. By methodically inventorying your environment, making scoped configuration changes, verifying results, and having recovery plans, you reduce the risk of security incidents and operational disruptions. Remember to test changes thoroughly, monitor continuously, and keep your runtime versions up to date.

Start with one low-risk hardening item from this guide, such as enabling NoNewPrivileges in containerd or setting a default seccomp profile in CRI-O. Record the current state, apply the change on a test node, and verify with the provided commands. Document everything and schedule regular reviews. Over time, these practices will become second nature, leading to a more secure and resilient Kubernetes environment.