Kubernetes Security Context Troubleshooting: A Practical Field Guide

Intro

When a Pod fails with CreateContainerConfigError or a container exits immediately with operation not permitted, the cause is often hidden in the Pod's securityContext. This guide turns those opaque failures into a repeatable troubleshooting process. It is written for developers, platform engineers, DevOps consultants, and anyone who operates Kubernetes clusters and wants to resolve security context issues without guesswork.

The article walks through realistic failure scenarios, shows the exact commands to diagnose them, and provides minimal fixes that keep the cluster secure. Every recovery step is reversible where Kubernetes allows it, and each section includes a verification command so you know the problem is actually solved.

Before making changes, always observe the current state first: check the Kubernetes version, the Pod's security configuration, and the node's security policies. Capture timestamps and command output. Limit any change to a single scoped field, and have a rollback plan ready. That operational discipline avoids turning a small misconfiguration into a wider outage.

Version and Environment Inventory

Start by noting the Kubernetes version, container runtime, and the node's operating system. Security context behavior differs between versions, especially around seccomp and AppArmor. Use the following read-only commands to gather details:

kubectl version --short
kubectl get nodes -o wide
kubectl describe node <node-name> | grep -A5 "System Info"

Expected output for kubectl version --short looks like:

Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.2

If the server version is older than v1.19, the seccompProfile field in the Pod's security context may not be available as a stable API. That immediately narrows the troubleshooting path.

Next, get an overview of all Pods and their security-relevant fields:

kubectl get pods -o custom-columns='NAME:.metadata.name,STATUS:.status.phase,RUNASUSER:.spec.containers[*].securityContext.runAsUser,RUNASGROUP:.spec.containers[*].securityContext.runAsGroup,PRIVILEGED:.spec.containers[*].securityContext.privileged'

Sample output:

NAME                     STATUS    RUNASUSER   RUNASGROUP   PRIVILEGED
nginx-6799fc88d8-abcde   Running   1000        <none>       false
api-server-7b58f5c9-x    Error     <none>      <none>       false

The second Pod is in Error state and has no runAsUser set. That is a clue but not yet a diagnosis.

For a specific failing Pod, inspect its events:

kubectl describe pod api-server-7b58f5c9-x

Look for events like:

Warning  Failed     5s (x2 over 10s)   kubelet            Error: container has runAsNonRoot and image will run as root

That event tells you exactly which security context field is conflicting. Before changing anything, capture the current manifest:

kubectl get pod api-server-7b58f5c9-x -o yaml > failing-pod-backup.yaml

Now you have a baseline and a backup. The next step is to isolate the smallest change that fixes the issue without weakening security.

Safe Configuration Path

The safe path for changing a security context is to test changes in a non-production namespace first, or on a single Pod with a canary label. Never edit a running Deployment's Pod template directly without testing; instead, create a copy of the manifest, modify it, and apply it to a test namespace.

Here is a minimal Pod that runs as a non-root user:

apiVersion: v1
kind: Pod
metadata:
  name: security-test
  namespace: test
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    runAsGroup: 3000
    fsGroup: 2000
  containers:
  - name: app
    image: nginx:1.25
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop: ["ALL"]
      readOnlyRootFilesystem: true

Apply this to the test namespace, then verify the Pod is running:

kubectl apply -f security-test.yaml
kubectl get pod security-test -n test -o wide

If the Pod runs successfully, the security context fields are acceptable to the cluster. If it fails, describe it to see the exact event:

kubectl describe pod security-test -n test

Common failures include:

• runAsNonRoot is true but the image's default user is root (UID 0). The container runtime refuses to start. Fix: specify runAsUser to a non-zero UID that exists in the image.
• readOnlyRootFilesystem: true causes the application to fail when it tries to write to /tmp or /var/log. Fix: mount an emptyDir volume at those paths, or adjust the application to write elsewhere.
• Dropping all capabilities with drop: ["ALL"] breaks applications that need NET_BIND_SERVICE to bind to ports below 1024. Fix: add back only the required capabilities using add: ["NET_BIND_SERVICE"].

Always try the smallest change first. For example, if the container fails because of readOnlyRootFilesystem, do not also change runAsUser; change one field, test, and then proceed.

Verification and Diagnostics

Diagnosing security context issues requires inspecting the container's actual runtime state, not just the Pod's spec. The following commands provide deep visibility.

Check the Pod's effective security settings by querying the API server with kubectl get pod -o yaml; the status section may include a containerStatuses entry with a lastState or reason.

View container logs, including the previous crashed instance:

kubectl logs security-test -n test --previous

If the container never started, logs may be empty. In that case, use kubectl describe pod to read the events.

Inspect the container runtime directly if you have node access. For containerd:

crictl ps -a | grep security-test
crictl inspect <container-id> | jq '.info.runtimeSpec.linux.securityContext'

The output shows the exact Linux security context applied to the container, including runAsUser, capabilities, and seccomp.

Check seccomp profile applied to the container:

crictl inspect <container-id> | jq '.info.runtimeSpec.linux.seccomp'

If the profile is localhost/default but the node does not have that profile file, the container fails with cannot load seccomp profile. The fix is to use RuntimeDefault or provide the profile.

Validate AppArmor or SELinux profiles when they are used. For SELinux, check the Pod's security context seLinuxOptions and the node's current enforcing mode:

getenforce

If the node is in enforcing mode and the container's SELinux type is not permitted, the container will be denied. You can temporarily set permissive mode for testing (with caution), then refine the policy.

Check for Pod Security Admission warnings. If the namespace enforces a restricted policy, a Pod that violates it may be rejected or warned. View the namespace labels:

kubectl get namespace test -o yaml | grep pod-security

If pod-security.kubernetes.io/enforce=restricted is set, then runAsNonRoot: true is mandatory, and capabilities must be dropped. Adjust the Pod's security context to comply.

A structured diagnostic table can help organize findings. Below is an example of what to record:

Fill in actual observed values each time; the table above is illustrative.

Failure Modes and Recovery

Several failure modes are common with security contexts. Understanding them speeds recovery.

Failure 1: Container exits with operation not permitted

Cause: The container attempts a privileged operation that is blocked by the security context, such as changing file ownership, binding to a low port, or using a kernel module.

Diagnosis: Look at the container logs and exit code. For example:

Error: listen EACCES: permission denied 0.0.0.0:80

That indicates the container does not have NET_BIND_SERVICE capability and tries to bind to port 80.

Recovery: Add the capability explicitly in the container's security context:

securityContext:
  capabilities:
    add: ["NET_BIND_SERVICE"]

Alternatively, change the application to bind to a port above 1024, e.g., 8080. That is more secure.

Failure 2: Pod stuck in CreateContainerConfigError

Cause: The security context references a non-existent seccomp profile, AppArmor profile, or SELinux label.

Diagnosis: kubectl describe pod shows an event like:

Error: cannot load seccomp profile "/var/lib/kubelet/seccomp/custom.json": no such file or directory

Recovery: Either provide the profile file on the node, or change the Pod spec to use seccompProfile: type: RuntimeDefault.

Failure 3: Pod is blocked by Pod Security Admission

Cause: The namespace enforces a policy level (baseline or restricted) that the Pod violates.

Diagnosis: kubectl describe pod may show a warning event, or the Pod is not created at all. Check the namespace labels:

kubectl get namespace myapp -o jsonpath='{.metadata.labels}'

If pod-security.kubernetes.io/enforce=restricted is set, then the Pod must have runAsNonRoot: true, drop all capabilities, and set seccompProfile to RuntimeDefault.

Recovery: Either modify the Pod to meet the policy, or change the namespace enforcement to a lower level if business requires (not recommended for production).

Failure 4: Volume permission issues with fsGroup

Cause: The Pod has fsGroup set, but the volume type does not support it, or the group ID does not exist in the container's /etc/group.

Diagnosis: Pod logs show mkdir: cannot create directory '/data': Permission denied.

Recovery: Ensure the volume is a type that supports fsGroup (e.g., emptyDir, most CSI volumes). If the application creates files that need group access, set fsGroup to a GID that exists in the image or use an initContainer to adjust permissions.

Each recovery should be followed by a verification step. For example, after adding NET_BIND_SERVICE, restart the Pod and check that the application is reachable:

kubectl port-forward pod/myapp 8080:80
curl localhost:8080

If you must roll back, apply the backup manifest:

kubectl apply -f failing-pod-backup.yaml

Operations Checklist

Use this checklist to standardize security context troubleshooting. Assign a single owner to each item, and revisit the checklist quarterly or after any cluster upgrade.

• Before any change: Capture the current Pod manifest with kubectl get pod <name> -o yaml > backup.yaml. Owner: On-call engineer.
• Version check: Record Kubernetes server version using kubectl version --short. Ensure seccomp and other features are supported. Owner: Platform lead.
• Read-only diagnostics: Run kubectl describe pod, kubectl logs --previous, and kubectl get events. Document findings in a shared ticket. Owner: Troubleshooting engineer.
• Isolate change: Modify one security context field at a time in a test namespace. Owner: Developer.
• Verify: After applying the fix, verify the Pod reaches Running and the application responds correctly. Use kubectl port-forward and curl. Owner: QA engineer or developer.
• Rollback plan: Keep the backup manifest and a written rollback command in the ticket. Owner: On-call engineer.
• Policy review: If the issue was caused by Pod Security Admission, review the namespace policy and decide whether to adjust the Pod or the policy. Owner: Security officer and platform lead, reviewed monthly.
• Documentation: Update runbooks or team docs with the failure mode and its fix. Owner: Tech writer or DevOps lead.
• Post-incident review: For high-severity issues, schedule a blameless post-mortem within 5 business days. Owner: Engineering manager.

Common Pitfalls

Here are pitfalls that frequently trip up operators when working with security contexts, along with how to avoid them.

Pitfall 1: Setting runAsNonRoot: true without verifying the image's default user

Many images run as root by default. When you add runAsNonRoot: true, the container fails immediately with Error: container has runAsNonRoot and image will run as root. Avoid this by first checking the image's user with docker inspect or by reading its documentation. Always specify runAsUser to a hardened UID.

Pitfall 2: Dropping all capabilities without testing the application

capabilities: drop: ["ALL"] is a security best practice, but many applications need at least one capability. For example, a web server binding to port 80 needs NET_BIND_SERVICE. Test the application with all capabilities dropped in a staging environment, and only add back the exact capabilities that are required.

Pitfall 3: Using readOnlyRootFilesystem: true without providing writable volumes

Applications often need to write temporary files, logs, or caches. With a read-only root filesystem, they fail with Permission denied. Plan ahead: mount emptyDir volumes to /tmp, /var/log, and other writable paths. Use an initContainer if the application needs to pre-populate directories.

Pitfall 4: Misunderstanding seccomp profile types

Setting seccompProfile: type: Localhost requires the profile file to be present on the node at a specific path under /var/lib/kubelet/seccomp/. If the file is missing, the Pod fails with CreateContainerConfigError. Using RuntimeDefault is safer and works everywhere. Only use custom profiles when you control the node filesystem and have a rollout process.

Pitfall 5: Using a container image that does not include the specified runAsUser UID

If you set runAsUser: 1000 but the image's /etc/passwd has no UID 1000, the container still runs with UID 1000 (the kernel does not need the user to exist), but the application may encounter permission problems when accessing files owned by other users. Ensure the image is built with the expected user, or use fsGroup and runAsGroup to align file ownership.

Pitfall 6: Applying security contexts only at the Pod level and forgetting container-level overrides

A Pod-level securityContext sets defaults for all containers, but each container can override it with its own securityContext. If one container needs different settings, verify that the container-level fields are correct. A common mistake is to set runAsUser at the Pod level but then override it inadvertently in one container.

Conclusion

Troubleshooting Kubernetes security contexts is a methodical process, not a guessing game. By following the sequence in this guide, you can resolve most failures quickly without weakening the cluster's security posture. Always observe before changing, make one scoped change at a time, verify the result, and have a rollback plan.

The next time you face a CreateContainerConfigError or an operation not permitted error, start with the version inventory and read-only diagnostics, then apply the minimal fix from the failure modes section. Keep this guide as a field reference, and update it as your clusters evolve.

Reliable Kubernetes operations demand that every security decision is intentional. Document your security contexts, test them in staging, and review them regularly. That discipline turns security from a source of incidents into a predictable part of your deployment pipeline.