## Intro

When a Pod fails with `CreateContainerConfigError` or a container exits immediately with `operation not permitted`, the cause is often hidden in the Pod's `securityContext`. This guide turns those opaque failures into a repeatable troubleshooting process. It is written for developers, platform engineers, DevOps consultants, and anyone who operates Kubernetes clusters and wants to resolve security context issues without guesswork.

The article walks through realistic failure scenarios, shows the exact commands to diagnose them, and provides minimal fixes that keep the cluster secure. Every recovery step is reversible where Kubernetes allows it, and each section includes a verification command so you know the problem is actually solved.

Before making changes, always observe the current state first: check the Kubernetes version, the Pod's security configuration, and the node's security policies. Capture timestamps and command output. Limit any change to a single scoped field, and have a rollback plan ready. That operational discipline avoids turning a small misconfiguration into a wider outage.

## Version and Environment Inventory

Start by noting the Kubernetes version, container runtime, and the node's operating system. Security context behavior differs between versions, especially around seccomp and AppArmor. Use the following read-only commands to gather details:

```bash
kubectl version --short
kubectl get nodes -o wide
kubectl describe node <node-name> | grep -A5 "System Info"
```

Expected output for `kubectl version --short` looks like:

```text
Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.2
```

If the server version is older than v1.19, the `seccompProfile` field in the Pod's security context may not be available as a stable API. That immediately narrows the troubleshooting path.

Next, get an overview of all Pods and their security-relevant fields:

```bash
kubectl get pods -o custom-columns='NAME:.metadata.name,STATUS:.status.phase,RUNASUSER:.spec.containers[*].securityContext.runAsUser,RUNASGROUP:.spec.containers[*].securityContext.runAsGroup,PRIVILEGED:.spec.containers[*].securityContext.privileged'
```

Sample output:

```text
NAME                     STATUS    RUNASUSER   RUNASGROUP   PRIVILEGED
nginx-6799fc88d8-abcde   Running   1000        <none>       false
api-server-7b58f5c9-x    Error     <none>      <none>       false
```

The second Pod is in `Error` state and has no `runAsUser` set. That is a clue but not yet a diagnosis.

For a specific failing Pod, inspect its events:

```bash
kubectl describe pod api-server-7b58f5c9-x
```

Look for events like:

```text
Warning  Failed     5s (x2 over 10s)   kubelet            Error: container has runAsNonRoot and image will run as root
```

That event tells you exactly which security context field is conflicting. Before changing anything, capture the current manifest:

```bash
kubectl get pod api-server-7b58f5c9-x -o yaml > failing-pod-backup.yaml
```

Now you have a baseline and a backup. The next step is to isolate the smallest change that fixes the issue without weakening security.

## Safe Configuration Path

The safe path for changing a security context is to test changes in a non-production namespace first, or on a single Pod with a canary label. Never edit a running Deployment's Pod template directly without testing; instead, create a copy of the manifest, modify it, and apply it to a test namespace.

Here is a minimal Pod that runs as a non-root user:

```yaml
apiVersion: v1
kind: Pod
metadata:
  name: security-test
  namespace: test
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    runAsGroup: 3000
    fsGroup: 2000
  containers:
  - name: app
    image: nginx:1.25
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop: ["ALL"]
      readOnlyRootFilesystem: true
```

Apply this to the test namespace, then verify the Pod is running:

```bash
kubectl apply -f security-test.yaml
kubectl get pod security-test -n test -o wide
```

If the Pod runs successfully, the security context fields are acceptable to the cluster. If it fails, describe it to see the exact event:

```bash
kubectl describe pod security-test -n test
```

Common failures include:

- `runAsNonRoot` is true but the image's default user is root (UID 0). The container runtime refuses to start. Fix: specify `runAsUser` to a non-zero UID that exists in the image.
- `readOnlyRootFilesystem: true` causes the application to fail when it tries to write to `/tmp` or `/var/log`. Fix: mount an `emptyDir` volume at those paths, or adjust the application to write elsewhere.
- Dropping all capabilities with `drop: ["ALL"]` breaks applications that need `NET_BIND_SERVICE` to bind to ports below 1024. Fix: add back only the required capabilities using `add: ["NET_BIND_SERVICE"]`.

Always try the smallest change first. For example, if the container fails because of `readOnlyRootFilesystem`, do not also change `runAsUser`; change one field, test, and then proceed.

## Verification and Diagnostics

Diagnosing security context issues requires inspecting the container's actual runtime state, not just the Pod's spec. The following commands provide deep visibility.

**Check the Pod's effective security settings** by querying the API server with `kubectl get pod -o yaml`; the `status` section may include a `containerStatuses` entry with a `lastState` or `reason`.

**View container logs**, including the previous crashed instance:

```bash
kubectl logs security-test -n test --previous
```

If the container never started, logs may be empty. In that case, use `kubectl describe pod` to read the events.

**Inspect the container runtime directly** if you have node access. For containerd:

```bash
crictl ps -a | grep security-test
crictl inspect <container-id> | jq '.info.runtimeSpec.linux.securityContext'
```

The output shows the exact Linux security context applied to the container, including `runAsUser`, `capabilities`, and `seccomp`.

**Check seccomp profile** applied to the container:

```bash
crictl inspect <container-id> | jq '.info.runtimeSpec.linux.seccomp'
```

If the profile is `localhost/default` but the node does not have that profile file, the container fails with `cannot load seccomp profile`. The fix is to use `RuntimeDefault` or provide the profile.

**Validate AppArmor or SELinux** profiles when they are used. For SELinux, check the Pod's security context `seLinuxOptions` and the node's current enforcing mode:

```bash
getenforce
```

If the node is in enforcing mode and the container's SELinux type is not permitted, the container will be denied. You can temporarily set permissive mode for testing (with caution), then refine the policy.

**Check for Pod Security Admission** warnings. If the namespace enforces a restricted policy, a Pod that violates it may be rejected or warned. View the namespace labels:

```bash
kubectl get namespace test -o yaml | grep pod-security
```

If `pod-security.kubernetes.io/enforce=restricted` is set, then `runAsNonRoot: true` is mandatory, and capabilities must be dropped. Adjust the Pod's security context to comply.

A structured diagnostic table can help organize findings. Below is an example of what to record:

| Check | Command | Expected | Observed | Next Step |
|---|---|---|---|---|
| Pod status | `kubectl get pod` | Running | Error | Describe Pod, check events |
| Events | `kubectl describe pod` | No warnings | `runAsNonRoot` conflict | Set `runAsUser` to non-zero |
| Logs | `kubectl logs --previous` | App started | Permission denied on write | Mount emptyDir or adjust permissions |
| Seccomp | `crictl inspect` | RuntimeDefault | localhost/default missing file | Change seccompProfile to RuntimeDefault |
| SELinux | `getenforce` | Enforcing | Enforcing | Check seLinuxOptions in Pod spec |

Fill in actual observed values each time; the table above is illustrative.

## Failure Modes and Recovery

Several failure modes are common with security contexts. Understanding them speeds recovery.

### Failure 1: Container exits with `operation not permitted`

**Cause**: The container attempts a privileged operation that is blocked by the security context, such as changing file ownership, binding to a low port, or using a kernel module.

**Diagnosis**: Look at the container logs and exit code. For example:

```text
Error: listen EACCES: permission denied 0.0.0.0:80
```

That indicates the container does not have `NET_BIND_SERVICE` capability and tries to bind to port 80.

**Recovery**: Add the capability explicitly in the container's security context:

```yaml
securityContext:
  capabilities:
    add: ["NET_BIND_SERVICE"]
```

Alternatively, change the application to bind to a port above 1024, e.g., 8080. That is more secure.

### Failure 2: Pod stuck in `CreateContainerConfigError`

**Cause**: The security context references a non-existent seccomp profile, AppArmor profile, or SELinux label.

**Diagnosis**: `kubectl describe pod` shows an event like:

```text
Error: cannot load seccomp profile "/var/lib/kubelet/seccomp/custom.json": no such file or directory
```

**Recovery**: Either provide the profile file on the node, or change the Pod spec to use `seccompProfile: type: RuntimeDefault`.

### Failure 3: Pod is blocked by Pod Security Admission

**Cause**: The namespace enforces a policy level (baseline or restricted) that the Pod violates.

**Diagnosis**: `kubectl describe pod` may show a warning event, or the Pod is not created at all. Check the namespace labels:

```bash
kubectl get namespace myapp -o jsonpath='{.metadata.labels}'
```

If `pod-security.kubernetes.io/enforce=restricted` is set, then the Pod must have `runAsNonRoot: true`, drop all capabilities, and set `seccompProfile` to `RuntimeDefault`.

**Recovery**: Either modify the Pod to meet the policy, or change the namespace enforcement to a lower level if business requires (not recommended for production).

### Failure 4: Volume permission issues with `fsGroup`

**Cause**: The Pod has `fsGroup` set, but the volume type does not support it, or the group ID does not exist in the container's `/etc/group`.

**Diagnosis**: Pod logs show `mkdir: cannot create directory '/data': Permission denied`.

**Recovery**: Ensure the volume is a type that supports `fsGroup` (e.g., `emptyDir`, most CSI volumes). If the application creates files that need group access, set `fsGroup` to a GID that exists in the image or use an `initContainer` to adjust permissions.

Each recovery should be followed by a verification step. For example, after adding `NET_BIND_SERVICE`, restart the Pod and check that the application is reachable:

```bash
kubectl port-forward pod/myapp 8080:80
curl localhost:8080
```

If you must roll back, apply the backup manifest:

```bash
kubectl apply -f failing-pod-backup.yaml
```

## Operations Checklist

Use this checklist to standardize security context troubleshooting. Assign a single owner to each item, and revisit the checklist quarterly or after any cluster upgrade.

- **Before any change**: Capture the current Pod manifest with `kubectl get pod <name> -o yaml > backup.yaml`. Owner: On-call engineer.
- **Version check**: Record Kubernetes server version using `kubectl version --short`. Ensure seccomp and other features are supported. Owner: Platform lead.
- **Read-only diagnostics**: Run `kubectl describe pod`, `kubectl logs --previous`, and `kubectl get events`. Document findings in a shared ticket. Owner: Troubleshooting engineer.
- **Isolate change**: Modify one security context field at a time in a test namespace. Owner: Developer.
- **Verify**: After applying the fix, verify the Pod reaches `Running` and the application responds correctly. Use `kubectl port-forward` and curl. Owner: QA engineer or developer.
- **Rollback plan**: Keep the backup manifest and a written rollback command in the ticket. Owner: On-call engineer.
- **Policy review**: If the issue was caused by Pod Security Admission, review the namespace policy and decide whether to adjust the Pod or the policy. Owner: Security officer and platform lead, reviewed monthly.
- **Documentation**: Update runbooks or team docs with the failure mode and its fix. Owner: Tech writer or DevOps lead.
- **Post-incident review**: For high-severity issues, schedule a blameless post-mortem within 5 business days. Owner: Engineering manager.

## Common Pitfalls

Here are pitfalls that frequently trip up operators when working with security contexts, along with how to avoid them.

### Pitfall 1: Setting `runAsNonRoot: true` without verifying the image's default user

Many images run as root by default. When you add `runAsNonRoot: true`, the container fails immediately with `Error: container has runAsNonRoot and image will run as root`. Avoid this by first checking the image's user with `docker inspect` or by reading its documentation. Always specify `runAsUser` to a hardened UID.

### Pitfall 2: Dropping all capabilities without testing the application

`capabilities: drop: ["ALL"]` is a security best practice, but many applications need at least one capability. For example, a web server binding to port 80 needs `NET_BIND_SERVICE`. Test the application with all capabilities dropped in a staging environment, and only add back the exact capabilities that are required.

### Pitfall 3: Using `readOnlyRootFilesystem: true` without providing writable volumes

Applications often need to write temporary files, logs, or caches. With a read-only root filesystem, they fail with `Permission denied`. Plan ahead: mount `emptyDir` volumes to `/tmp`, `/var/log`, and other writable paths. Use an `initContainer` if the application needs to pre-populate directories.

### Pitfall 4: Misunderstanding seccomp profile types

Setting `seccompProfile: type: Localhost` requires the profile file to be present on the node at a specific path under `/var/lib/kubelet/seccomp/`. If the file is missing, the Pod fails with `CreateContainerConfigError`. Using `RuntimeDefault` is safer and works everywhere. Only use custom profiles when you control the node filesystem and have a rollout process.

### Pitfall 5: Using a container image that does not include the specified `runAsUser` UID

If you set `runAsUser: 1000` but the image's `/etc/passwd` has no UID 1000, the container still runs with UID 1000 (the kernel does not need the user to exist), but the application may encounter permission problems when accessing files owned by other users. Ensure the image is built with the expected user, or use `fsGroup` and `runAsGroup` to align file ownership.

### Pitfall 6: Applying security contexts only at the Pod level and forgetting container-level overrides

A Pod-level `securityContext` sets defaults for all containers, but each container can override it with its own `securityContext`. If one container needs different settings, verify that the container-level fields are correct. A common mistake is to set `runAsUser` at the Pod level but then override it inadvertently in one container.

## Conclusion

Troubleshooting Kubernetes security contexts is a methodical process, not a guessing game. By following the sequence in this guide, you can resolve most failures quickly without weakening the cluster's security posture. Always observe before changing, make one scoped change at a time, verify the result, and have a rollback plan.

The next time you face a `CreateContainerConfigError` or an `operation not permitted` error, start with the version inventory and read-only diagnostics, then apply the minimal fix from the failure modes section. Keep this guide as a field reference, and update it as your clusters evolve.

Reliable Kubernetes operations demand that every security decision is intentional. Document your security contexts, test them in staging, and review them regularly. That discipline turns security from a source of incidents into a predictable part of your deployment pipeline.