## Intro Kubernetes Seccomp (Secure Computing Mode) is a Linux kernel feature that restricts the system calls a container can make, reducing the attack surface if a container is compromised. Managing seccomp profiles across a Kubernetes upgrade or migration is a delicate operation: a misconfigured profile can break application functionality or, worse, silently weaken security. This guide provides a practical, field-tested approach for Kubernetes Seccomp upgrade and migration, aimed at developers, DevOps consultants, and technical startup teams who need to move from an observed problem to a verified result. We will cover the entire lifecycle: version and environment inventory, safe configuration paths, verification and diagnostics, failure modes and recovery, and an operations checklist. Each section includes concrete commands, expected output, failure signals, and recovery decisions. Our goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached. ## Version and Environment Inventory Before any seccomp upgrade or migration, you must understand your current environment. This section provides a systematic inventory of versions, prerequisites, and current state. ### Identify Kubernetes and Container Runtime Versions Seccomp support and default behavior depend on the Kubernetes version and the container runtime. Run the following commands to capture your environment: kubectl version --short # Example output: # Client Version: v1.27.3 # Server Version: v1.27.3 For the container runtime (assuming containerd): kubectl get nodes -o wide # Example output: # NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME # worker-1 Ready 10d v1.27.3 10.0.0.4 Ubuntu 22.04.2 LTS 5.15.0-76-generic containerd://1.7.2 Note the container runtime version; seccomp defaulting was introduced in Kubernetes v1.25 and is stable in v1.27. If your nodes run an older Kubernetes or runtime without seccomp defaulting, your upgrade path will differ. ### Check Current Seccomp Usage Across Workloads To understand which pods are using seccomp profiles, inspect the security context of running pods. Use the following command to list pods with their seccomp annotations (if using the deprecated annotation method) or securityContext fields: kubectl get pods --all-namespaces -o json | jq -r '.items[] | select(.spec.securityContext.seccompProfile != null or .metadata.annotations."seccomp.security.alpha.kubernetes.io/pod" != null) | .metadata.namespace + "/" + .metadata.name' If you are using the deprecated alpha annotation, you must migrate to the seccompProfile field in the pod's security context. The annotation seccomp.security.alpha.kubernetes.io/pod was deprecated in v1.19 and removed in v1.25. ### Inventory Existing Seccomp Profiles Custom seccomp profiles are usually stored as files on the node or as ConfigMaps. To find ConfigMap-based profiles: kubectl get configmaps --all-namespaces -o json | jq -r '.items[] | select(.data | keys[] | test("seccomp")) | .metadata.namespace + "/" + .metadata.name' For node-local profiles, you would need to check the default seccomp profile directory, typically /var/lib/kubelet/seccomp/ . You can use a DaemonSet to inspect node files, but for inventory, it may be sufficient to know if any pods reference local profiles. ### Prerequisites for Upgrade - Kubernetes version : At least v1.25 for stable seccompProfile field and defaulting to RuntimeDefault . Verify with kubectl version . - Container runtime : Must support seccomp and the RuntimeDefault profile. containerd and CRI-O do; Docker Engine with the deprecations may require additional configuration. - RBAC permissions : You need get and list permissions on pods and configmaps across namespaces, and patch or update on workloads you intend to modify. - Test environment : A staging or development cluster with the same runtime and Kubernetes version to validate changes before production. ### Read-Only Observation Commands Before making any changes, record baseline behavior: # List all pods and their status kubectl get pods -A -o wide # Describe a specific pod to see events and security context kubectl describe pod -n # Check logs of a running container kubectl logs -n # Check logs of a crashed container kubectl logs -n --previous ### Smallest Justified Change For seccomp upgrade, the smallest justified change might be enabling RuntimeDefault on a single test pod that currently has Unconfined . This verifies that the runtime default seccomp profile does not break the application before you apply it broadly. Example manifest before change ( unconfined-pod.yaml ): apiVersion: v1 kind: Pod metadata: name: test-pod spec: containers: - name: app image: nginx:1.25 securityContext: seccompProfile: type: Unconfined After change ( runtime-default-pod.yaml ): apiVersion: v1 kind: Pod metadata: name: test-pod spec: containers: - name: app image: nginx:1.25 securityContext: seccompProfile: type: RuntimeDefault Apply and verify: kubectl apply -f runtime-default-pod.yaml kubectl get pod test-pod kubectl logs test-pod If the pod runs and logs show normal startup, you can proceed to scale the change. ### Blast Radius and Recovery Path Always test on a non-critical workload first. If the pod fails with seccomp violations (e.g., "operation not permitted" in logs), you can quickly rollback by reapplying the previous manifest with Unconfined . Document this recovery step before you make the change. ## Safe Configuration Path This section details how to configure seccomp profiles safely during an upgrade or migration. We cover both built-in profiles and custom profiles. ### Understanding Seccomp Profile Types Kubernetes supports three seccomp profile types in the seccompProfile field: - Unconfined : No seccomp restrictions (default if not specified in many older clusters, but not recommended). - RuntimeDefault : Uses the container runtime's default seccomp profile, which is generally safe for most applications. - Localhost : Uses a custom profile defined in a file on the node under the kubelet's seccomp profile root directory (usually /var/lib/kubelet/seccomp/ ). ### Migrating from Deprecated Annotations to Security Context Fields If your workloads still use the alpha annotation seccomp.security.alpha.kubernetes.io/pod , you need to migrate to the seccompProfile field. The annotation is interpreted as a path relative to the seccomp profile root, and it is equivalent to type: Localhost with localhostProfile: . Example old annotation: metadata: annotations: seccomp.security.alpha.kubernetes.io/pod: "localhost/profiles/audit.json" Equivalent new field: spec: securityContext: seccompProfile: type: Localhost localhostProfile: "profiles/audit.json" Note that the localhostProfile path should not include the "localhost/" prefix; it is relative to the seccomp profile root. To automate this migration, you can use kubectl with a JSON patch or a tool like kube-neat to rewrite manifests. For a large fleet, consider using a policy engine like Kyverno or OPA Gatekeeper to mutate pods on admission. Example kubectl patch for a deployment: kubectl patch deployment myapp -n production --type='json' -p='[{"op": "remove", "path": "/spec/template/metadata/annotations/seccomp.security.alpha.kubernetes.io~1pod"}, {"op": "add", "path": "/spec/template/spec/securityContext/seccompProfile", "value": {"type": "Localhost", "localhostProfile": "profiles/audit.json"}}]' Careful: The ~1 is the JSON pointer escape for / . Test this on a staging deployment first. ### Creating and Deploying Custom Seccomp Profiles Custom seccomp profiles are JSON files that define allowed syscalls, default action, and more. Here is a minimal custom profile that denies the chmod syscall and logs all others: custom-profile.json: { "defaultAction": "SCMP_ACT_LOG", "architectures": [ "SCMP_ARCH_X86_64", "SCMP_ARCH_X86", "SCMP_ARCH_X32" ], "syscalls": [ { "names": ["chmod"], "action": "SCMP_ACT_ERRNO" } ] } To deploy this profile to nodes, you need to place it in the seccomp profile directory on each node. The simplest secure method is to use a DaemonSet that copies the profile from a ConfigMap to the node's filesystem. Here is an example DaemonSet: apiVersion: apps/v1 kind: DaemonSet metadata: name: seccomp-profile-installer namespace: kube-system spec: selector: matchLabels: app: seccomp-profile-installer template: metadata: labels: app: seccomp-profile-installer spec: initContainers: - name: installer image: alpine:3.18 command: ["sh", "-c", "cp /profiles/* /host/seccomp/ && chmod 644 /host/seccomp/*"] volumeMounts: - name: profiles mountPath: /profiles - name: host-seccomp mountPath: /host/seccomp containers: - name: pause image: gcr.io/google_containers/pause:3.9 volumes: - name: profiles configMap: name: seccomp-profiles - name: host-seccomp hostPath: path: /var/lib/kubelet/seccomp type: DirectoryOrCreate Ensure the hostPath is the same as the kubelet's seccomp profile root. You may need to adjust the path based on your Kubernetes distribution. ### Applying Seccomp Profiles to Workloads To apply a custom profile to a pod, use the Localhost type with the path relative to the profile root: apiVersion: v1 kind: Pod metadata: name: custom-seccomp-pod spec: securityContext: seccompProfile: type: Localhost localhostProfile: "custom-profile.json" containers: - name: app image: busybox:1.36 command: ["sh", "-c", "sleep 3600"] Apply and test: kubectl apply -f custom-seccomp-pod.yaml kubectl exec -it custom-seccomp-pod -- sh -c "chmod 777 /tmp/test" # Expected: Operation not permitted kubectl exec -it custom-seccomp-pod -- sh -c "ls /tmp" # Expected: works fine The chmod syscall should return EPERM, and the profile logs can be inspected with dmesg | grep seccomp on the node. ### Gradual Rollout Strategy When upgrading seccomp profiles across many workloads, use a gradual rollout: - Audit mode : Deploy the profile with defaultAction: SCMP_ACT_LOG to log syscall denials without enforcing (as shown in the custom profile above). Monitor logs for legitimate syscalls that would be blocked in enforce mode. - Canary deployment : Apply the enforcing profile to a single replica or a canary deployment. Monitor application metrics and logs. - Expand gradually : Increase the percentage of pods using the new profile while monitoring. - Full rollout : Once confident, apply to all workloads. To implement audit mode with the RuntimeDefault profile, you cannot directly set audit; you must create a custom profile that mimics the runtime default but logs instead of blocking. However, for many runtimes, the default profile can be copied and modified. ## Verification and Diagnostics Verification ensures that seccomp profiles are correctly applied and that workloads function as expected. This section provides concrete commands and techniques. ### Verify Seccomp Profile is Applied to a Pod Use kubectl get pod -o yaml to inspect the pod's security context: kubectl get pod myapp- -n production -o jsonpath='{.spec.securityContext.seccompProfile}{"\n"}' # Example output: {"type":"RuntimeDefault"} If the pod inherits seccomp from a higher level (e.g., PodSecurityPolicy, now replaced by Pod Security Admission, or a namespace-level policy), the field may not be directly set. You can check the effective seccomp by examining the container runtime config on the node, but that is more advanced. For most purposes, if the field is set, it is effective. ### Check for Seccomp-Related Denials in Logs If an application is blocked by seccomp, the container logs may show errors like "Operation not permitted" or "Invalid argument", and the node kernel logs will have seccomp audit messages. To view pod logs: kubectl logs myapp- -n production --tail=50 Example of a denial log from a container: 2024/05/20 10:15:32 [error] 1234#1234: *1 chmod() "/var/www/html/test" failed (1: Operation not permitted) To view node kernel logs (if you have node access): journalctl -k | grep seccomp # Example output: # audit: type=1326 audit(1716198932.123:456): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=1234 comm="nginx" exe="/usr/sbin/nginx" sig=0 arch=c000003e syscall=90 compat=0 ip=0x7f... code=0x50000 In this example, syscall 90 is chmod on x86_64. ### Using Seccomp Notify for Advanced Diagnostics Some runtimes support seccomp notify, which allows a userspace process to handle seccomp decisions. This is complex and beyond the scope of this guide; however, tools like oci-seccomp-bpf-hook can help simulate and debug profiles. ### Testing Profiles in Isolation Create a test pod that runs the same container image and commands as your production workload, but with the candidate seccomp profile. Use kubectl run for quick tests: kubectl run seccomp-test --image=nginx:1.25 --restart=Never --dry-run=client -o yaml | kubectl apply -f - Then patch the test pod with the desired profile and exec into it to run application-specific commands. ### Validating Profile Syntax Before deploying a custom seccomp profile, validate the JSON syntax and semantics. Use jq to parse: jq empty custom-profile.json && echo "Valid JSON" For semantic validation, you can use the seccomp tool from libseccomp or an online validator. There is no built-in Kubernetes validation for profile content; invalid profiles may cause containers to fail at start. ## Failure Modes and Recovery Despite careful planning, failures can occur. This section identifies common failure modes and provides recovery procedures. ### Pod Fails to Start with Seccomp Error Symptom : Pod remains in ContainerCreating or CrashLoopBackOff , and kubectl describe pod shows an error like: Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting seccomp filter: invalid argument Cause : Invalid seccomp profile (syntax error, unsupported architecture, etc.) or missing profile file. Recovery : - Check if the profile file exists on the node and has correct permissions. - Validate profile JSON. - If the profile is invalid, correct it and redeploy. - As an immediate mitigation, change the pod's seccompProfile type to Unconfined (temporarily) or RuntimeDefault to allow the pod to start. kubectl patch pod myapp- -n production --type='json' -p='[{"op": "replace", "path": "/spec/securityContext/seccompProfile", "value": {"type": "Unconfined"}}]' Note: You cannot patch a running pod's securityContext; you need to delete and recreate the pod or update the deployment. ### Application Breaks After Seccomp Enforcement Symptom : Pod starts but application functionality fails (e.g., cannot write files, network calls fail). Logs show "Operation not permitted" for syscalls that the profile blocks. Cause : The seccomp profile is too restrictive for the application's legitimate syscalls. Recovery : - Identify which syscall is being blocked from logs or seccomp audit. - Modify the custom profile to allow that syscall (add to names with action SCMP_ACT_ALLOW ). - If using RuntimeDefault and it's too restrictive, you may need to switch to Unconfined or create a custom profile based on the runtime default with additional allowances. - Rollback the deployment to the previous version with a known-good seccomp configuration. Example rollback command: kubectl rollout undo deployment/myapp -n production ### Profile File Missing on Some Nodes Symptom : Pods scheduled on certain nodes fail with "cannot load seccomp profile", while others succeed. Cause : The DaemonSet that copies profiles did not run or failed on those nodes, or the kubelet seccomp root differs. Recovery : - Check DaemonSet status: kubectl get ds seccomp-profile-installer -n kube-system . - Check pods on failed nodes: kubectl get pods -n kube-system -o wide | grep seccomp-profile-installer . - Ensure the hostPath is correct; some Kubernetes distributions use /var/lib/kubelet/seccomp while others use /var/lib/rancher/rke2/agent/containerd/seccomp or similar. - Manually copy the profile to the node as an emergency measure. ### Upgrade of Kubernetes Causes Seccomp Behavior Change Symptom : After upgrading Kubernetes version, previously working pods with custom seccomp profiles fail. Cause : The seccomp field semantics changed, or the runtime's default profile changed. Recovery : - Review the Kubernetes changelog for seccomp-related changes. - Ensure all deprecated annotations are migrated before upgrading to v1.25+. - If the RuntimeDefault profile changed, test your applications against the new default in a staging environment. - If necessary, pin the runtime version or adjust profiles to match the new behavior. ### General Recovery Checklist - Document rollback commands for every change. - Use version control for manifests and profiles; git revert can rollback configuration quickly. - Implement canary deployments to limit impact. - Monitor seccomp denials with metrics or log aggregation to detect issues early. ## Operations Checklist Use this checklist before, during, and after a seccomp upgrade or migration to ensure completeness. ### Pre-Upgrade - [ ] Verify Kubernetes version supports stable seccomp field (>= v1.25). Run kubectl version --short . - [ ] Inventory all pods using seccomp annotations or fields with kubectl get pods -A -o json . - [ ] Identify custom seccomp profiles and their locations (ConfigMaps, node files). - [ ] Validate custom profile JSON syntax. - [ ] Test candidate profile in a staging environment with representative workloads. - [ ] Set up log aggregation for kernel audit logs to capture seccomp denials. - [ ] Document rollback plan for each workload. ### During Upgrade - [ ] Migrate annotations to seccompProfile field using patch or policy engine. - [ ] Deploy custom profiles to all nodes via DaemonSet; verify DaemonSet pods are running on all nodes. - [ ] Apply seccomp changes to canary workloads first and monitor for 24-48 hours. - [ ] Check application logs for denied syscalls; adjust profiles if necessary. - [ ] Gradually roll out to remaining workloads in batches. ### Post-Upgrade Verification - [ ] Confirm seccomp profile is effective on pods with kubectl get pod -o yaml . - [ ] Run application-specific test suite to ensure full functionality. - [ ] Monitor node kernel logs for unexpected seccomp denials over a week. - [ ] Review security posture: no pods should be running Unconfined unless explicitly required. ### Example Verification Commands # Check seccomp type for all pods in a namespace kubectl get pods -n production -o json | jq -r '.items[] | .metadata.name + " -> " + (.spec.securityContext.seccompProfile.type // "Unconfined")' # Expected output (abbreviated): # myapp-6d4b7c9f8-abcde -> RuntimeDefault # myapp-6d4b7c9f8-fghij -> RuntimeDefault # legacy-app-5c8b6f4d9-klmno -> Localhost If any pod shows Unconfined unexpectedly, investigate and apply the intended profile. ## Conclusion Kubernetes Seccomp upgrade and migration require careful planning and execution. By following the field guide in this article, you can minimize risk and ensure a smooth transition to a more secure cluster. Remember to: - Observe before changing : Inventory current state and validate profiles in a test environment. - Limit blast radius : Use canary deployments and gradual rollout. - Verify results : Check pod security contexts and monitor for denials. - Plan recovery : Document rollback commands and keep version-controlled manifests. As a next step, choose one low-risk workload and apply the RuntimeDefault seccomp profile (if not already set). Record the current state, run the workload, monitor for a day, and then expand to more workloads. If you encounter issues, use the failure modes section to diagnose and recover. With these practices, you can achieve a secure and stable Kubernetes environment.