Migrating from Kubernetes Pod Security Policies: A Safe Local Lab with Practical Examples

Introduction

Migrating away from Kubernetes Pod Security Policies (PSP) is a critical step for teams running Kubernetes 1.25 and later, where PSP was removed entirely. The replacement, Pod Security Admission (PSA), enforces security standards at the namespace level using built-in labels, without the complexity of custom admission plugins. This article walks through a safe, local lab environment to test the migration workflow before touching production clusters.

We focus on a local setup using Minikube or Kind, so you can experiment without risk. You will learn how to:

• Inventory your current PSP usage.
• Set up a local Kubernetes cluster with a version that supports PSA.
• Create Pod Security Admission configurations.
• Test workloads that were previously governed by PSPs.
• Diagnose and recover from common failure modes.

The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify results, and document recovery procedures.

Version and Environment Inventory

Before any migration, you must know the current state of your cluster. Begin by identifying the Kubernetes version, because PSP was removed in v1.25. If you are on v1.21 to v1.24, PSP is deprecated but still functional; if on v1.25+, PSP no longer exists. The replacement, Pod Security Admission, is available since v1.22 and stable in v1.25.

For a local lab, choose a tool that matches your production environment as closely as possible. The following table lists common local Kubernetes options:

For this guide, we will use Minikube with Kubernetes v1.25 to simulate a cluster where PSP is removed, but the migration process is similar for other tools.

Check your cluster version with:

kubectl version --short
# Output example:
# Client Version: v1.25.3
# Server Version: v1.25.3

Next, inventory existing Pod Security Admission configurations. Since PSA enforces policies using namespace labels, check for security labels on namespaces:

kubectl get namespaces --show-labels
# Look for labels pod-security.kubernetes.io/enforce, audit, warn.

If you are migrating from an older cluster that still has PodSecurityPolicies, consult the official migration guide to map existing PSP rules to PSA levels before upgrading. In Kubernetes v1.25 and later, PSP resources do not exist, so do not attempt to back them up on the new cluster.

Safe Configuration Path

Now, set up a local cluster to test migration in isolation. Install Minikube if not already present, then start a cluster with Kubernetes v1.25:

minikube start --kubernetes-version=v1.25.3

Verify the cluster is running:

kubectl get nodes
# Expected output:
# NAME       STATUS   ROLES           AGE   VERSION
# minikube   Ready    control-plane   10s   v1.25.3

Confirm that PSP is no longer available:

kubectl get psp
# Output: error: the server doesn't have a resource type "psp"

Now, enable Pod Security Admission. In Kubernetes v1.25, PSA is enabled by default and cannot be disabled. The admission controller uses namespace labels to enforce three predefined security levels: privileged, baseline, and restricted.

To test different levels, create namespaces with appropriate labels. For example, create a namespace for a workload that should run unrestricted (legacy privileged):

kubectl create namespace privileged-apps
kubectl label namespace privileged-apps pod-security.kubernetes.io/enforce=privileged
kubectl label namespace privileged-apps pod-security.kubernetes.io/audit=privileged
kubectl label namespace privileged-apps pod-security.kubernetes.io/warn=privileged

And a namespace for a workload that should be restricted:

kubectl create namespace restricted-apps
kubectl label namespace restricted-apps pod-security.kubernetes.io/enforce=restricted
kubectl label namespace restricted-apps pod-security.kubernetes.io/audit=restricted
kubectl label namespace restricted-apps pod-security.kubernetes.io/warn=restricted

Labels explained:

• enforce: rejects pods that violate the policy.
• audit: adds audit annotations for violations but allows the pod.
• warn: warns the user but allows the pod.

For a smooth migration, start with audit and warn only, observe what would be rejected, then gradually switch to enforce. This reduces surprises.

Verification and Diagnostics

With the lab set up, test how PSA handles different pod specs. Let's create a simple pod that violates the restricted policy by running as root. First, create a manifest pod-violating.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: privileged-pod
spec:
  containers:
  - name: nginx
    image: nginx:1.23
    securityContext:
      runAsNonRoot: false   # This violates restricted policy
      runAsUser: 0

Attempt to deploy it in the restricted namespace:

kubectl apply -f pod-violating.yaml -n restricted-apps

Expected output (rejection):

Error from server (Forbidden): error when creating "pod-violating.yaml": pods "privileged-pod" is forbidden: violates PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "nginx" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "nginx" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod must not set securityContext.runAsNonRoot=false), seccompProfile (pod or container "nginx" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

Now, deploy the same pod in the privileged namespace:

kubectl apply -f pod-violating.yaml -n privileged-apps
# Expected output: pod/privileged-pod created

This confirms that PSA enforces policies per namespace. To diagnose other pod issues, use:

kubectl describe pod <pod-name> -n <namespace>
# Look for Events section for security-related rejections.

Also, check pod logs to ensure the application started correctly:

kubectl logs privileged-pod -n privileged-apps
# Expected output leading to nginx startup logs.

For a more realistic test, deploy a deployment with multiple replicas and verify rollout:

kubectl create deployment nginx-deployment --image=nginx:1.23 -n restricted-apps
# Observe rejection if the default security context violates restricted policy.
kubectl rollout status deployment/nginx-deployment -n restricted-apps

If the rollout fails, inspect the deployment events:

kubectl describe deployment nginx-deployment -n restricted-apps

This will show the pod security admission error. To fix, you need to adjust the pod security context to comply with restricted policy, for example:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: restricted-apps
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        seccompProfile:
          type: RuntimeDefault
      containers:
      - name: nginx
        image: nginx:1.23
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop: ["ALL"]

Apply and check:

kubectl apply -f deployment-fixed.yaml
kubectl rollout status deployment/nginx-deployment -n restricted-apps
# Expected: deployment "nginx-deployment" successfully rolled out

This iterative approach verifies that workloads can be migrated to the restricted baseline.

Failure Modes and Recovery

During migration, you may encounter several common failure scenarios. Below are typical issues, their causes, and recovery steps.

Pod rejected with "violates PodSecurity"

This occurs when the pod spec does not meet the namespace's security level. The error message lists the specific violations. To recover, either:

• Adjust the pod security context to comply (recommended).
• Change the namespace label to a less restrictive level if the workload genuinely requires it (e.g., privileged for legacy apps).

Example: if a pod requires NET_ADMIN capability, restricted policy will reject it. You could move the pod to a namespace labeled baseline or privileged, but evaluate the security implications first.

PSP still active in older clusters

If you are on Kubernetes v1.21-1.24, PSP may still be enabled. To transition, you can run PSA alongside PSP. Enable PSA feature gate and gradually migrate namespaces. In this case, a pod must pass both PSP and PSA to be admitted. To minimize conflicts, replicate the PSP restrictions in PSA labels.

Namespace labels not applied correctly

If labels are missing or mistyped, PSA will not enforce as expected. Check namespace labels with:

kubectl get namespace <name> --show-labels

Verify pod-security.kubernetes.io/enforce is set to the desired level.

Audit logs not showing violations

If you set only audit label, pods are allowed but audit annotations are added. Check pod events or audit logs (if backend configured) to see violations. To review audit annotations on a pod:

kubectl describe pod <pod-name> -n <namespace> | grep -A5 Annotations

Rollback to PSP temporarily

On clusters still supporting PSP, you can reapply previously exported PSP manifests if available, and remove PSA labels from affected namespaces. On Kubernetes v1.25 or later, PSP cannot be restored; instead, temporarily switch the namespace's enforce label to a less restrictive level such as baseline or privileged while you fix the underlying issue. However, plan to fix the underlying issue rather than relying on PSP long-term.

Common Pitfalls and How to Avoid Them

Migration from PSP to PSA often fails due to several common mistakes. Understanding these pitfalls helps you avoid costly downtime.

Assuming restricted policy fits all workloads

The restricted level is the strictest and may break many existing applications that rely on root or Linux capabilities. Many teams label all namespaces as restricted and cause widespread pod rejections. To avoid this, start with audit and warn labels only, analyze the violations, and then decide the appropriate level per namespace. Use a gradual rollout: first audit, then warn, then enforce on a subset of namespaces.

Not inventorying PSP usage before migration

Without a clear inventory, you may miss critical workloads. Use tools like kubectl psp-migrate (from the Kubernetes sig-auth team) or scripts to list pods that reference PSPs. If your pods use service accounts, check role bindings that grant PSP usage. A thorough inventory is essential for a smooth transition.

Ignoring difference between PSA levels and PSP features

PSP had many controls, such as allowed host paths, allowed FlexVolumes, and default seccomp profiles. PSA levels do not cover all PSP fields; for advanced controls, you may need to use other admission controllers like Kyverno or OPA Gatekeeper. Do not assume PSP and PSA are equivalent. Map your PSP requirements to PSA levels and identify gaps.

Applying policy changes without testing in a lab

Testing in production is dangerous. Always use a local lab or a staging cluster to validate migration steps. The lab environment described in this article allows you to experiment safely. Replicate your production workloads in the lab and test with representative configurations.

Not updating CI/CD pipelines

Pipelines that deploy pods with old security contexts will fail after migrating to PSA. Update your CI/CD templates to include compliant security contexts. Also, scan your container images for compliance with restricted policies using tools like kube-score or polaris.

Operations Checklist

Use the following checklist to guide your migration process. Assign an owner for each item and a review frequency.

Assigning a single accountable owner ensures clear responsibility. Review progress in weekly migration meetings and adjust the plan as needed.

Conclusion

Migrating from Kubernetes Pod Security Policies to Pod Security Admission is a necessary step for clusters on Kubernetes 1.25 and later. By using a local lab, you can safely test the process, identify potential issues, and train your team before production changes.

In this article, we covered:

• Inventorying your current cluster and PSPs.
• Setting up a local cluster with Minikube.
• Configuring namespace labels for PSA levels.
• Testing pod compliance and diagnosing errors.
• Handling common failure modes and pitfalls.
• Providing an operations checklist with owners.

Next, choose one low-risk verification from the checklist, run it in your lab, and document the results. This incremental approach will make your migration less stressful and more successful. Remember, the goal is not just to remove PSP, but to adopt a robust security posture with Pod Security Admission and complementary tools where needed.

A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision.