## Intro Kubernetes Pod Priority and Preemption is a scheduler feature that helps cluster operators protect critical workloads during resource shortages. When a cluster runs out of capacity, the scheduler can evict lower-priority Pods to make room for higher-priority ones. This mechanism is powerful, but it can cause unexpected disruptions if it is not planned carefully. This article is written for developers, DevOps consultants, and technical startup teams who need to plan cluster capacity with priority and preemption in mind. It connects the core concepts of priority classes, preemption policies, resource requests, and node capacity to practical commands, expected outputs, failure signals, and recovery decisions. You will learn how to observe the current state before making changes, limit the blast radius of any intervention, verify the result, and document recovery paths. The goal is operational safety. You should never apply a priority change without understanding which Pods might be evicted as a result. Use read-only inspection first, protect sensitive values with placeholders, change one scoped item at a time, and always verify that the expected state has been reached. This article provides a structured workflow that you can adapt to your own clusters. ## Version and Environment Inventory Before using priority and preemption, confirm which Kubernetes version you are running and whether the feature is enabled. Pod Priority and Preemption is stable and enabled by default in Kubernetes v1.14 and later. For older versions, you may need to enable the feature gate PodPriority on the API server and scheduler. Check your cluster version with: kubectl version --short Expected output includes both client and server versions. If the server version is below v1.14, plan an upgrade or verify that the feature gate is explicitly enabled. Next, check the current PriorityClasses in the cluster: kubectl get priorityclass A typical output looks like: NAME VALUE GLOBAL-DEFAULT AGE system-cluster-critical 2000000000 false 30d system-node-critical 2000001000 false 30d If no custom PriorityClasses exist, you will see only the two built-in system classes. These system classes are used by critical cluster components and should not be assigned to user workloads. For capacity planning, you also need to know the total allocatable resources on your nodes: kubectl describe nodes | grep -A 5 "Allocatable" This shows CPU and memory available for Pods on each node. Compare these numbers with the total resource requests of running Pods: kubectl get pods -A -o=jsonpath='{range .items[*]}{.metadata.namespace}{"\t"}{.metadata.name}{"\t"}{.spec.containers[*].resources.requests}{"\n"}{end}' This command prints the resource requests for every container. Sum them per namespace or per node to understand current utilization. For example, if three Pods on a node each request 500m CPU and the node allocatable CPU is 2000m, the node still has 500m available for new Pods. When planning priority changes, identify which Deployments or StatefulSets might be affected. Inspect a specific workload with: kubectl get deployment -o yaml Look for priorityClassName in the Pod template. If it is not set, the Pod gets the globalDefault PriorityClass if one exists, otherwise priority 0. Keep your local test environment small. Before changing priorities on a production cluster, reproduce the scenario on a local cluster using kind or minikube. Apply one manifest at a time, inspect the generated resources, and verify that scheduling behaves as expected. Use kubectl port-forward to test services locally before moving to a cloud load balancer or ingress controller. ## Safe Configuration Path A safe configuration path for priority and preemption starts with defining clear PriorityClasses that reflect your business tiers. Typical tiers are: - Critical infrastructure (value >= 1000000000): monitoring agents, logging forwarders, service mesh proxies that must run on every node. - Production applications (value around 100000 or 10000): user-facing services that must remain available but can tolerate brief eviction in favor of infrastructure. - Batch or development workloads (value 100 or less): optional jobs, test deployments, and background processing that can be evicted without impact. The value is a 32-bit integer. Higher values mean higher priority. Only use values above 1 billion for system-critical components; user workloads should stay below that threshold. Create a PriorityClass for production applications: apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: production-priority value: 10000 globalDefault: false description: "Priority for production applications" Apply it: kubectl apply -f production-priority.yaml Verify it exists: kubectl get priorityclass production-priority Now assign this PriorityClass to a Deployment. Add priorityClassName: production-priority to the Pod template: apiVersion: apps/v1 kind: Deployment metadata: name: web-frontend spec: replicas: 3 selector: matchLabels: app: web-frontend template: metadata: labels: app: web-frontend spec: priorityClassName: production-priority containers: - name: nginx image: nginx:1.25 resources: requests: cpu: "200m" memory: "256Mi" Apply the Deployment and check that Pods are created with the priority set: kubectl apply -f web-frontend.yaml kubectl get pods -l app=web-frontend -o wide kubectl describe pod | grep -i priority The describe output should include a line like: Priority: 10000 To test preemption without harming real workloads, create a low-priority filler Pod that consumes a large amount of resources, then attempt to schedule a high-priority Pod that requires those resources. For example, create a filler Pod that requests 1 CPU on a node with 2 allocatable CPUs. Then create a high-priority Pod requesting 1.5 CPUs. The scheduler should evict the filler Pod to place the high-priority one. Here is a filler Pod with priority 0 (no explicit class): apiVersion: v1 kind: Pod metadata: name: filler spec: containers: - name: stress image: polinux/stress resources: requests: cpu: "1000m" memory: "500Mi" nodeSelector: kubernetes.io/hostname: And a high-priority Pod: apiVersion: v1 kind: Pod metadata: name: important-job spec: priorityClassName: production-priority containers: - name: app image: nginx resources: requests: cpu: "1500m" memory: "500Mi" nodeSelector: kubernetes.io/hostname: After applying the high-priority Pod, watch events: kubectl get events --watch You should see events indicating that the filler Pod was preempted: Preempted: pod/filler Now verify the high-priority Pod is scheduled and running, while the filler is terminated: kubectl get pods Always test in a dedicated namespace or cluster to avoid accidental eviction of important workloads. ## Verification and Diagnostics After applying priority configurations, you must verify that the scheduler is behaving as intended and diagnose any issues. Use the following commands to inspect state and events. First, check the status of all Pods: kubectl get pods -o wide Look for Pods stuck in Pending state. A Pending Pod may indicate insufficient resources or preemption not occurring as expected. Inspect a specific pending Pod: kubectl describe pod The events section will show why the Pod cannot be scheduled. Common messages include: - 0/3 nodes are available: 3 Insufficient cpu. - Preemption is not helpful for scheduling. The second message appears when the scheduler cannot evict lower-priority Pods to make room, possibly because the pending Pod has a lower or equal priority than existing Pods, or because Pod Disruption Budgets prevent eviction. Check the scheduler logs for preemption decisions. If you have access to the control plane: kubectl logs -n kube-system | grep -i preempt Scheduler logs include entries like: "Preempting" pod="namespace/name" "Preemption succeeded" "Preemption failed" If preemption fails, check whether the Pod has a priorityClassName set and whether the PriorityClass value is higher than the victims. Also check if any Pod Disruption Budget (PDB) blocks the eviction: kubectl get pdb -A Verify resource requests and limits are correctly configured. Use: kubectl get pod -o yaml | grep -A 5 resources If requests are not set, the Pod is treated as requesting zero resources, which can lead to overcommitment and eviction. Always set requests for critical workloads. For capacity planning, compare actual usage with requests using metrics-server (if installed): kubectl top pods -A This shows CPU and memory usage per Pod. If many Pods use far less than their requests, you may be over-requesting and causing unnecessary preemption. Adjust requests to realistic values. ## Failure Modes and Recovery Priority and preemption can fail in several ways. Understanding these failure modes and having a recovery plan is essential. Failure Mode 1: High-priority Pod remains Pending If a high-priority Pod is stuck in Pending, check the scheduler events: kubectl describe pod | tail -20 Possible causes: - No lower-priority Pods on the nodes that satisfy the scheduling constraints. - Preemption is disabled by feature gate. - Pod Disruption Budget prevents eviction. - Node selectors or affinity rules restrict the Pod to nodes without enough resources. Recovery: adjust the Pod's scheduling constraints, increase cluster capacity, or temporarily relax PDBs after assessing risk. Failure Mode 2: Critical workload evicted unexpectedly If a production Pod is evicted, you will see it in a Terminating state or a new Pod replacing it. Check the event history: kubectl get events --sort-by=.lastTimestamp | grep -i preempt To recover, ensure that the critical workload has a high enough PriorityClass. Create a new PriorityClass with a higher value and assign it. For example: kubectl create priorityclass critical-app --value=100000 --global-default=false kubectl patch deployment -p '{"spec":{"template":{"spec":{"priorityClassName":"critical-app"}}}}' Then verify the new Pods are running with the correct priority: kubectl get pods -l app= -o wide kubectl describe pod | grep -i priority Failure Mode 3: Preemption loops Sometimes, two Deployments with equal priority may repeatedly preempt each other if they both request resources that the other holds. This can cause instability. To detect this, look at scheduler logs for repeated preemption events for the same Pods. Prevent this by assigning different priorities to competing workloads or by using node affinity to separate them. Use a tool like: kubectl logs -n kube-system --since=1h | grep "Preemption" Failure Mode 4: Preemption does not happen because PriorityClass is missing If a Pod template references a non-existent PriorityClass, the Pod will not be created, and the Deployment will show an error: kubectl describe deployment You will see events like: Error creating: pods "" is forbidden: no PriorityClass with name production-priority was found Recovery: create the missing PriorityClass or fix the typo in the Deployment. Always document the recovery steps for each failure mode before deploying priority changes to production. Use rollback mechanisms like kubectl rollout undo deployment/ if necessary. ## Operations Checklist Use this checklist before and after implementing priority and preemption changes. Before change: - [ ] Record current cluster state: kubectl get nodes -o wide , kubectl get pods -A -o wide - [ ] Note existing PriorityClasses: kubectl get priorityclass - [ ] Identify all Deployments and Pods that might be affected. - [ ] Check resource requests and limits: kubectl get pods -A -o=jsonpath='{...}' - [ ] Review Pod Disruption Budgets: kubectl get pdb -A - [ ] Ensure you have a rollback plan (e.g., save current YAMLs). - [ ] Test the change in a non-production environment. Apply change: - [ ] Apply new PriorityClass: kubectl apply -f priorityclass.yaml - [ ] Patch or apply Deployment with priorityClassName . - [ ] Use kubectl rollout status deployment/ to confirm rollout. After change: - [ ] Verify Pods are running: kubectl get pods -o wide - [ ] Check priority of new Pods: kubectl describe pod | grep Priority - [ ] Watch events for unexpected preemption: kubectl get events --sort-by=.lastTimestamp - [ ] If using metrics-server, compare resource usage: kubectl top pods -A - [ ] Update documentation with observed behavior. Recovery verification: - [ ] Simulate a failure by creating a low-priority Pod and a high-priority Pod in a test namespace. Confirm preemption works. - [ ] If preemption does not work as expected, check scheduler logs and PDBs. - [ ] Verify that rollback restores previous state: apply the saved YAMLs or kubectl rollout undo . Store all commands in a runbook that your team can access. Use placeholders for sensitive values and keep secrets in Kubernetes Secrets or external secret managers. ## Conclusion Kubernetes Pod Priority and Preemption is a key tool for capacity planning, but it must be used with care. By defining clear PriorityClasses, setting proper resource requests, and testing preemption behavior in a controlled environment, you can ensure that critical workloads survive resource shortages while lower-priority workloads are evicted gracefully. This article provided a practical workflow: start with version and environment inventory, move to safe configuration, verify behavior with diagnostics, understand failure modes, and follow an operations checklist. Each step includes concrete commands and expected outputs so you can replicate the examples in your own cluster. As a next step, choose one low-risk verification from the article. For example, create two test Pods with different priorities in a dedicated namespace and observe the preemption events. Record the current state, run the documented commands, and compare the results. Then review your production Deployments for missing priorityClassName and resource requests. A reliable operational workflow makes failure visible, protects sensitive values, limits changes to intended resources, and defines recovery verification before an incident occurs. With the principles and examples in this article, you can confidently plan capacity using Kubernetes Pod Priority and Preemption.