## Intro Kubernetes Priority Classes govern the order in which pods are scheduled and which pods get evicted when a cluster runs out of resources. When priority settings are misconfigured, the results can be subtle: critical workloads starved, lower-priority pods deleted unexpectedly, or preemption not working at all. This guide offers a practical approach to troubleshoot Priority Class issues. We focus on common failure patterns, safe configuration changes, verification commands, and recovery steps. By the end, you will be able to diagnose and fix priority-related problems with confidence. We begin by gathering the necessary environment details, then discuss safe configuration paths before diving into diagnostics and recovery. Throughout, we include concrete commands and expected outputs, so you can follow along in your own cluster. Remember: a narrow, measurable pilot is the best way to validate priority behavior before scaling out. ## Version and Environment Inventory Before changing anything, gather information about your cluster and current Priority Classes. This establishes a baseline and helps identify version-specific behavior. ### Prerequisites - kubectl installed and configured with access to the target cluster. - Permissions to view and modify PriorityClass objects (typically cluster-admin). - The cluster's Kubernetes version (priority features have evolved over releases). ### Gather Cluster Version Run the following command to check both client and server versions: ```bash kubectl version --short ``` Expected output (example): ``` Client Version: v1.25.3 Server Version: v1.25.3 ``` Note the server version. PriorityClass has been stable since v1.14, but behavior around preemption and pod priority admission may vary slightly. For instance, older versions before v1.14 do not support PriorityClass at all. Starting with v1.24, the `--short` flag is deprecated, but the command still works in a pinch. In newer versions, use `kubectl version` without `--short` for full details. ### List Existing Priority Classes ```bash kubectl get priorityclass ``` Example output: ``` NAME VALUE GLOBAL-DEFAULT AGE system-cluster-critical 2000000000 false 2d system-node-critical 2000001000 false 2d high-priority 1000 false 1h default 0 true 2d ``` The VALUE column is the priority; higher numbers mean higher priority. The GLOBAL-DEFAULT column indicates whether this class is applied to pods without a `priorityClassName`. ### Inspect Running Pods and Their Priorities Check the current priority of running pods across all namespaces: ```bash kubectl get pods -A -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,PRIORITY:.spec.priority,PRIORITYCLASS:.spec.priorityClassName,STATUS:.status.phase ``` Example output (truncated): ``` NAMESPACE NAME PRIORITY PRIORITYCLASS STATUS default nginx-deployment-6d4cf56db-abcde 0 default Running kube-system coredns-558bd4d5db-xyz12 2000000000 system-cluster-critical Running ``` This shows which pods are using which Priority Class and their effective priority. If a pod does not have a `priorityClassName` and no global default is set, its priority is 0. ### Topology Notes Priority behavior can be influenced by node capacity, resource requests, and taints. Make a note of the number of nodes and their allocatable resources if preemption is involved. A quick summary: ```bash kubectl get nodes -o custom-columns=NAME:.metadata.name,ALLOCATABLE_CPU:.status.allocatable.cpu,ALLOCATABLE_MEMORY:.status.allocatable.memory ``` Keep this information handy; you'll reference it when checking preemption decisions. For example, if a node has 2 CPUs allocatable and you try to schedule a pod requesting 2 CPUs, preemption may be needed if other pods occupy that capacity. ## Safe Configuration Path When modifying Priority Classes, follow a safe, scoped approach. Avoid changing the global default or system classes unless absolutely necessary. Instead, create new classes or adjust existing ones with clear intent. ### Understanding Priority Class Fields A PriorityClass object looks like this: ```yaml apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: high-priority value: 1000 globalDefault: false description: "Used for critical user workloads" ``` Key fields: - `value`: Integer priority. Higher value means higher priority. The range is typically 1 to 1,000,000,000, but can be customized via admission configuration. - `globalDefault`: If true, pods without `priorityClassName` get this priority. Only one can be global default. - `description`: Human-readable explanation. ### Safe Change: Create or Update a Non-Default Priority Class Start by creating a new PriorityClass with a clear value instead of modifying existing ones: ```bash cat < -n ``` ### Scoped Implementation Choices - Use distinct PriorityClasses for different workload tiers (e.g., `high`, `medium`, `low`) with clear value ranges. - Avoid setting `globalDefault: true` unless you have a deliberate default priority for all pods. The default priority for pods without a class is 0. - Do not modify `system-cluster-critical` or `system-node-critical`; they are reserved for Kubernetes system components and have very high values (2,000,000,000 and above). Changing them can break cluster stability. After making changes, proceed to verify behavior. ## Verification and Diagnostics Verification involves checking pod priority assignment, preemption events, and scheduler logs if needed. We'll walk through common checks. ### Confirm Pod Priority Assignment For any pod, you can check its priority and priorityClassName: ```bash kubectl get pod -n -o jsonpath='{.spec.priority} {.spec.priorityClassName}' ``` Example: ``` 500 medium-priority ``` If the pod shows no priority, it might be using the global default. Check the global default class: ```bash kubectl get priorityclass -o json | jq '.items[] | select(.globalDefault==true) | .metadata.name' ``` ### Simulating Preemption Behavior To test preemption, create a high-priority pod when the cluster is full. First, fill a node with low-priority pods. Then, submit a high-priority pod and observe events. Create a low-priority class (if not present) with value 0: ```bash kubectl create priorityclass low-priority --value=0 --global-default=false ``` Create several pods that request enough CPU to fill a node. For a node with 2 CPUs, you could create two pods each requesting 1 CPU, or one pod requesting 2 CPUs. Example low-priority pod spec: ```yaml apiVersion: v1 kind: Pod metadata: name: low-pod-1 spec: priorityClassName: low-priority containers: - name: busybox image: busybox command: ["sleep", "3600"] resources: requests: cpu: "1000m" ``` Apply multiple such pods until the node is full. Then, create a high-priority pod that requests more CPU than available: ```yaml apiVersion: v1 kind: Pod metadata: name: high-priority-pod spec: priorityClassName: high-priority containers: - name: busybox image: busybox command: ["sleep", "3600"] resources: requests: cpu: "1500m" ``` Note: The high-priority pod must be scheduled on the same node. To force that, you can use a node selector or ensure only one node is available. Watch events: ```bash kubectl get events --sort-by=.lastTimestamp ``` Look for messages like `Preempted to accommodate higher priority pod`. Example event: ``` LAST SEEN TYPE REASON OBJECT MESSAGE 2m Normal Preempted pod/low-pod-1 Preempted by high-priority-pod on node node-1 ``` This confirms preemption works. ### Checking Scheduler Decisions If preemption is not happening, inspect the scheduler logs. If you have access to the control plane, you can search for priority-related logs: ```bash kubectl logs -n kube-system | grep -i priority ``` Alternatively, if using a managed cluster (e.g., EKS, GKE, AKS), you may need to rely on events and pod status because control plane logs are not directly accessible. In that case, describe the pending pod and check for scheduler messages. ### Diagnose "Pod is Pending" with Priority Issues A pending pod might not be scheduled because no preemption occurred due to certain conditions. Describe the pod: ```bash kubectl describe pod -n ``` Look for events. Common reasons: - `Insufficient cpu` without mention of preemption: The scheduler may not preempt if the pod's priority is not high enough relative to existing pods. - `Preemption is not helpful for scheduling`: This happens when the pod's priority is too low to evict any existing pods. Ensure the pod's priority is actually higher than the pods that would be evicted. For example, if the pending pod has priority 1000 and the running pods have priority 2000, no preemption will occur. ### Verify PriorityClass Values Are in Range Priority values must be within the range allowed by the admission controller. By default, the valid range is 1 to 1,000,000,000 (1 billion). Values above may be rejected. Check pod events for errors like: ``` Error creating: pods "test" is forbidden: spec.priority: Invalid value: 2000000000: must be less than or equal to 1000000000 ``` If you have custom admission configurations, verify the allowed range. For example, some clusters set a maximum priority lower than 1 billion to prevent accidental high priorities. ### Check PriorityClass Binding to Pods Sometimes a pod references a nonexistent PriorityClass. The pod will be rejected with an error like: ``` Error from server (Forbidden): error when creating "pod.yaml": pods "test" is forbidden: no PriorityClass with name nonexistent-priority was found ``` Fix by creating the class or correcting the pod spec. ## Common Pitfalls and How to Avoid Them Priority Class misconfigurations often stem from a few recurring mistakes. Understanding these pitfalls can save you hours of debugging. ### Pitfall 1: Assuming Priority Changes Apply to Running Pods **Why it happens:** Priority is assigned at pod admission time. When you update a PriorityClass's value, existing pods already have a numeric priority set and do not automatically update. **How to avoid:** After modifying a PriorityClass, explicitly restart the affected pods or use a rolling update. For Deployments, use `kubectl rollout restart`. For standalone pods, delete and recreate them. ### Pitfall 2: Setting globalDefault Without Testing **Why it happens:** Setting `globalDefault: true` on a PriorityClass affects every pod that does not explicitly specify a `priorityClassName`. This can inadvertently give high priority to pods that should be low priority, causing unexpected preemptions. **How to avoid:** Never change the global default in production without first testing in a sandbox namespace. Prefer creating new non-default PriorityClasses and assigning them explicitly to pods. ### Pitfall 3: Misunderstanding Priority Values **Why it happens:** Priority values are arbitrary integers, but their relative magnitude matters. A pod with priority 1000 will not preempt a pod with priority 2000. However, the absolute number does not convey the importance; only comparison matters. **How to avoid:** Define a clear priority scale for your organization. For example, use ranges like 0-999 for batch jobs, 1000-9999 for standard services, and 10000+ for critical services. Document this scale. ### Pitfall 4: Forgetting Resource Requests **Why it happens:** Preemption requires resource requests to determine if evicting lower-priority pods will free enough capacity for the higher-priority pod. If the pending pod has no resource requests, the scheduler may not consider preemption. **How to avoid:** Always set resource requests for pods that may need preemption. Without requests, the scheduler assumes zero resource usage and may not preempt even if the node is actually full. ### Pitfall 5: Modifying System Priority Classes **Why it happens:** It's tempting to adjust `system-cluster-critical` or `system-node-critical` to influence scheduling, but these are reserved for system components. Changing their values can disrupt cluster stability. **How to avoid:** Never touch these classes. Instead, create your own high-priority classes with values below the system classes (e.g., below 2,000,000,000). ### Pitfall 6: Not Testing Preemption in a Controlled Environment **Why it happens:** Preemption can cause unexpected evictions in production if not tested. A misconfigured priority can evict critical workloads. **How to avoid:** Always test preemption in a dedicated namespace or a test cluster before rolling out priority changes to production. Simulate high-priority pod scheduling and observe which pods get evicted. ## Failure Modes and Recovery Here we cover common failure scenarios and how to recover. ### 1. Incorrect Priority Value **Symptom:** Pods are preempted unexpectedly or not preempted at all. **Diagnosis:** Check the pod's priority and compare with the intended PriorityClass. Use `kubectl get priorityclass` to view values. **Recovery:** Update the PriorityClass value. To immediately apply to running pods, restart the pods (e.g., delete them and let the controller recreate). For standalone pods, you need to recreate them. ```bash kubectl delete pod -n ``` ### 2. Global Default Misconfiguration **Symptom:** New pods get an unexpected priority. For example, every pod gets priority 1000 instead of 0. **Diagnosis:** Check which PriorityClass has `globalDefault: true`: ```bash kubectl get priorityclass -o json | jq '.items[] | select(.globalDefault==true) | {name: .metadata.name, value: .value}' ``` **Recovery:** Set `globalDefault: false` on that class and optionally set it on another. Remember that only one can be global default. If you disable all, pods without priority get priority 0 (implicit). ```bash kubectl patch priorityclass -p '{"globalDefault":false}' ``` ### 3. Preemption Not Working as Expected **Symptom:** High-priority pods remain pending even though lower-priority pods are running. **Possible Causes:** - The high-priority pod's priority is not higher than the lower-priority pods' priorities. Compute the values. - The scheduler's preemption logic may be disabled or restricted by feature gates. Check scheduler configuration. - Resource requests are not set on the pending pod; preemption requires resource requests to determine eviction. - Node selectors or taints may limit scheduling. **Recovery:** Ensure the pending pod has appropriate resource requests and a sufficiently high priority. Then check events: ```bash kubectl describe pod ``` Look for messages like `Preempting to accommodate higher priority pod`. If none, review scheduler logs (if accessible). ### 4. Pod Cannot Be Created Due to Invalid Priority Class **Symptom:** Pod creation fails with error referencing PriorityClass. **Recovery:** Create the missing PriorityClass or fix the pod spec. After creation, the pod will be admitted if priority is within range. ### 5. PriorityClass Deleted While in Use **Symptom:** Existing pods still run, but new pods referencing that class fail. **Diagnosis:** `kubectl get priorityclass` shows it missing. Pods with that `priorityClassName` will fail to be created. **Recovery:** Recreate the PriorityClass with the same name and appropriate value. Existing pods will retain their assigned priority, but new ones will work. ### Rollback Procedures Always have a rollback plan. If a change causes issues, you can: - Revert the PriorityClass manifest to the previous version using `kubectl apply`. - Delete the PriorityClass (if it was newly added) and update pods to remove the reference. - For `globalDefault`, restore the previous setting. Document changes beforehand to facilitate quick rollback. ## Operations Checklist with Ownership and Review Cadence Use this checklist as a repeatable procedure when managing Priority Classes. Each step includes a responsible role and a review frequency. | Step | Action | Verification | Owner | Review Cadence | |------|--------|--------------|-------|----------------| | 1 | Review existing PriorityClasses | `kubectl get priorityclass` | Platform Engineer (e.g., Alex Johnson) | Monthly | | 2 | Identify workloads and their required priority | Check pod specs | Application Owner (e.g., Sarah Lee) | Each release | | 3 | Create or update PriorityClass with a clear name and value | `kubectl apply -f priorityclass.yaml` | Platform Engineer | As needed | | 4 | Apply priorityClassName to pods | Pod spec includes `priorityClassName` | Application Developer | Each deployment | | 5 | Verify pod priority after creation | `kubectl get pod -o jsonpath='{.spec.priority}'` | Platform Engineer | Immediately after | | 6 | Test preemption in a controlled environment | Deploy filler pods and high-priority pod, check events | QA Engineer (e.g., Maria Garcia) | Before production rollout | | 7 | Monitor for unexpected evictions or pending pods | Watch events and scheduler logs | Site Reliability Engineer (e.g., David Kim) | Weekly | | 8 | Document any changes for future rollback | Maintain manifests in version control | All owners | Each change | Additional reminders: - Avoid changing system priority classes. - Keep priority values within the allowed range. - Use descriptive names and description fields. - Test before applying to production. ## Conclusion Priority Class troubleshooting requires methodical verification and careful changes. By following this guide, you can diagnose common issues such as incorrect priorities, preemption failures, and misconfigured global defaults. Start with a narrow pilot, verify behavior with concrete commands, and always plan for rollback. With the checklist and recovery steps, you can maintain reliable pod scheduling and resource preemption in your Kubernetes clusters. Verified next steps: - Inventory your current PriorityClasses and pod assignments. - Implement a new non-default PriorityClass for a test workload. - Test preemption behavior in a safe namespace. - Monitor events and scheduler logs for unexpected behavior. - Gradually roll out priority settings to production workloads with clear rollback plans. By institutionalizing these practices, you turn Priority Class management into a repeatable operational routine.