Kubernetes Priority Class Troubleshooting with Practical Examples

Intro

Kubernetes Priority Classes govern the order in which pods are scheduled and which pods get evicted when a cluster runs out of resources. When priority settings are misconfigured, the results can be subtle: critical workloads starved, lower-priority pods deleted unexpectedly, or preemption not working at all. This guide offers a practical approach to troubleshoot Priority Class issues. We focus on common failure patterns, safe configuration changes, verification commands, and recovery steps. By the end, you will be able to diagnose and fix priority-related problems with confidence.

We begin by gathering the necessary environment details, then discuss safe configuration paths before diving into diagnostics and recovery. Throughout, we include concrete commands and expected outputs, so you can follow along in your own cluster. Remember: a narrow, measurable pilot is the best way to validate priority behavior before scaling out.

Version and Environment Inventory

Before changing anything, gather information about your cluster and current Priority Classes. This establishes a baseline and helps identify version-specific behavior.

Prerequisites

• kubectl installed and configured with access to the target cluster.
• Permissions to view and modify PriorityClass objects (typically cluster-admin).
• The cluster's Kubernetes version (priority features have evolved over releases).

Gather Cluster Version

Run the following command to check both client and server versions:

kubectl version --short

Expected output (example):

Client Version: v1.25.3
Server Version: v1.25.3

Note the server version. PriorityClass has been stable since v1.14, but behavior around preemption and pod priority admission may vary slightly. For instance, older versions before v1.14 do not support PriorityClass at all. Starting with v1.24, the --short flag is deprecated, but the command still works in a pinch. In newer versions, use kubectl version without --short for full details.

List Existing Priority Classes

kubectl get priorityclass

Example output:

NAME                      VALUE        GLOBAL-DEFAULT   AGE
system-cluster-critical   2000000000   false            2d
system-node-critical      2000001000   false            2d
high-priority             1000         false            1h
default                   0            true             2d

The VALUE column is the priority; higher numbers mean higher priority. The GLOBAL-DEFAULT column indicates whether this class is applied to pods without a priorityClassName.

Inspect Running Pods and Their Priorities

Check the current priority of running pods across all namespaces:

kubectl get pods -A -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,PRIORITY:.spec.priority,PRIORITYCLASS:.spec.priorityClassName,STATUS:.status.phase

Example output (truncated):

NAMESPACE     NAME                              PRIORITY     PRIORITYCLASS            STATUS
default       nginx-deployment-6d4cf56db-abcde  0            default                  Running
kube-system   coredns-558bd4d5db-xyz12         2000000000   system-cluster-critical  Running

This shows which pods are using which Priority Class and their effective priority. If a pod does not have a priorityClassName and no global default is set, its priority is 0.

Topology Notes

Priority behavior can be influenced by node capacity, resource requests, and taints. Make a note of the number of nodes and their allocatable resources if preemption is involved. A quick summary:

kubectl get nodes -o custom-columns=NAME:.metadata.name,ALLOCATABLE_CPU:.status.allocatable.cpu,ALLOCATABLE_MEMORY:.status.allocatable.memory

Keep this information handy; you'll reference it when checking preemption decisions. For example, if a node has 2 CPUs allocatable and you try to schedule a pod requesting 2 CPUs, preemption may be needed if other pods occupy that capacity.

Safe Configuration Path

When modifying Priority Classes, follow a safe, scoped approach. Avoid changing the global default or system classes unless absolutely necessary. Instead, create new classes or adjust existing ones with clear intent.

Understanding Priority Class Fields

A PriorityClass object looks like this:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 1000
globalDefault: false
description: "Used for critical user workloads"

Key fields:

• value: Integer priority. Higher value means higher priority. The range is typically 1 to 1,000,000,000, but can be customized via admission configuration.
• globalDefault: If true, pods without priorityClassName get this priority. Only one can be global default.
• description: Human-readable explanation.

Safe Change: Create or Update a Non-Default Priority Class

Start by creating a new PriorityClass with a clear value instead of modifying existing ones:

cat <<EOF | kubectl apply -f -
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: medium-priority
value: 500
globalDefault: false
description: "Medium priority for batch jobs"
EOF

Expected output:

priorityclass.scheduling.k8s.io/medium-priority created

Applying to a Pod

Reference the new class in a pod spec:

apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  priorityClassName: medium-priority
  containers:
  - name: busybox
    image: busybox
    command: ["sleep", "3600"]
    resources:
      requests:
        memory: "64Mi"
        cpu: "100m"

Apply and check the assigned priority:

kubectl apply -f test-pod.yaml
kubectl get pod test-pod -o jsonpath='{.spec.priority}'

Expected output:

500

Modifying an Existing PriorityClass

If you need to change a value, be aware that existing pods retain the priority they were assigned at admission. Only new pods get the updated value. For example, to change medium-priority to 600:

kubectl patch priorityclass medium-priority -p '{"value":600}'

Then verify:

kubectl get priorityclass medium-priority -o jsonpath='{.value}'

Output:

600

Existing pods using medium-priority will still show their old priority until recreated. Check with:

kubectl get pods -l priorityclass=medium-priority -o custom-columns=NAME:.metadata.name,PRIORITY:.spec.priority

This characteristic is often a source of confusion. Always restart pods to adopt new priority values if needed. For Deployments, a rolling restart can be triggered by:

kubectl rollout restart deployment <deployment-name> -n <namespace>

Scoped Implementation Choices

• Use distinct PriorityClasses for different workload tiers (e.g., high, medium, low) with clear value ranges.
• Avoid setting globalDefault: true unless you have a deliberate default priority for all pods. The default priority for pods without a class is 0.
• Do not modify system-cluster-critical or system-node-critical; they are reserved for Kubernetes system components and have very high values (2,000,000,000 and above). Changing them can break cluster stability.

After making changes, proceed to verify behavior.

Verification and Diagnostics

Verification involves checking pod priority assignment, preemption events, and scheduler logs if needed. We'll walk through common checks.

Confirm Pod Priority Assignment

For any pod, you can check its priority and priorityClassName:

kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.priority} {.spec.priorityClassName}'

Example:

500 medium-priority

If the pod shows no priority, it might be using the global default. Check the global default class:

kubectl get priorityclass -o json | jq '.items[] | select(.globalDefault==true) | .metadata.name'

Simulating Preemption Behavior

To test preemption, create a high-priority pod when the cluster is full. First, fill a node with low-priority pods. Then, submit a high-priority pod and observe events.

Create a low-priority class (if not present) with value 0:

kubectl create priorityclass low-priority --value=0 --global-default=false

Create several pods that request enough CPU to fill a node. For a node with 2 CPUs, you could create two pods each requesting 1 CPU, or one pod requesting 2 CPUs. Example low-priority pod spec:

apiVersion: v1
kind: Pod
metadata:
  name: low-pod-1
spec:
  priorityClassName: low-priority
  containers:
  - name: busybox
    image: busybox
    command: ["sleep", "3600"]
    resources:
      requests:
        cpu: "1000m"

Apply multiple such pods until the node is full. Then, create a high-priority pod that requests more CPU than available:

apiVersion: v1
kind: Pod
metadata:
  name: high-priority-pod
spec:
  priorityClassName: high-priority
  containers:
  - name: busybox
    image: busybox
    command: ["sleep", "3600"]
    resources:
      requests:
        cpu: "1500m"

Note: The high-priority pod must be scheduled on the same node. To force that, you can use a node selector or ensure only one node is available.

Watch events:

kubectl get events --sort-by=.lastTimestamp

Look for messages like Preempted to accommodate higher priority pod. Example event:

LAST SEEN   TYPE     REASON      OBJECT              MESSAGE
2m          Normal   Preempted   pod/low-pod-1       Preempted by high-priority-pod on node node-1

This confirms preemption works.

Checking Scheduler Decisions

If preemption is not happening, inspect the scheduler logs. If you have access to the control plane, you can search for priority-related logs:

kubectl logs -n kube-system <scheduler-pod-name> | grep -i priority

Alternatively, if using a managed cluster (e.g., EKS, GKE, AKS), you may need to rely on events and pod status because control plane logs are not directly accessible. In that case, describe the pending pod and check for scheduler messages.

Diagnose "Pod is Pending" with Priority Issues

A pending pod might not be scheduled because no preemption occurred due to certain conditions. Describe the pod:

kubectl describe pod <pod-name> -n <namespace>

Look for events. Common reasons:

• Insufficient cpu without mention of preemption: The scheduler may not preempt if the pod's priority is not high enough relative to existing pods.
• Preemption is not helpful for scheduling: This happens when the pod's priority is too low to evict any existing pods.

Ensure the pod's priority is actually higher than the pods that would be evicted. For example, if the pending pod has priority 1000 and the running pods have priority 2000, no preemption will occur.

Verify PriorityClass Values Are in Range

Priority values must be within the range allowed by the admission controller. By default, the valid range is 1 to 1,000,000,000 (1 billion). Values above may be rejected. Check pod events for errors like:

Error creating: pods "test" is forbidden: spec.priority: Invalid value: 2000000000: must be less than or equal to 1000000000

If you have custom admission configurations, verify the allowed range. For example, some clusters set a maximum priority lower than 1 billion to prevent accidental high priorities.

Check PriorityClass Binding to Pods

Sometimes a pod references a nonexistent PriorityClass. The pod will be rejected with an error like:

Error from server (Forbidden): error when creating "pod.yaml": pods "test" is forbidden: no PriorityClass with name nonexistent-priority was found

Fix by creating the class or correcting the pod spec.

Common Pitfalls and How to Avoid Them

Priority Class misconfigurations often stem from a few recurring mistakes. Understanding these pitfalls can save you hours of debugging.

Pitfall 1: Assuming Priority Changes Apply to Running Pods

Why it happens: Priority is assigned at pod admission time. When you update a PriorityClass's value, existing pods already have a numeric priority set and do not automatically update.

How to avoid: After modifying a PriorityClass, explicitly restart the affected pods or use a rolling update. For Deployments, use kubectl rollout restart. For standalone pods, delete and recreate them.

Pitfall 2: Setting globalDefault Without Testing

Why it happens: Setting globalDefault: true on a PriorityClass affects every pod that does not explicitly specify a priorityClassName. This can inadvertently give high priority to pods that should be low priority, causing unexpected preemptions.

How to avoid: Never change the global default in production without first testing in a sandbox namespace. Prefer creating new non-default PriorityClasses and assigning them explicitly to pods.

Pitfall 3: Misunderstanding Priority Values

Why it happens: Priority values are arbitrary integers, but their relative magnitude matters. A pod with priority 1000 will not preempt a pod with priority 2000. However, the absolute number does not convey the importance; only comparison matters.

How to avoid: Define a clear priority scale for your organization. For example, use ranges like 0-999 for batch jobs, 1000-9999 for standard services, and 10000+ for critical services. Document this scale.

Pitfall 4: Forgetting Resource Requests

Why it happens: Preemption requires resource requests to determine if evicting lower-priority pods will free enough capacity for the higher-priority pod. If the pending pod has no resource requests, the scheduler may not consider preemption.

How to avoid: Always set resource requests for pods that may need preemption. Without requests, the scheduler assumes zero resource usage and may not preempt even if the node is actually full.

Pitfall 5: Modifying System Priority Classes

Why it happens: It's tempting to adjust system-cluster-critical or system-node-critical to influence scheduling, but these are reserved for system components. Changing their values can disrupt cluster stability.

How to avoid: Never touch these classes. Instead, create your own high-priority classes with values below the system classes (e.g., below 2,000,000,000).

Pitfall 6: Not Testing Preemption in a Controlled Environment

Why it happens: Preemption can cause unexpected evictions in production if not tested. A misconfigured priority can evict critical workloads.

How to avoid: Always test preemption in a dedicated namespace or a test cluster before rolling out priority changes to production. Simulate high-priority pod scheduling and observe which pods get evicted.

Failure Modes and Recovery

Here we cover common failure scenarios and how to recover.

1. Incorrect Priority Value

Symptom: Pods are preempted unexpectedly or not preempted at all.

Diagnosis: Check the pod's priority and compare with the intended PriorityClass. Use kubectl get priorityclass to view values.

Recovery: Update the PriorityClass value. To immediately apply to running pods, restart the pods (e.g., delete them and let the controller recreate). For standalone pods, you need to recreate them.

kubectl delete pod <pod-name> -n <namespace>

2. Global Default Misconfiguration

Symptom: New pods get an unexpected priority. For example, every pod gets priority 1000 instead of 0.

Diagnosis: Check which PriorityClass has globalDefault: true:

kubectl get priorityclass -o json | jq '.items[] | select(.globalDefault==true) | {name: .metadata.name, value: .value}'

Recovery: Set globalDefault: false on that class and optionally set it on another. Remember that only one can be global default. If you disable all, pods without priority get priority 0 (implicit).

kubectl patch priorityclass <name> -p '{"globalDefault":false}'

3. Preemption Not Working as Expected

Symptom: High-priority pods remain pending even though lower-priority pods are running.

Possible Causes:

• The high-priority pod's priority is not higher than the lower-priority pods' priorities. Compute the values.
• The scheduler's preemption logic may be disabled or restricted by feature gates. Check scheduler configuration.
• Resource requests are not set on the pending pod; preemption requires resource requests to determine eviction.
• Node selectors or taints may limit scheduling.

Recovery: Ensure the pending pod has appropriate resource requests and a sufficiently high priority. Then check events:

kubectl describe pod <pending-pod>

Look for messages like Preempting to accommodate higher priority pod. If none, review scheduler logs (if accessible).

4. Pod Cannot Be Created Due to Invalid Priority Class

Symptom: Pod creation fails with error referencing PriorityClass.

Recovery: Create the missing PriorityClass or fix the pod spec. After creation, the pod will be admitted if priority is within range.

5. PriorityClass Deleted While in Use

Symptom: Existing pods still run, but new pods referencing that class fail.

Diagnosis: kubectl get priorityclass shows it missing. Pods with that priorityClassName will fail to be created.

Recovery: Recreate the PriorityClass with the same name and appropriate value. Existing pods will retain their assigned priority, but new ones will work.

Rollback Procedures

Always have a rollback plan. If a change causes issues, you can:

• Revert the PriorityClass manifest to the previous version using kubectl apply.
• Delete the PriorityClass (if it was newly added) and update pods to remove the reference.
• For globalDefault, restore the previous setting.

Document changes beforehand to facilitate quick rollback.

Operations Checklist with Ownership and Review Cadence

Use this checklist as a repeatable procedure when managing Priority Classes. Each step includes a responsible role and a review frequency.

Additional reminders:

• Avoid changing system priority classes.
• Keep priority values within the allowed range.
• Use descriptive names and description fields.
• Test before applying to production.

Conclusion

Priority Class troubleshooting requires methodical verification and careful changes. By following this guide, you can diagnose common issues such as incorrect priorities, preemption failures, and misconfigured global defaults. Start with a narrow pilot, verify behavior with concrete commands, and always plan for rollback. With the checklist and recovery steps, you can maintain reliable pod scheduling and resource preemption in your Kubernetes clusters.

Verified next steps:

• Inventory your current PriorityClasses and pod assignments.
• Implement a new non-default PriorityClass for a test workload.
• Test preemption behavior in a safe namespace.
• Monitor events and scheduler logs for unexpected behavior.
• Gradually roll out priority settings to production workloads with clear rollback plans.

By institutionalizing these practices, you turn Priority Class management into a repeatable operational routine.