## Intro Kubernetes PriorityClass configuration mistakes with practical examples help operators move from an observed problem to a verified result. Start by identifying the installed version, deployment topology, prerequisites, and the exact component being inspected. This article focuses on Kubernetes PriorityClass configuration for developers, DevOps consultants, and technical startup teams. It connects PriorityClass configuration mistakes, validation, rollback, and troubleshooting to commands, expected output, failure signals, and recovery decisions that match the selected technology. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached. ## Version and Environment Inventory For Kubernetes PriorityClass configuration, the Version and Environment Inventory should name the relevant component, the supported version range, prerequisites, a read-only observation, the smallest justified change, and the command or signal that verifies the outcome. Within Version and Environment Inventory, separate observation from intervention. Capture current state and timestamps first, protect credentials and private material, then change one scoped item only when its blast radius and recovery path are understood. ### What are PriorityClasses? PriorityClass objects are non-namespaced resources that assign an integer value to a priority name. Pods reference a PriorityClass by name in their priorityClassName field. The scheduler uses this priority to decide which pods to preempt when the cluster runs out of capacity, and it also affects the order in which pods are admitted and evicted. A PriorityClass has two key fields: - value : A 32-bit integer. Higher values mean higher priority. This value must be less than or equal to 1,000,000,000 (one billion). - globalDefault : A boolean. If set to true on one PriorityClass, it becomes the default for pods that do not specify a priorityClassName . Only one PriorityClass can have globalDefault: true . Additionally, PriorityClasses can have a preemptionPolicy field, set to PreemptLowerPriority (default) or Never . The Never policy prevents pods with this PriorityClass from preempting other pods. Example PriorityClass manifest ( high-priority.yaml ): apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: high-priority value: 1000 globalDefault: false preemptionPolicy: PreemptLowerPriority description: "Use for critical services that need fast scheduling" Apply it with kubectl apply -f high-priority.yaml , then verify with kubectl get priorityclass high-priority -o yaml . ### Cluster Version Check PriorityClasses are stable in Kubernetes v1.14 and later. However, certain features, such as default preemption behavior, may differ in older versions. Always check your cluster version: kubectl version --short Expected output includes the client and server version, for example: Client Version: v1.27.2 Kustomize Version: v5.0.1 Server Version: v1.27.2 If the server version is below 1.14, PriorityClass is not available as a stable API (it was beta since 1.11). Upgrade or use the appropriate beta API version scheduling.k8s.io/v1beta1 for older clusters. ### Environment Inventory Before modifying PriorityClasses, gather the current state: kubectl get priorityclass kubectl get nodes -o wide kubectl get pods -A -o wide --field-selector=status.phase=Running Store the output with a timestamp for later comparison: date > pre-change-state.txt kubectl get priorityclass -o yaml >> pre-change-state.txt kubectl get nodes -o wide >> pre-change-state.txt This read-only observation forms a snapshot for troubleshooting and rollback. ### Prerequisites - A running Kubernetes cluster (v1.14+ recommended). - kubectl configured with appropriate RBAC permissions. You need create , update , delete on priorityclasses.scheduling.k8s.io and pods if you plan to test pod behavior. - At least two namespaces or two default pods to test preemption safely. Check permissions with: kubectl auth can-i create priorityclasses kubectl auth can-i delete priorityclasses Expected output: yes for both if permissions are sufficient. ### Blast Radius and Recovery Path Changing or deleting a PriorityClass can cause immediate preemption or eviction of pods referencing it. Always understand which pods use a PriorityClass before altering it: kubectl get pods -A -o json | jq -r '.items[] | select(.spec.priorityClassName != null) | "\(.metadata.namespace)/\(.metadata.name): \(.spec.priorityClassName)"' This lists all pods with a priority class. If a PriorityClass is deleted while pods still reference it, those pods cannot be scheduled until the PriorityClass is recreated or the pod is updated. To recover, recreate the PriorityClass with the same name and value. Practical Kubernetes check for Version and Environment Inventory: start with kubectl get pods -o wide , then use kubectl describe pod for scheduling and event details, kubectl logs --previous for crash loops, and kubectl rollout status deployment/ before assuming a release succeeded. Keep the local test small. Apply one manifest, inspect the generated resources, and verify traffic with kubectl port-forward or a local service type before moving to a cloud load balancer or ingress controller.
Quick check 1 of 2 What are the two built-in PriorityClasses provided by Kubernetes? - system-cluster-critical and system-node-critical - high-priority and low-priority - default-priority and high-priority - critical and normal
Kubernetes provides two built-in PriorityClasses: system-cluster-critical for system components critical to the cluster, and system-node-critical for system components critical to individual nodes.
## Safe Configuration Path For Kubernetes PriorityClass configuration, the Safe Configuration Path should name the relevant component, the supported version range, prerequisites, a read-only observation, the smallest justified change, and the command or signal that verifies the outcome. Within the Safe Configuration Path, separate observation from intervention. Capture current state and timestamps first, protect credentials and private material, then change one scoped item only when its blast radius and recovery path are understood. ### Step 1: Define Clear Priority Tiers A common mistake is creating too many PriorityClasses with arbitrary values, leading to unpredictable preemption. Instead, define a small set of tiers:
PriorityClass nameValueIntended use
system-cluster-critical2000000000 (built-in, not user-defined)Kubernetes system pods
high-priority1000Production critical microservices
medium-priority100Normal workloads
low-priority10Batch jobs, development pods
Keep values separated by at least a factor of 10 to allow future insertion. Document each class in a versioned manifest. Example medium-priority.yaml : apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: medium-priority value: 100 globalDefault: true description: "Default priority class for all namespaces" Setting globalDefault: true on medium-priority means pods without priorityClassName get value 100. Note: only one PriorityClass can be globalDefault: true . To change the default, set globalDefault: false on the current default class before applying a new one. ### Step 2: Apply Changes in a Controlled Manner Always apply changes in a scoped, reversible way: - Make a backup of the current PriorityClass: kubectl get priorityclass high-priority -o yaml > high-priority-backup.yaml - Apply the changed manifest: kubectl apply -f high-priority.yaml - Observe the result: kubectl get priorityclass high-priority -o yaml Expected output includes the new value field. If the apply fails, check for validation errors such as value out of range or duplicate globalDefault . ### Step 3: Test with a Canary Pod Before assigning a new PriorityClass to production pods, test with a canary pod in a separate namespace: # canary-pod.yaml apiVersion: v1 kind: Pod metadata: name: canary-high namespace: test spec: priorityClassName: high-priority containers: - name: nginx image: nginx:1.21 resources: requests: cpu: "50m" memory: "64Mi" Apply and check: kubectl apply -f canary-pod.yaml kubectl get pod canary-high -n test -o yaml | grep priorityClassName Expected output: priorityClassName: high-priority If the pod is in Pending , the PriorityClass may not exist or the pod may lack sufficient resources. ### Step 4: Verify Preemption Behavior To verify preemption, create two pods: one low priority and one high priority, with limited node capacity. For example, if a node has 1 CPU and the low-priority pod requests 0.9 CPU, applying a high-priority pod requesting 0.5 CPU should preempt the low-priority pod. Here is a practical sequence: - Create a low-priority pod that requests most of the node's resources: # low-pod.yaml apiVersion: v1 kind: Pod metadata: name: low-pod spec: priorityClassName: low-priority containers: - name: busybox image: busybox command: ["sleep", "3600"] resources: requests: cpu: "900m" memory: "100Mi" - Apply it and wait until it is Running . Then create a high-priority pod: # high-pod.yaml apiVersion: v1 kind: Pod metadata: name: high-pod spec: priorityClassName: high-priority containers: - name: nginx image: nginx resources: requests: cpu: "500m" memory: "100Mi" - Apply the high-priority pod and observe: kubectl apply -f high-pod.yaml kubectl get pods -w Expected: low-pod is preempted (status Terminating ), and high-pod becomes Running . This verifies that the priority values work as intended. If the low-priority pod is not preempted, check that the PriorityClass values are correct, the preemptionPolicy on the high-priority class is not Never , and the scheduler has the PodPriority feature gate enabled (only for older clusters). Keep the local test small. Preemption can disrupt cluster state, so test in a dedicated namespace or on a non-production cluster first. ## Verification and Diagnostics For Kubernetes PriorityClass configuration, Verification and Diagnostics should name the relevant component, the supported version range, prerequisites, a read-only observation, the smallest justified change, and the command or signal that verifies the outcome. Within Verification and Diagnostics, separate observation from intervention. Capture current state and timestamps first, protect credentials and private material, then change one scoped item only when its blast radius and recovery path are understood. ### Diagnosing PriorityClass Issues Common symptoms and their causes:
SymptomLikely causeDiagnostic command
Pod stuck in Pending with event "no PriorityClass with name X found"PriorityClass deleted or renamedkubectl describe pod <pod-name>
Pod preempted unexpectedlyHigh-priority pod scheduled, low-priority pod had lower prioritykubectl get events --sort-by='.lastTimestamp'
High-priority pod cannot preemptpreemptionPolicy: Never on its PriorityClasskubectl get priorityclass <name> -o yaml
Pod gets default priority 0No globalDefault set, pod lacks priorityClassNamekubectl get pod <pod-name> -o yaml | grep priorityClassName
Example: To see why a pod is pending, use: kubectl describe pod high-pod -n test Look under Events : Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 2m default-scheduler 0/1 nodes are available: 1 Insufficient cpu, 1 node(s) didn't match Pod's node affinity/selector. If the event mentions "no PriorityClass with name", the PriorityClass is missing. ### Validating PriorityClass Objects Use kubectl apply --dry-run=client to validate syntax without changing the cluster: kubectl apply -f high-priority.yaml --dry-run=client Expected output: priorityclass.scheduling.k8s.io/high-priority created (dry run) For deeper validation, use kubectl apply --dry-run=server (available in Kubernetes 1.18+): kubectl apply -f high-priority.yaml --dry-run=server If there is a validation error, such as a value exceeding 1,000,000,000, the server returns an error like: The PriorityClass "high-priority" is invalid: value: Invalid value: 2000000000: must be less than or equal to 1000000000 ### Auditing Pod Priorities List all pods with their priority values: kubectl get pods -A -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,PRIORITY:.spec.priority,PRIORITY_CLASS:.spec.priorityClassName Example output: NAMESPACE NAME PRIORITY PRIORITY_CLASS default frontend 0 kube-system coredns-... 2000000000 system-cluster-critical Note: Pods without a priority class show priority 0 and . ### Diagnostic Commands - kubectl get events -A --sort-by='.lastTimestamp' to view recent preemption events. - kubectl logs --previous to check for crash loops unrelated to priority. - kubectl rollout status deployment/ to verify a deployment that includes priority class changes. Always compare these diagnostics against the pre-change state captured earlier.
Quick check 2 of 2 Which command creates a PriorityClass named 'high-priority' that cannot preempt pods with lower priority? - kubectl create priorityclass high-priority --value=1000 --description="high priority" --preemption-policy="Never" - kubectl create priorityclass high-priority --value=1000 --description="high priority" --preemption-policy="PreemptLowerPriority" - kubectl create priorityclass high-priority --value=1000 --global-default=true --description="high priority" - kubectl create priorityclass high-priority --value=1000 --description="high priority"
The example shows that to create a high-priority class that cannot preempt lower priority pods, you must include the --preemption-policy="Never" flag.
## Failure Modes and Recovery For Kubernetes PriorityClass configuration, Failure Modes and Recovery should name the relevant component, the supported version range, prerequisites, a read-only observation, the smallest justified change, and the command or signal that verifies the outcome. Within Failure Modes and Recovery, separate observation from intervention. Capture current state and timestamps first, protect credentials and private material, then change one scoped item only when its blast radius and recovery path are understood. ### Failure Mode 1: Deleting a PriorityClass in Use If you delete a PriorityClass that is still referenced by pods, those pods cannot be scheduled (new pods) and may be evicted if the kubelet is restarted. To recover: - Recreate the PriorityClass from backup or from memory: kubectl create -f high-priority-backup.yaml - Verify it is restored: kubectl get priorityclass high-priority - Check for any pods still in error state: kubectl get pods -A | grep -v Running | grep -v Completed ### Failure Mode 2: Setting a PriorityClass as globalDefault Accidentally If you set globalDefault: true on a high-priority class, all new pods without a priorityClassName will inherit that high priority, which can lead to unexpected preemption of important workloads. To recover: kubectl patch priorityclass high-priority -p '{"globalDefault":false}' Then ensure a lower priority class (or none) is the default. If you had a previous default, set it back: kubectl patch priorityclass medium-priority -p '{"globalDefault":true}' Verify: kubectl get priorityclass -o custom-columns=NAME:.metadata.name,VALUE:.value,GLOBAL_DEFAULT:.globalDefault ### Failure Mode 3: Preemption Causing Application Outage If a high-priority pod preempts critical low-priority pods, the cluster may see a spike in pod restarts. To mitigate: - Identify which pods were preempted by checking events: kubectl get events -A | grep Preempted Example event: 5m Warning Preempted pod/low-pod Preempted by pod/high-pod on node node1 - If the preemption was unintended, adjust the preemption policy on the high-priority class: kubectl patch priorityclass high-priority -p '{"preemptionPolicy":"Never"}' Note: This prevents any pods using this class from preempting others; they will wait for resources instead. - Alternatively, lower the priority value of the offending class: kubectl patch priorityclass high-priority -p '{"value":50}' ### Failure Mode 4: PriorityClass Name Typo A common mistake is a typo in the priorityClassName in a pod spec. The pod will fail to schedule if the class does not exist. Diagnose: kubectl describe pod | grep -A5 Events Look for: Warning FailedScheduling 2m default-scheduler 0/1 nodes are available: 1 no PriorityClass with name high-priorty found. Fix by correcting the pod spec or creating the PriorityClass with the expected name. ### Recovery Checklist - Always keep backups of PriorityClass YAMLs in version control. - Before deleting, run kubectl get pods -A -o json | jq '.items[] | select(.spec.priorityClassName=="")' to see affected pods. - Use kubectl apply with --dry-run=server to catch errors. - Have a rollback plan: re-apply the previous PriorityClass manifest. ## Operations Checklist For Kubernetes PriorityClass configuration, the Operations Checklist should name the relevant component, the supported version range, prerequisites, a read-only observation, the smallest justified change, and the command or signal that verifies the outcome. Within the Operations Checklist, separate observation from intervention. Capture current state and timestamps first, protect credentials and private material, then change one scoped item only when its blast radius and recovery path are understood. ### Pre-Change Checklist - [ ] Check cluster version: kubectl version --short , verify server >= v1.14. - [ ] List existing PriorityClasses: kubectl get priorityclass -o wide . - [ ] Identify pods using each PriorityClass: kubectl get pods -A -o json | jq '.items[] | select(.spec.priorityClassName != null) | "\(.metadata.namespace)/\(.metadata.name): \(.spec.priorityClassName)"' . - [ ] Capture current state to a file: kubectl get priorityclass -o yaml > priorityclass-snapshot-$(date +%Y%m%d).yaml . - [ ] Validate RBAC permissions: kubectl auth can-i update priorityclasses . - [ ] Prepare rollback manifest for the target PriorityClass. ### Change Execution Checklist - [ ] Apply change with kubectl apply -f .yaml --dry-run=server first to catch validation errors. - [ ] If dry-run succeeds, apply for real: kubectl apply -f .yaml . - [ ] Immediately verify the object: kubectl get priorityclass -o yaml and check value , globalDefault , preemptionPolicy . - [ ] For pod changes, apply one pod at a time and observe with kubectl get pods -o wide . - [ ] Check scheduler events: kubectl get events --sort-by='.lastTimestamp' | grep -i preempt . ### Post-Change Verification Checklist - [ ] Confirm expected pods are running and at the correct priority: kubectl get pods -o custom-columns=POD:.metadata.name,PRIORITY:.spec.priority,PRIORITY_CLASS:.spec.priorityClassName . - [ ] Monitor for unintended preemptions for at least 10 minutes. - [ ] If any issues, revert using the backup manifest. - [ ] Document the change and outcome in your operations log. ### Common Pitfalls and How to Avoid Them - Too many PriorityClasses : Keep the number small (3-5). Use kubectl get priorityclass to review. - No globalDefault set : Pods without a priority class get value 0. Decide if you want a default. - Ignoring preemption policy : If you never want a class to preempt, set preemptionPolicy: Never . - Changing values on live clusters : This can cause immediate preemption; treat as a change with a rollback plan. Implement this checklist as a pre-commit hook or CI step if PriorityClasses are managed via GitOps. ## Conclusion Kubernetes PriorityClass configuration mistakes with practical examples is useful only when each recommendation is version-scoped, observable, and reversible where the technology permits. Copying a command without checking prerequisites and expected output is not an operations procedure. As a next step, choose one low-risk verification for Kubernetes PriorityClass configuration: record the current state, run the documented check, compare the result with the expected signal, and review dependencies such as Pod, Node, and Resource Quota (which can also affect scheduling and preemption). A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision.