## Intro When a Kubernetes cluster runs out of resources, the scheduler must decide which pods to evict to make room for higher-priority workloads. Pod Priority and Preemption is the mechanism that governs these decisions, but it often operates silently until a production incident occurs. Without proper monitoring and alerting, you may discover that critical pods were evicted only after users report downtime. This guide provides a practical approach to monitoring Kubernetes Pod Priority and Preemption. It is designed for developers, DevOps consultants, and technical startup teams who need to move from observing a potential problem to verifying a solution. We connect alerts, metrics, dashboards, and incident response to concrete commands, expected outputs, failure signals, and recovery decisions. Our goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify results, and document recovery paths before an incident forces a decision. Each section includes a version and environment inventory, safe configuration path, verification and diagnostics, failure modes and recovery, and an operations checklist. ## Version and Environment Inventory Before diving into monitoring, establish a clear inventory of your environment. This includes the Kubernetes version, scheduler configuration, existing PriorityClasses, and how pods are assigned priorities. ### Identify the Kubernetes Version and Scheduler First, check your cluster version: ```bash kubectl version --short ``` Expected output (example): ``` Client Version: v1.27.3 Kustomize Version: v5.0.1 Server Version: v1.27.3 ``` Pod Priority and Preemption is enabled by default in Kubernetes 1.14 and later. If you are on an older version, you must enable the `PodPriority` feature gate. For supported versions (1.14+), no additional configuration is needed. Check the kube-scheduler pod to confirm it is running: ```bash kubectl get pods -n kube-system -l component=kube-scheduler ``` Expected output: ``` NAME READY STATUS RESTARTS AGE kube-scheduler-... 1/1 Running 0 5d ``` ### List Existing PriorityClasses PriorityClasses are cluster-scoped resources that define priority values and preemption policies. List them: ```bash kubectl get priorityclasses ``` Example output: ``` NAME VALUE GLOBAL-DEFAULT AGE system-cluster-critical 2000000000 false 30d system-node-critical 2000001000 false 30d high-priority 1000000 false 10d default 0 true 30d ``` Note the `GLOBAL-DEFAULT` column: only one PriorityClass can be the global default. Pods without a `priorityClassName` use the global default (typically value 0). If no global default is set, pods without a priority class have priority 0 but no preemption policy. Inspect a specific PriorityClass: ```bash kubectl describe priorityclass high-priority ``` Expected output: ``` Name: high-priority Value: 1000000 GlobalDefault: false PreemptionPolicy: PreemptLowerPriority Description: For critical applications Annotations: Events: ``` The `PreemptionPolicy` field can be `PreemptLowerPriority` (default) or `Never`. If `Never`, pods with this class will not preempt other pods, but they can still be preempted by higher-priority pods. ### Verify a Pod's Priority Check a running pod's priority: ```bash kubectl get pod -o jsonpath='{.spec.priority}{"\n"}' ``` Example: ```bash kubectl get pod nginx-deployment-7d5b6d4c9-abcde -o jsonpath='{.spec.priority}{"\n"}' ``` Output: ``` 1000000 ``` If no priority is set, it defaults to 0 (or the global default). You can also see the priority class name: ```bash kubectl get pod -o jsonpath='{.spec.priorityClassName}{"\n"}' ``` ### Check Scheduler Configuration for Preemption The kube-scheduler has flags that affect preemption behavior. View the scheduler pod's command: ```bash kubectl get pod -n kube-system -l component=kube-scheduler -o jsonpath='{.items[0].spec.containers[0].command}' | jq . ``` Look for flags such as `--preemption=true` (default) or `--preemption=false`. If preemption is disabled, no pods will be preempted regardless of priority. ### Practical Read-Only Checks Before making any changes, run these read-only commands to capture the current state: ```bash kubectl get pods -A -o wide | grep -v Running ``` This shows any pods not in Running state, which may indicate preemption or eviction issues. ```bash kubectl get events -A --sort-by='.lastTimestamp' | tail -50 ``` Look for events with reasons such as `Preempted`, `FailedScheduling`, or `Evicted`. This is your first signal of preemption activity. ## Safe Configuration Path Configuring priority and preemption properly involves creating PriorityClasses, assigning them to pods, and testing the behavior in a controlled way. Always test changes in a non-production namespace or cluster first. ### 1. Create a PriorityClass Define a PriorityClass with a clear name and value. Higher values mean higher priority. Values must be less than or equal to 1,000,000,000 (1 billion). Example `priority-class.yaml`: ```yaml apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: high-priority value: 1000000 preemptionPolicy: PreemptLowerPriority globalDefault: false description: "For critical applications" ``` Apply it: ```bash kubectl apply -f priority-class.yaml ``` Verify: ```bash kubectl get priorityclass high-priority ``` ### 2. Assign PriorityClass to a Workload Modify a Deployment to use the PriorityClass. Example `nginx-deployment.yaml`: ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment spec: replicas: 2 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: priorityClassName: high-priority containers: - name: nginx image: nginx:1.21 resources: requests: cpu: "100m" memory: "128Mi" ``` Apply and verify priority: ```bash kubectl apply -f nginx-deployment.yaml kubectl get pods -l app=nginx -o wide ``` Check the priority of the new pods: ```bash kubectl get pods -l app=nginx -o jsonpath='{.items[0].spec.priority}{"\n"}' ``` Expected output: `1000000`. ### 3. Test Preemption Behavior Safely To test preemption without affecting production, create a dedicated namespace and simulate resource pressure. Use the `--dry-run=server` flag to validate manifests before applying. First, create a test namespace: ```bash kubectl create namespace preemption-test ``` Deploy a low-priority pod that requests a specific resource, e.g., a full node's CPU. Ensure your cluster has at least one node with limited capacity. You can use a pod with a large CPU request to fill the node: ```yaml apiVersion: v1 kind: Pod metadata: name: low-priority-pod namespace: preemption-test spec: priorityClassName: default # or leave empty for priority 0 containers: - name: stress image: polinux/stress resources: requests: cpu: "1" memory: "1Gi" command: ["stress"] args: ["--cpu", "1"] ``` Apply it and wait until it is running. Then, deploy a high-priority pod that requires the same resources with no preemption policy set to `Never`: ```yaml apiVersion: v1 kind: Pod metadata: name: high-priority-pod namespace: preemption-test spec: priorityClassName: high-priority containers: - name: nginx image: nginx:1.21 resources: requests: cpu: "1" memory: "1Gi" ``` Apply the high-priority pod and watch events: ```bash kubectl apply -f high-priority-pod.yaml kubectl get events -n preemption-test --watch ``` Expected events: ``` 0s Normal Scheduled pod/high-priority-pod Successfully assigned preemption-test/high-priority-pod to node-1 0s Normal Preempted pod/low-priority-pod Preempted by preemption-test/high-priority-pod on node node-1 ``` The low-priority pod will be evicted and rescheduled elsewhere if resources are available. This confirms preemption works. ### 4. Rollback and Cleanup After testing, delete the test namespace to clean up: ```bash kubectl delete namespace preemption-test ``` If a configuration change causes issues, you can revert by applying the previous manifest or deleting the PriorityClass after ensuring no pods use it. ## Verification and Diagnostics Once you have priority and preemption configured, you need to verify that it is working as expected and diagnose any issues. This section provides commands and metrics to monitor the behavior. ### Verify Preemption Events Use `kubectl get events` filtered by reason: ```bash kubectl get events -A --field-selector reason=Preempted ``` Example output: ``` NAMESPACE LAST SEEN TYPE REASON OBJECT MESSAGE preemption-test 3m Normal Preempted pod/low-priority-pod Preempted by preemption-test/high-priority-pod on node node-1 ``` If you see no events, preemption may not be occurring. Check that the high-priority pod has a higher priority value than the low-priority pod and that the high-priority pod's `preemptionPolicy` is not `Never`. ### Check Scheduler Logs Scheduler logs can reveal preemption decisions. Access the scheduler pod logs: ```bash kubectl logs -n kube-system -l component=kube-scheduler --tail=100 ``` Search for lines containing "preemption" or "Preempting": ```bash kubectl logs -n kube-system -l component=kube-scheduler --tail=100 | grep -i preempt ``` Example log line: ``` I0615 10:23:45.123456 1 preemption.go:276] Preempting 1 pod(s) to make room for high-priority-pod ``` If logs are not available due to log rotation, consider enabling scheduler metrics (below). ### Monitor Scheduler Metrics Enable scheduler metrics by adding `--authorization-always-allow-paths=/metrics` and `--bind-address=0.0.0.0` to the kube-scheduler command. Then scrape the metrics endpoint: ```bash kubectl get --raw /api/v1/namespaces/kube-system/pods/kube-scheduler-:10251/proxy/metrics | grep scheduler_preemption ``` Replace `` with your scheduler pod's node. Expected metrics (example): ``` scheduler_preemption_attempts_total{result="success"} 2 scheduler_preemption_victims_total 3 ``` Key metrics to watch: - `scheduler_preemption_attempts_total`: Number of preemption attempts, labeled by result (success, failure). - `scheduler_preemption_victims_total`: Number of pods preempted. - `scheduler_schedule_attempts_total`: Total scheduling attempts. These metrics help you build dashboards and alerts (see next sections). ### Diagnose Scheduling Failures If a high-priority pod is pending, describe it: ```bash kubectl describe pod -n ``` Look for events such as: ``` Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 2m default-scheduler 0/3 nodes are available: 3 Insufficient cpu, 3 Insufficient memory. ``` This indicates that even with preemption, there are insufficient resources. Check if preemption was attempted by looking for `Preempted` events or scheduler logs. Another useful command: ```bash kubectl get events -n --sort-by='.lastTimestamp' ``` ## Failure Modes and Recovery Preemption can lead to various failure modes. Understanding these and having a recovery plan is crucial for maintaining application availability. ### Common Failure Modes 1. **Unintended Preemption**: Low-priority pods are preempted more often than expected, causing service disruption. This often happens when priorities are misconfigured (e.g., a batch job gets a high priority). - **Recovery**: Review PriorityClass assignments and adjust values. Use `kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.priorityClassName}{"\n"}{end}'` to audit. 2. **Preemption Not Triggering**: High-priority pods remain pending because preemption is disabled or priority classes are incorrect. - **Recovery**: Check scheduler flags (`--preemption=true`), verify PriorityClass values, and ensure `preemptionPolicy` is not `Never`. 3. **Pod Eviction Loop**: A pod is preempted and rescheduled to another node, only to be preempted again, causing a crash loop. - **Recovery**: Adjust resource requests to fit available nodes, or increase cluster capacity. Use pod disruption budgets to limit voluntary disruptions for critical pods. 4. **Priority Inflation**: Over time, many pods get high priority values, reducing the effectiveness of preemption. - **Recovery**: Establish a priority hierarchy document and enforce it via policy (e.g., OPA or Kyverno). ### Step-by-Step Recovery for a Preempted Critical Pod If a critical pod was preempted unintentionally, follow these steps: 1. Identify the preempted pod: ```bash kubectl get events -A --field-selector reason=Preempted | grep ``` 2. Check its current status: ```bash kubectl get pod -n -o wide ``` 3. If it is pending or crash-looping, describe it: ```bash kubectl describe pod -n ``` 4. Raise its priority by assigning a higher PriorityClass (if justified). Create or update a PriorityClass, then patch the pod's owner (e.g., Deployment): ```bash kubectl patch deployment -n -p '{"spec":{"template":{"spec":{"priorityClassName":"system-cluster-critical"}}}}' ``` 5. Alternatively, scale down lower-priority workloads to free resources. 6. Monitor the pod to ensure it becomes Ready: ```bash kubectl rollout status deployment/ -n ``` ### Preventing Future Incidents - Set PodDisruptionBudgets for critical workloads: ```yaml apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: nginx-pdb spec: minAvailable: 1 selector: matchLabels: app: nginx ``` - Use ResourceQuota and LimitRange to prevent resource hogging. - Regularly audit PriorityClasses: ```bash kubectl get priorityclasses -o custom-columns=NAME:.metadata.name,VALUE:.value,GLOBAL:.globalDefault,PREEMPTION:.preemptionPolicy ``` ## Operations Checklist Use this checklist to ensure you have comprehensive monitoring and alerting for pod priority and preemption. ### Monitoring Setup - [ ] Confirm PodPriority feature gate is enabled (default in K8s 1.14+). Run: `kubectl version --short` and check version. - [ ] List and document all PriorityClasses: `kubectl get priorityclasses`. - [ ] Enable scheduler metrics by ensuring `--bind-address=0.0.0.0` and `--authorization-always-allow-paths=/metrics` are set. Verify by running: `kubectl get --raw /api/v1/namespaces/kube-system/pods/kube-scheduler-:10251/proxy/metrics | grep scheduler_preemption`. - [ ] Set up a metrics scraper (Prometheus) to collect scheduler metrics. ### Alerting Rules Create alerts based on scheduler metrics. Example Prometheus alert rules: ```yaml groups: - name: preemption-alerts rules: - alert: HighPreemptionRate expr: rate(scheduler_preemption_attempts_total{result="success"}[5m]) > 0.1 for: 10m labels: severity: warning annotations: summary: "High preemption rate in cluster" description: "Preemption is occurring frequently ({{ $value }} per second), possibly causing instability." - alert: PreemptionFailures expr: increase(scheduler_preemption_attempts_total{result="failure"}[15m]) > 0 labels: severity: critical annotations: summary: "Preemption failures detected" description: "Scheduler failed to preempt pods for scheduling." ``` - [ ] Configure alert notifications (e.g., Slack, PagerDuty). ### Dashboard Panels In Grafana (or similar), create panels for: - Preemption attempts rate: `rate(scheduler_preemption_attempts_total[5m])` - Preemption victims count: `scheduler_preemption_victims_total` - Pending pods with high priority: `kube_pod_status_phase{phase="Pending"} * on(pod) group_left(priority_class) kube_pod_info` (requires kube-state-metrics). - Node resource pressure: `sum by (node) (kube_node_status_capacity{resource="cpu"}) - sum by (node) (kube_pod_container_resource_requests{resource="cpu"})` ### Incident Response Runbook 1. On alert, verify preemption events: `kubectl get events -A --field-selector reason=Preempted --sort-by='.lastTimestamp'`. 2. Identify preempted pods and their owners. 3. Check cluster resource utilization: `kubectl top nodes`. 4. If critical pods are affected, adjust priorities or scale resources. 5. Document incident and review priority classes. ## Conclusion Monitoring Kubernetes Pod Priority and Preemption is essential for maintaining application stability. By understanding how the scheduler makes preemption decisions and observing key metrics, you can prevent unexpected evictions and respond quickly to incidents. We covered the full lifecycle: from inventorying your environment and safely configuring priorities, to verifying preemption behavior, diagnosing failures, and establishing operational checklists with alerts and dashboards. As a next step, choose one low-risk verification from this guide. For example, list your PriorityClasses and audit a few critical deployments to ensure they have appropriate priorities. Then, enable scheduler metrics and create a simple alert for preemption attempts. This will give you immediate visibility into scheduler behavior. Remember that preemption is a powerful but potentially disruptive feature. Always test changes in a controlled environment, document your priority hierarchy, and have recovery procedures ready. With the right monitoring and alerting, you can harness pod priority and preemption to improve resource utilization without sacrificing reliability.