Kubernetes Pod Priority and Preemption: A Practical Monitoring and Alerting Guide

Intro

When a Kubernetes cluster runs out of resources, the scheduler must decide which pods to evict to make room for higher-priority workloads. Pod Priority and Preemption is the mechanism that governs these decisions, but it often operates silently until a production incident occurs. Without proper monitoring and alerting, you may discover that critical pods were evicted only after users report downtime.

This guide provides a practical approach to monitoring Kubernetes Pod Priority and Preemption. It is designed for developers, DevOps consultants, and technical startup teams who need to move from observing a potential problem to verifying a solution. We connect alerts, metrics, dashboards, and incident response to concrete commands, expected outputs, failure signals, and recovery decisions.

Our goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify results, and document recovery paths before an incident forces a decision. Each section includes a version and environment inventory, safe configuration path, verification and diagnostics, failure modes and recovery, and an operations checklist.

Version and Environment Inventory

Before diving into monitoring, establish a clear inventory of your environment. This includes the Kubernetes version, scheduler configuration, existing PriorityClasses, and how pods are assigned priorities.

Identify the Kubernetes Version and Scheduler

First, check your cluster version:

kubectl version --short

Expected output (example):

Client Version: v1.27.3
Kustomize Version: v5.0.1
Server Version: v1.27.3

Pod Priority and Preemption is enabled by default in Kubernetes 1.14 and later. If you are on an older version, you must enable the PodPriority feature gate. For supported versions (1.14+), no additional configuration is needed.

Check the kube-scheduler pod to confirm it is running:

kubectl get pods -n kube-system -l component=kube-scheduler

Expected output:

NAME                 READY   STATUS    RESTARTS   AGE
kube-scheduler-...   1/1     Running   0          5d

List Existing PriorityClasses

PriorityClasses are cluster-scoped resources that define priority values and preemption policies. List them:

kubectl get priorityclasses

Example output:

NAME                      VALUE        GLOBAL-DEFAULT   AGE
system-cluster-critical   2000000000   false            30d
system-node-critical      2000001000   false            30d
high-priority             1000000      false            10d
default                   0            true             30d

Note the GLOBAL-DEFAULT column: only one PriorityClass can be the global default. Pods without a priorityClassName use the global default (typically value 0). If no global default is set, pods without a priority class have priority 0 but no preemption policy.

Inspect a specific PriorityClass:

kubectl describe priorityclass high-priority

Expected output:

Name:           high-priority
Value:          1000000
GlobalDefault:  false
PreemptionPolicy: PreemptLowerPriority
Description:    For critical applications
Annotations:    <none>
Events:         <none>

The PreemptionPolicy field can be PreemptLowerPriority (default) or Never. If Never, pods with this class will not preempt other pods, but they can still be preempted by higher-priority pods.

Verify a Pod's Priority

Check a running pod's priority:

kubectl get pod <pod-name> -o jsonpath='{.spec.priority}{"\n"}'

Example:

kubectl get pod nginx-deployment-7d5b6d4c9-abcde -o jsonpath='{.spec.priority}{"\n"}'

Output:

1000000

If no priority is set, it defaults to 0 (or the global default). You can also see the priority class name:

kubectl get pod <pod-name> -o jsonpath='{.spec.priorityClassName}{"\n"}'

Check Scheduler Configuration for Preemption

The kube-scheduler has flags that affect preemption behavior. View the scheduler pod's command:

kubectl get pod -n kube-system -l component=kube-scheduler -o jsonpath='{.items[0].spec.containers[0].command}' | jq .

Look for flags such as --preemption=true (default) or --preemption=false. If preemption is disabled, no pods will be preempted regardless of priority.

Practical Read-Only Checks

Before making any changes, run these read-only commands to capture the current state:

kubectl get pods -A -o wide | grep -v Running

This shows any pods not in Running state, which may indicate preemption or eviction issues.

kubectl get events -A --sort-by='.lastTimestamp' | tail -50

Look for events with reasons such as Preempted, FailedScheduling, or Evicted. This is your first signal of preemption activity.

Safe Configuration Path

Configuring priority and preemption properly involves creating PriorityClasses, assigning them to pods, and testing the behavior in a controlled way. Always test changes in a non-production namespace or cluster first.

1. Create a PriorityClass

Define a PriorityClass with a clear name and value. Higher values mean higher priority. Values must be less than or equal to 1,000,000,000 (1 billion).

Example priority-class.yaml:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 1000000
preemptionPolicy: PreemptLowerPriority
globalDefault: false
description: "For critical applications"

Apply it:

kubectl apply -f priority-class.yaml

Verify:

kubectl get priorityclass high-priority

2. Assign PriorityClass to a Workload

Modify a Deployment to use the PriorityClass. Example nginx-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      priorityClassName: high-priority
      containers:
      - name: nginx
        image: nginx:1.21
        resources:
          requests:
            cpu: "100m"
            memory: "128Mi"

Apply and verify priority:

kubectl apply -f nginx-deployment.yaml
kubectl get pods -l app=nginx -o wide

Check the priority of the new pods:

kubectl get pods -l app=nginx -o jsonpath='{.items[0].spec.priority}{"\n"}'

Expected output: 1000000.

3. Test Preemption Behavior Safely

To test preemption without affecting production, create a dedicated namespace and simulate resource pressure. Use the --dry-run=server flag to validate manifests before applying.

First, create a test namespace:

kubectl create namespace preemption-test

Deploy a low-priority pod that requests a specific resource, e.g., a full node's CPU. Ensure your cluster has at least one node with limited capacity. You can use a pod with a large CPU request to fill the node:

apiVersion: v1
kind: Pod
metadata:
  name: low-priority-pod
  namespace: preemption-test
spec:
  priorityClassName: default  # or leave empty for priority 0
  containers:
  - name: stress
    image: polinux/stress
    resources:
      requests:
        cpu: "1"
        memory: "1Gi"
    command: ["stress"]
    args: ["--cpu", "1"]

Apply it and wait until it is running. Then, deploy a high-priority pod that requires the same resources with no preemption policy set to Never:

apiVersion: v1
kind: Pod
metadata:
  name: high-priority-pod
  namespace: preemption-test
spec:
  priorityClassName: high-priority
  containers:
  - name: nginx
    image: nginx:1.21
    resources:
      requests:
        cpu: "1"
        memory: "1Gi"

Apply the high-priority pod and watch events:

kubectl apply -f high-priority-pod.yaml
kubectl get events -n preemption-test --watch

Expected events:

0s          Normal    Scheduled                pod/high-priority-pod   Successfully assigned preemption-test/high-priority-pod to node-1
0s          Normal    Preempted                pod/low-priority-pod    Preempted by preemption-test/high-priority-pod on node node-1

The low-priority pod will be evicted and rescheduled elsewhere if resources are available. This confirms preemption works.

4. Rollback and Cleanup

After testing, delete the test namespace to clean up:

kubectl delete namespace preemption-test

If a configuration change causes issues, you can revert by applying the previous manifest or deleting the PriorityClass after ensuring no pods use it.

Verification and Diagnostics

Once you have priority and preemption configured, you need to verify that it is working as expected and diagnose any issues. This section provides commands and metrics to monitor the behavior.

Verify Preemption Events

Use kubectl get events filtered by reason:

kubectl get events -A --field-selector reason=Preempted

Example output:

NAMESPACE        LAST SEEN   TYPE     REASON      OBJECT               MESSAGE
preemption-test  3m          Normal   Preempted   pod/low-priority-pod  Preempted by preemption-test/high-priority-pod on node node-1

If you see no events, preemption may not be occurring. Check that the high-priority pod has a higher priority value than the low-priority pod and that the high-priority pod's preemptionPolicy is not Never.

Check Scheduler Logs

Scheduler logs can reveal preemption decisions. Access the scheduler pod logs:

kubectl logs -n kube-system -l component=kube-scheduler --tail=100

Search for lines containing "preemption" or "Preempting":

kubectl logs -n kube-system -l component=kube-scheduler --tail=100 | grep -i preempt

Example log line:

I0615 10:23:45.123456       1 preemption.go:276] Preempting 1 pod(s) to make room for high-priority-pod

If logs are not available due to log rotation, consider enabling scheduler metrics (below).

Monitor Scheduler Metrics

Enable scheduler metrics by adding --authorization-always-allow-paths=/metrics and --bind-address=0.0.0.0 to the kube-scheduler command. Then scrape the metrics endpoint:

kubectl get --raw /api/v1/namespaces/kube-system/pods/kube-scheduler-<node>:10251/proxy/metrics | grep scheduler_preemption

Replace <node> with your scheduler pod's node. Expected metrics (example):

scheduler_preemption_attempts_total{result="success"} 2
scheduler_preemption_victims_total 3

Key metrics to watch:

• scheduler_preemption_attempts_total: Number of preemption attempts, labeled by result (success, failure).
• scheduler_preemption_victims_total: Number of pods preempted.
• scheduler_schedule_attempts_total: Total scheduling attempts.

These metrics help you build dashboards and alerts (see next sections).

Diagnose Scheduling Failures

If a high-priority pod is pending, describe it:

kubectl describe pod <high-priority-pod> -n <namespace>

Look for events such as:

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  2m    default-scheduler  0/3 nodes are available: 3 Insufficient cpu, 3 Insufficient memory.

This indicates that even with preemption, there are insufficient resources. Check if preemption was attempted by looking for Preempted events or scheduler logs.

Another useful command:

kubectl get events -n <namespace> --sort-by='.lastTimestamp'

Failure Modes and Recovery

Preemption can lead to various failure modes. Understanding these and having a recovery plan is crucial for maintaining application availability.

Common Failure Modes

1. Unintended Preemption: Low-priority pods are preempted more often than expected, causing service disruption. This often happens when priorities are misconfigured (e.g., a batch job gets a high priority).

• Recovery: Review PriorityClass assignments and adjust values. Use kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.priorityClassName}{"\n"}{end}' to audit.

1. Preemption Not Triggering: High-priority pods remain pending because preemption is disabled or priority classes are incorrect.

• Recovery: Check scheduler flags (--preemption=true), verify PriorityClass values, and ensure preemptionPolicy is not Never.

1. Pod Eviction Loop: A pod is preempted and rescheduled to another node, only to be preempted again, causing a crash loop.

• Recovery: Adjust resource requests to fit available nodes, or increase cluster capacity. Use pod disruption budgets to limit voluntary disruptions for critical pods.

1. Priority Inflation: Over time, many pods get high priority values, reducing the effectiveness of preemption.

• Recovery: Establish a priority hierarchy document and enforce it via policy (e.g., OPA or Kyverno).

Step-by-Step Recovery for a Preempted Critical Pod

If a critical pod was preempted unintentionally, follow these steps:

1. Identify the preempted pod:

kubectl get events -A --field-selector reason=Preempted | grep <pod-name-prefix>

1. Check its current status:

kubectl get pod <pod-name> -n <namespace> -o wide

1. If it is pending or crash-looping, describe it:

kubectl describe pod <pod-name> -n <namespace>

1. Raise its priority by assigning a higher PriorityClass (if justified). Create or update a PriorityClass, then patch the pod's owner (e.g., Deployment):

kubectl patch deployment <deployment-name> -n <namespace> -p '{"spec":{"template":{"spec":{"priorityClassName":"system-cluster-critical"}}}}'

1. Alternatively, scale down lower-priority workloads to free resources.

1. Monitor the pod to ensure it becomes Ready:

kubectl rollout status deployment/<deployment-name> -n <namespace>

Preventing Future Incidents

• Set PodDisruptionBudgets for critical workloads:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: nginx-pdb
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: nginx

• Use ResourceQuota and LimitRange to prevent resource hogging.
• Regularly audit PriorityClasses:

kubectl get priorityclasses -o custom-columns=NAME:.metadata.name,VALUE:.value,GLOBAL:.globalDefault,PREEMPTION:.preemptionPolicy

Operations Checklist

Use this checklist to ensure you have comprehensive monitoring and alerting for pod priority and preemption.

Monitoring Setup

• [ ] Confirm PodPriority feature gate is enabled (default in K8s 1.14+). Run: kubectl version --short and check version.
• [ ] List and document all PriorityClasses: kubectl get priorityclasses.
• [ ] Enable scheduler metrics by ensuring --bind-address=0.0.0.0 and --authorization-always-allow-paths=/metrics are set. Verify by running: kubectl get --raw /api/v1/namespaces/kube-system/pods/kube-scheduler-<node>:10251/proxy/metrics | grep scheduler_preemption.
• [ ] Set up a metrics scraper (Prometheus) to collect scheduler metrics.

Alerting Rules

Create alerts based on scheduler metrics. Example Prometheus alert rules:

groups:
- name: preemption-alerts
  rules:
  - alert: HighPreemptionRate
    expr: rate(scheduler_preemption_attempts_total{result="success"}[5m]) > 0.1
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "High preemption rate in cluster"
      description: "Preemption is occurring frequently ({{ $value }} per second), possibly causing instability."
  - alert: PreemptionFailures
    expr: increase(scheduler_preemption_attempts_total{result="failure"}[15m]) > 0
    labels:
      severity: critical
    annotations:
      summary: "Preemption failures detected"
      description: "Scheduler failed to preempt pods for scheduling."

• [ ] Configure alert notifications (e.g., Slack, PagerDuty).

Dashboard Panels

In Grafana (or similar), create panels for:

• Preemption attempts rate: rate(scheduler_preemption_attempts_total[5m])
• Preemption victims count: scheduler_preemption_victims_total
• Pending pods with high priority: kube_pod_status_phase{phase="Pending"} * on(pod) group_left(priority_class) kube_pod_info (requires kube-state-metrics).
• Node resource pressure: sum by (node) (kube_node_status_capacity{resource="cpu"}) - sum by (node) (kube_pod_container_resource_requests{resource="cpu"})

Incident Response Runbook

1. On alert, verify preemption events: kubectl get events -A --field-selector reason=Preempted --sort-by='.lastTimestamp'.
2. Identify preempted pods and their owners.
3. Check cluster resource utilization: kubectl top nodes.
4. If critical pods are affected, adjust priorities or scale resources.
5. Document incident and review priority classes.

Conclusion

Monitoring Kubernetes Pod Priority and Preemption is essential for maintaining application stability. By understanding how the scheduler makes preemption decisions and observing key metrics, you can prevent unexpected evictions and respond quickly to incidents.

We covered the full lifecycle: from inventorying your environment and safely configuring priorities, to verifying preemption behavior, diagnosing failures, and establishing operational checklists with alerts and dashboards.

As a next step, choose one low-risk verification from this guide. For example, list your PriorityClasses and audit a few critical deployments to ensure they have appropriate priorities. Then, enable scheduler metrics and create a simple alert for preemption attempts. This will give you immediate visibility into scheduler behavior.

Remember that preemption is a powerful but potentially disruptive feature. Always test changes in a controlled environment, document your priority hierarchy, and have recovery procedures ready. With the right monitoring and alerting, you can harness pod priority and preemption to improve resource utilization without sacrificing reliability.