Kubernetes Scheduling Framework Troubleshooting: A Practical Guide

Intro

The Kubernetes Scheduling Framework is a powerful extension mechanism that allows custom logic to influence pod placement. While it offers flexibility, troubleshooting framework-related failures can be challenging. This guide provides a practical approach to diagnosing and resolving common issues using logs, commands, and recovery workflows.

Understanding the framework's components and their interactions is essential for effective troubleshooting. By following the steps outlined here, you can quickly identify root causes and apply safe fixes. The focus is on practical, hands-on techniques that work in real clusters.

A systematic approach saves time and reduces the risk of misconfiguration. We will cover environment inventory, safe configuration, verification, failure modes, and an operations checklist. Each section includes concrete examples to illustrate key points.

Version and Environment Inventory

Before troubleshooting, gather accurate information about your cluster's version, topology, and scheduler configuration. This baseline helps identify version-specific bugs or misconfigurations.

Prerequisites

Access to the cluster with kubectl and permissions to view scheduler logs and configuration.
Knowledge of the Kubernetes version and distribution.
Understanding of the Scheduling Framework version in use, as behavior may change between releases.

Example: Collecting Environment Information

Run the following commands to capture essential details:

kubectl version --client
kubectl get nodes -o wide
kubectl describe pod <pod-name> -n <namespace>

Expected output includes the client and server versions, node statuses and labels, and pod events. For instance, the pod description may show scheduling failures with reasons like FailedScheduling. Here is an example of what the kubectl get nodes -o wide output might look like:

NAME   STATUS   ROLES    AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
node1  Ready    master   10d   v1.28.3   192.168.1.10  <none>        Ubuntu 22.04.3 LTS   5.15.0-91-generic  containerd://1.7.8
node2  Ready    <none>   10d   v1.28.3   192.168.1.11  <none>        Ubuntu 22.04.3 LTS   5.15.0-91-generic  containerd://1.7.8
node3  Ready    <none>   10d   v1.28.3   192.168.1.12  <none>        Ubuntu 22.04.3 LTS   5.15.0-91-generic  containerd://1.7.8

Key Information to Record

Kubernetes control plane version (e.g., v1.28.3)
Scheduler image version and any custom plugins
Node labels, taints, and resources
Pod specifications, including affinity, anti-affinity, and topology spread constraints

This inventory serves as a reference point for comparing behavior after changes. Store this information in a shared document or a runbook for team reference.

Safe Configuration Path

When modifying scheduler configuration, adopt a scoped and reversible approach. Avoid making broad changes without testing, as they can disrupt scheduling cluster-wide.

Start with a Pilot

The first useful pilot should be narrow, measurable, and easy to inspect locally. For example, enable a custom scoring plugin for only a specific namespace or workload type using a scheduler profile.

Example: Defining a Scheduler Profile

Create a ConfigMap with a scheduler policy that includes a custom plugin for a test namespace:

apiVersion: v1
kind: ConfigMap
metadata:
  name: my-scheduler-config
  namespace: kube-system
data:
  scheduler-config.yaml: |
    apiVersion: kubescheduler.config.k8s.io/v1
    kind: KubeSchedulerConfiguration
    profiles:
    - schedulerName: my-scheduler
      plugins:
        score:
          enabled:
          - name: MyCustomScorer
      pluginConfig:
      - name: MyCustomScorer
        args:
          customParam: value

Apply the ConfigMap, then create pods specifying schedulerName: my-scheduler to test the custom path without affecting default scheduling.

Validation Before Rollout

Use kubectl apply --dry-run=client to validate YAML syntax, and consider using a staging cluster for full validation. Always back up the existing scheduler configuration before applying changes. For example:

kubectl get configmap my-scheduler-config -n kube-system -o yaml > my-scheduler-config-backup.yaml

This backup allows quick rollback if the new configuration causes issues.

Verification and Diagnostics

Effective verification relies on observing scheduler behavior through logs, events, and metrics. This section covers key diagnostic commands and expected results.

Checking Scheduler Logs

Access scheduler logs to see framework plugin activity. If the scheduler runs as a static pod, use:

kubectl logs -n kube-system <scheduler-pod-name>

Look for lines indicating plugin execution, such as Running Score plugin, or errors like failed to score pod. Example log line for a successful scoring plugin:

I0321 10:00:00.123456       1 schedule_one.go:254] \"Scoring pod\" pod=\"default/test-pod\" plugin=\"MyCustomScorer\" score=80

If the scheduler is not a static pod but a deployment, you may need to check logs from the deployment's pods. To get the scheduler pod name, first run:

kubectl get pods -n kube-system -l component=kube-scheduler

Pod Events

Events provide initial clues to scheduling failures.

kubectl describe pod <pod-name> -n <namespace>

Example event:

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  10s   default-scheduler  0/3 nodes are available: 1 node(s) had taint {key: value}, that the pod didn't tolerate, 2 Insufficient cpu.

This indicates taints and resource shortages. To see all events in a namespace, use:

kubectl events -n <namespace>

Scheduler Performance Metrics

If metrics are enabled (e.g., via --bind-address and Prometheus scraping), query for scheduling attempt durations and plugin latencies.

curl -s http://<scheduler-ip>:10251/metrics | grep scheduler_plugin_execution_duration

High latencies in custom plugins may point to performance issues. For example, if the 99th percentile for a plugin exceeds 100ms, it may be causing scheduling delays. Consider using a tool like kube-state-metrics to collect these metrics into Prometheus for alerting.

Dry-Run and Simulation

Kubernetes does not provide a built-in dry-run for scheduling, but you can use the kube-scheduler binary locally with a simulated pod and node set to test plugin behavior. This is an advanced technique but useful for isolated debugging. A basic approach is to run the scheduler with a test configuration against a fake clientset. Example using scheduler-simulator (a third-party tool):

git clone https://github.com/kubernetes-sigs/scheduler-simulator
cd scheduler-simulator
make build
./bin/scheduler-simulator --config test-config.yaml

Then create a simulated pod and observe which nodes are selected. This helps validate plugin logic without affecting a live cluster.

Failure Modes and Recovery

Common failure modes include plugin misconfiguration, resource contention, and version incompatibility. This section outlines symptoms, root causes, and recovery steps.

Misconfigured Plugin Arguments

If a plugin expects a specific argument structure and it is missing, the scheduler may crash or fail to schedule pods. Check logs for errors like invalid plugin config and verify against the plugin documentation.

Recovery: Correct the ConfigMap and restart the scheduler.

kubectl edit configmap my-scheduler-config -n kube-system
# Fix the YAML
kubectl delete pod <scheduler-pod-name> -n kube-system # to restart if static pod

For a static pod, the kubelet will restart it automatically after the manifest changes. If the scheduler is a deployment, restart the deployment:

kubectl rollout restart deployment kube-scheduler -n kube-system

Node Resource Shortage

When pods remain in Pending state due to insufficient CPU or memory, the scheduler cannot place them. Use kubectl describe nodes to check capacity and allocations.

kubectl describe node node1

Look for Allocated resources section:

Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests      Limits
  --------           --------      ------
  cpu                1500m (75%)   0 (0%)
  memory             2Gi (50%)     4Gi (100%)

This shows that CPU is 75% requested, so only 25% remaining. If a pod needs 500m CPU, it may still fit, but if it needs more than 500m, it will fail.

Recovery: Add nodes, scale down other workloads, or adjust pod resource requests.

Version Mismatch

Custom plugins compiled against a different Kubernetes version may fail. Ensure the plugin image and scheduler version are compatible.

Recovery: Rebuild the plugin with the correct version or revert to a previous scheduler configuration. To check the scheduler version:

kubectl exec -n kube-system <scheduler-pod-name> -- kube-scheduler --version

This should output the version. Then verify the plugin's compatibility matrix.

Rollback Procedure

If a scheduler change causes widespread issues, revert to the previous configuration. For static pods, the scheduler manifest is typically in /etc/kubernetes/manifests/ on the control plane. Restore the backup and the kubelet will restart the scheduler automatically.

Example backup and restore:

sudo cp /etc/kubernetes/manifests/kube-scheduler.yaml /tmp/kube-scheduler.yaml.bak
# Make changes, then to rollback:
sudo cp /tmp/kube-scheduler.yaml.bak /etc/kubernetes/manifests/kube-scheduler.yaml

For non-static scheduler (e.g., deployed as a Deployment), roll back the deployment:

kubectl rollout undo deployment kube-scheduler -n kube-system

Verification After Recovery

After rollback, monitor pod scheduling and logs to confirm normal behavior. Create a test pod and ensure it schedules within a reasonable time:

kubectl run test-pod --image=nginx
kubectl get pod test-pod -w

Check that the pod becomes Running within a few minutes.

Operations Checklist

Use this checklist for routine checks and post-change validation.

Step	Action	Expected Result
1	Verify scheduler pod is running and ready	`kubectl get pods -n kube-system \| grep scheduler` shows `1/1 Running`
2	Check recent scheduler logs for errors	No repeated error patterns
3	Inspect pending pods and their events	Events show scheduling attempts or lack thereof
4	Validate node resources and taints	Nodes have sufficient capacity; taints are tolerated by workloads
5	Review scheduler configuration	ConfigMap or flag settings match intended values
6	Test a canary pod after any change	Pod schedules within expected time and on correct nodes
7	Monitor scheduler metrics for anomalies	Latency within baseline; no sudden spikes

Perform this checklist before and after any significant change, and periodically during normal operations. For example, run it weekly as part of cluster health checks.

Conclusion

Troubleshooting the Kubernetes Scheduling Framework requires a systematic approach. By collecting environment data, making scoped configuration changes, verifying through logs and events, and having a solid recovery plan, you can resolve issues efficiently.

Remember to start small, validate thoroughly, and always keep backups. The operations checklist provided serves as a practical tool for ongoing maintenance.

As next steps, consider enabling more detailed scheduler logging for persistent issues, exploring metrics for performance tuning, and sharing lessons learned with your team to improve scheduling reliability. For example, you can create a runbook that documents each incident, the root cause, and the resolution, so the team builds institutional knowledge and reduces mean time to recovery.