Kubernetes Scheduling Framework: Production Operations Checklist with Practical Examples

Intro

Kubernetes Scheduling Framework production operations require a systematic approach: observe the current state, understand the scheduling pipeline, make scoped changes, and verify results. This article provides a practical checklist with commands, expected outputs, failure signals, and recovery steps for engineers operating the scheduling framework in production.

The Kubernetes Scheduling Framework is a pluggable architecture inside kube-scheduler that allows you to extend scheduling behavior with custom plugins. In production, misconfiguration or lack of observability can lead to pods stuck in Pending, resource waste, or missed SLOs. This checklist is intended for DevOps engineers, platform teams, and SREs who manage Kubernetes clusters and need to ensure the scheduling layer is healthy, correctly configured, and recoverable.

We will cover version and environment inventory, safe configuration paths, verification and diagnostics, failure modes and recovery, and a consolidated operations checklist. Each section provides concrete commands and examples. By following this guide, you will be able to detect problems early, change configuration safely, and recover from incidents with confidence.

Version and Environment Inventory

Understanding your scheduling framework begins with a precise inventory. This includes cluster version, scheduler configuration, plugin versions, and dependent resources. A version mismatch or outdated configuration can silently change scheduling behavior, so start by collecting read-only information before making any changes.

Check Cluster and Scheduler Version

Run the following command to see the kube-scheduler version:

kubectl version --short
kubectl get pods -n kube-system -l component=kube-scheduler -o wide

Expected output includes the scheduler pod name, node, and status. For example:

NAME                 READY   STATUS    RESTARTS   AGE   IP           NODE
kube-scheduler-minikube   1/1     Running   0          10d   10.0.0.1     control-plane

If the scheduler pod is not running or is crashing, check logs with:

kubectl logs -n kube-system kube-scheduler-minikube --previous

This shows crash loop details. For a managed cluster (EKS, GKE, AKS), the control plane is not directly accessible, so rely on kubectl version and cloud provider documentation for scheduler version alignment.

Inspect Scheduler Configuration

The kube-scheduler configuration is often defined in a KubeSchedulerConfiguration file mounted as a ConfigMap or file. To view the current configuration, use:

kubectl get configmap -n kube-system kube-scheduler-config -o yaml

If the scheduler is running as a static pod, inspect its manifest:

kubectl get pod -n kube-system kube-scheduler-minikube -o yaml | grep -A 20 command

Look for flags like --config=/etc/kubernetes/scheduler-config.yaml. Then ``kubectl exec`` into the scheduler pod (if allowed) or access the node to read the file. Example configuration snippet:

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
profiles:
  - schedulerName: default-scheduler
    plugins:
      score:
        enabled:
          - name: NodeResourcesFit
            weight: 1

Record the plugin list and weights. This is critical for troubleshooting scoring issues.

Verify Prerequisites and Dependencies

List all CRDs related to scheduling extensions, such as PriorityClass, RuntimeClass, or custom scheduling plugins:

kubectl get crd | grep -E 'scheduling|priority|runtime'

Check node availability and taints, as they directly affect scheduling:

kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

Example output:

NAME                 TAINTS
node1               <none>
node2               [key=value:NoSchedule]

If a taint exists without a matching toleration, pods will not schedule on that node. This kind of inventory must be done before any change.

Blast Radius and Recovery Path

For any potential change, identify which resources are affected. For instance, modifying scheduler policy can affect all pods that use the default scheduler. If you add a plugin, ensure the scheduler pod restarts successfully and no pods are left unscheduled. Always have a rollback plan, such as restoring the previous ConfigMap and restarting the scheduler.

Safe Configuration Path

Changing scheduler configuration in production should follow a controlled process: test in a staging cluster, review the configuration schema, apply with minimal disruption, and verify scheduling outcomes. This section provides a step-by-step safe path.

Step 1: Understand the Desired Change

Example: You want to add a custom scoring plugin that prefers nodes with SSDs for certain workloads. Define the desired plugin configuration. In KubeSchedulerConfiguration, add the plugin under score and ensure the plugin binary is available in the scheduler image or via a mounted volume.

For managed Kubernetes, you may not be able to modify the scheduler directly; instead, you may need to deploy a second scheduler. In that case, create a new Deployment with a custom scheduler name and configuration, and update workloads to use that scheduler.

Step 2: Validate Configuration Schema

Before applying, validate the YAML syntax and schema. Use a local tool like kube-scheduler --config=config.yaml --validate if available, or run a dry-run with the API server:

kubectl apply --dry-run=client -f scheduler-config.yaml

Note: This only validates general Kubernetes resource schema, not the scheduler-specific config. For that, rely on documentation and version-specific validation. Check the Kubernetes version to ensure the configuration API version (kubescheduler.config.k8s.io/v1 vs v1beta1) matches. Example for Kubernetes 1.26+: use v1.

Step 3: Apply Change with Minimal Blast Radius

If modifying the default scheduler, update the ConfigMap or file, then restart the scheduler pod. For a static pod, the kubelet will restart it automatically upon file change. For a Deployment-based scheduler, trigger a rolling restart:

kubectl rollout restart deployment custom-scheduler -n kube-system

Monitor the rollout:

kubectl rollout status deployment/custom-scheduler -n kube-system

Expected output: deployment "custom-scheduler" successfully rolled out.

If you are using a second scheduler for specific workloads, update the Deployment for that scheduler only. Existing pods are not affected, but new pods using that scheduler will be affected.

Step 4: Test Scheduling with a Sample Pod

Create a test pod that uses the new scheduler or the modified configuration. For example, to test the default scheduler with a new scoring plugin, deploy a pod with the appropriate scheduler name:

apiVersion: v1
kind: Pod
metadata:
  name: test-scheduling
spec:
  schedulerName: default-scheduler
  containers:
  - name: test
    image: busybox
    command: ["sleep", "3600"]
    resources:
      requests:
        cpu: "100m"
        memory: "128Mi"

Apply and check status:

kubectl apply -f test-pod.yaml
kubectl get pod test-scheduling -o wide

The pod should be in Running state, and the NODE column should indicate a suitable node. If it stays Pending, use kubectl describe pod to see events from the scheduler.

Step 5: Verify Scheduling Decision

Inspect scheduler logs to confirm the plugin was used. If scheduler verbosity is set, you may see scoring details. To increase logging temporarily, set --v=4 in scheduler command, but avoid high verbosity in production due to performance impact. For debugging, you can run a separate scheduler instance with higher verbosity pointed to the same cluster for a short period, but do not schedule production pods through it unless carefully isolated with schedulerName.

Always document the change and rollback plan. For example:

• Change: Added NodeResourcesFit scoring plugin with weight 2 for scheduler default-scheduler.
• Rollback: Restore previous ConfigMap kube-scheduler-config from backup, delete scheduler pod to restart.

Verification and Diagnostics

After any change, or during routine operation, verify scheduler health and diagnose common issues. This section covers commands and signals for verification.

Health Check Commands

Check scheduler pod status and logs:

kubectl get pods -n kube-system -l component=kube-scheduler
kubectl logs -n kube-system kube-scheduler-minikube --tail=100

Look for errors like failed to load config, plugin not found, or repeated leader election messages (normal in HA setups).

Check scheduler leader election (if multiple replicas):

kubectl get endpoints -n kube-system kube-scheduler -o yaml

The annotation control-plane.alpha.kubernetes.io/leader shows the current leader. Example:

annotations:
  control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"kube-scheduler-minikube", ...}'

Diagnostic for Pending Pods

When a pod is stuck in Pending, describe it to see events:

kubectl describe pod <pod-name>

Common scheduling failure messages:

• 0/3 nodes are available: 3 Insufficient cpu. → resource requests too high.
• 0/3 nodes are available: 1 node(s) had taint {key: value}, that the pod didn't tolerate. → add toleration or remove taint.
• 0/3 nodes are available: 3 node(s) didn't match node selector. → adjust nodeSelector or label nodes.

Use kubectl get events --sort-by=.lastTimestamp to see recent scheduling events.

Verify Scheduling Framework Plugins

To confirm which plugins are active, query the scheduler configuration. If you have access to the scheduler pod's config file, view it. For a ConfigMap-based setup:

kubectl get configmap kube-scheduler-config -n kube-system -o jsonpath='{.data}' | jq .

Alternatively, use kubectl describe on the scheduler pod to see the command arguments that reference the config file, then exec into the pod (if possible) to cat the file.

Metrics and Observability

If Prometheus is configured, query scheduler metrics:

scheduler_pending_pods
scheduler_schedule_attempts_total
scheduler_scheduling_duration_seconds

Example alert: If scheduler_pending_pods remains high for more than 5 minutes, investigate unschedulable pods.

Failure Modes and Recovery

Scheduler failures can manifest as pods not scheduling, incorrect placement, or scheduler crash loops. This section describes common failure modes and recovery steps.

Failure Mode 1: Scheduler Pod Crash Loop

Symptom: kubectl get pods -n kube-system -l component=kube-scheduler shows CrashLoopBackOff.

Diagnose:

kubectl logs -n kube-system kube-scheduler-minikube --previous

Common causes:

• Invalid configuration file syntax.
• Plugin binary not found or version mismatch.
• Missing RBAC permissions.

Recovery:

• Restore the last known good configuration from backup.
• If a custom plugin is causing the crash, remove the plugin from config or fix the plugin binary.
• Restart the scheduler pod: kubectl delete pod -n kube-system kube-scheduler-minikube (if static pod, it will be recreated).

Failure Mode 2: Pods Stuck Pending

Symptom: A pod remains Pending with no node assigned.

Diagnose:

kubectl describe pod <pod-name> | grep -A 10 Events

Look for reasons like FailedScheduling. Common remedies:

• Increase cluster capacity or reduce pod resource requests.
• Add node labels or adjust nodeSelector.
• Add tolerations for taints.
• Check if pod is using a non-existent scheduler name: spec.schedulerName must match an available scheduler.

Failure Mode 3: Incorrect Scheduling Decisions

Symptom: Pods are scheduled to nodes but the placement does not match expectations (e.g., workload on non-SSD nodes despite plugin).

Diagnose:

• Verify the plugin is enabled in the scheduler configuration.
• Check scheduler logs for scoring details (with verbosity).
• Confirm the node labels are correct: kubectl get nodes --show-labels.
• Ensure the pod's resource requests are within node capacity.

Recovery:

• Adjust plugin weights or configuration.
• If necessary, evict the pod (kubectl delete pod) and let it reschedule with the corrected configuration.

Failure Mode 4: Scheduler Unavailable (Leader Election Lost)

Symptom: No scheduler leader or multiple leaders causing inconsistent scheduling.

Diagnose:

• Check endpoints as mentioned earlier.
• Check for network partition or high load on control plane.

Recovery:

• Ensure all scheduler replicas have consistent configuration and can communicate with the API server.
• If a split-brain is suspected, restart the scheduler pods one by one.

General Recovery Steps

1. Identify the failure: use kubectl get pods, describe, logs.
2. Isolate the blast radius: stop new scheduling by setting pause: true in scheduler config (if supported) or cordoning nodes.
3. Rollback configuration: restore previous ConfigMap or file, restart scheduler.
4. Verify recovery: observe scheduler logs and pod scheduling.
5. Document the incident and update runbook.

Operations Checklist

This checklist consolidates routine checks and actions for operating the Kubernetes Scheduling Framework in production. Use it as a baseline for daily, weekly, and monthly reviews.

Daily Checks

• [ ] Verify scheduler pods are Running: kubectl get pods -n kube-system -l component=kube-scheduler
• [ ] Check for pods in Pending state: kubectl get pods --all-namespaces --field-selector=status.phase=Pending
• [ ] Review scheduler logs for errors (tail recent logs)
• [ ] Watch scheduler metrics for anomalies (if Prometheus is used)

Weekly Checks

• [ ] Validate scheduler configuration against version documentation (ensuring no deprecated API versions)
• [ ] Review node capacity and resource requests vs. allocatable to anticipate scheduling pressure
• [ ] Check for any custom scheduling plugins updates or necessary patches
• [ ] Ensure backup of scheduler configuration exists and is up to date

Monthly Checks

• [ ] Perform a failover test: kill the scheduler pod and verify a new leader is elected (if HA) and scheduling continues
• [ ] Review and update scheduler plugin list based on workload changes
• [ ] Audit scheduler-related RBAC permissions for least privilege
• [ ] Run capacity planning: simulate increased load and check if scheduler can keep up (consider scheduler throughput limits)

Change Management Checklist

Before any scheduler change:

• [ ] Understand the goal and impact: which workloads, nodes, plugins are affected?
• [ ] Review configuration schema and supported API version for your Kubernetes version
• [ ] Test change in a non-production cluster with identical scheduler plugins
• [ ] Prepare rollback plan: snapshot current configuration, know how to restore
• [ ] Announce change window and expected behavior
• [ ] Apply change incrementally (if using multiple schedulers, migrate workloads gradually)
• [ ] Monitor scheduler health and pod scheduling during and after change
• [ ] Document the change and any observed side effects

Emergency Runbook Reference

Conclusion

Operating the Kubernetes Scheduling Framework in production demands a disciplined approach: inventory your environment, modify configuration through safe paths, verify with diagnostics, and recover from failures using predefined runbooks. This checklist provides a foundation, but you should adapt it to your cluster's specifics, including custom plugins and organizational policies.

Remember that scheduling is a critical control plane function; errors can disrupt all workloads. Therefore, always observe before changing, keep changes scoped and reversible, and maintain observability through logs and metrics. As you gain experience, extend this checklist with your own learnings and incorporate it into your operational runbooks.

For further depth, consult the official Kubernetes documentation on the Scheduling Framework, kube-scheduler configuration, and plugin development. Continuously review release notes for scheduler changes when upgrading clusters.