Quick check 1 of 2
What command can you use to view events for a Pod that is pending with a FailedScheduling message?
- kubectl get events
- kubectl describe pod | grep -A 9999999999 Events
- kubectl logs
- kubectl get pods -o yaml
Submit
The reference states that you can use `kubectl describe pod frontend | grep -A 9999999999 Events` to view the events for a Pod.
## Safe Configuration Path
Kube Scheduler configuration is typically defined by a YAML file passed via the --config flag, or by a set of command-line flags. Changing configuration without testing can break scheduling for the entire cluster, so proceed cautiously.
### Locate the Configuration Source
If the scheduler runs as a static pod, view its manifest to find the config file path and flags:
cat /etc/kubernetes/manifests/kube-scheduler.yaml
Look for lines like:
spec:
containers:
- command:
- kube-scheduler
- --config=/etc/kubernetes/scheduler-config.yaml
Or, if using flags directly:
- --profiles=default-scheduler
- --leader-elect=true
If the scheduler runs as a systemd service, examine the unit file:
systemctl cat kube-scheduler
Example output may show an ExecStart line:
ExecStart=/usr/local/bin/kube-scheduler \
--config=/etc/kubernetes/scheduler-config.yaml \
--v=2
### Validate Configuration Before Applying
Before making any change, validate the configuration file using the scheduler's own binary with --config and a dry-run:
kube-scheduler --config=/etc/kubernetes/scheduler-config.yaml --dry-run
If the configuration is valid, the scheduler will start and immediately exit in dry-run mode. Any errors are printed to stdout/stderr. For example, a typo in the API version would produce:
error: error unmarshaling configuration: unknown field "apiVersion" in v1beta3.KubeSchedulerConfiguration
Then fix the file before applying.
### Make One Scoped Change at a Time
When editing the scheduler config, change only one parameter and keep a backup of the original file:
cp /etc/kubernetes/scheduler-config.yaml /etc/kubernetes/scheduler-config.yaml.bak
For example, suppose you want to increase scheduler logging verbosity to debug level. In the config file, you might edit:
apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
clientConnection:
kubeconfig: /etc/kubernetes/scheduler.conf
leaderElection:
leaderElect: true
profiles:
- schedulerName: default-scheduler
Add verbosity: 5 under clientConnection or set the --v=5 flag if using flags. After saving, restart the scheduler process. For static pods, the kubelet automatically restarts pods when the manifest changes; for systemd, run:
systemctl restart kube-scheduler
### Verify the Change
Check that the scheduler is running and logging at the new verbosity:
kubectl logs -n kube-system -l component=kube-scheduler --tail=20
You should see more detailed log lines, including scheduling attempts for each pod. If the scheduler fails to start, check the logs for errors and revert to the backup if necessary:
cp /etc/kubernetes/scheduler-config.yaml.bak /etc/kubernetes/scheduler-config.yaml
systemctl restart kube-scheduler
Always define what a successful verification looks like before making the change. For a verbosity increase, success means more log entries appear without errors, and the scheduler pod shows Running status.
## Verification and Diagnostics
Diagnosing scheduler issues often involves checking pod events, scheduler logs, and cluster state. Here are practical diagnostic steps.
### Check Pending Pods
Identify pods that remain in Pending state because no node can satisfy them:
kubectl get pods --all-namespaces --field-selector=status.phase=Pending
Example output:
NAMESPACE NAME READY STATUS RESTARTS AGE
default nginx-pod 0/1 Pending 0 10m
Describe a pending pod to see scheduler events:
kubectl describe pod nginx-pod
Look for events like:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 10m default-scheduler 0/3 nodes are available: 3 Insufficient cpu.
This clearly indicates the pod requests more CPU than any node can provide. Adjust resource requests or add nodes.
### Examine Scheduler Logs
Scheduler logs are invaluable. Use kubectl logs as shown earlier. To see logs from a specific scheduler pod:
kubectl logs -n kube-system kube-scheduler-master01 --tail=50
If the scheduler is running as a systemd service, use journalctl :
journalctl -u kube-scheduler -n 50 --no-pager
Look for key messages:
- Successfully bound pod to node indicates a successful scheduling decision.
- Error scheduling pod with a reason like node(s) didn't match node selector points to node affinity/selector issues.
- Unable to schedule pod; no fit means no node satisfied all predicates.
### Use Scheduler Simulator (Optional)
For complex scenarios, use the scheduler's standalone simulator (e.g., kube-scheduler-simulator ) in a test environment. It allows you to feed pod specs and node inventory to see scheduling outcomes without affecting production.
You can run the simulator as a container:
docker run -p 3000:3000 ghcr.io/kubernetes-sigs/kube-scheduler-simulator:latest
Then navigate to http://localhost:3000 and use the web UI to create nodes and pods, and observe which nodes the scheduler would choose.
## Failure Modes and Recovery
Common scheduler failure modes and their recovery steps.
### Scheduler Pod CrashLoopBackOff
If the scheduler pod repeatedly crashes, check logs:
kubectl logs -n kube-system kube-scheduler-master01 --previous
Common causes:
- Missing or malformed configuration file.
- Incorrect TLS certificates.
- Lack of permissions for the scheduler's kubeconfig.
Recovery example: if the config file path is wrong, correct the manifest and let the kubelet restart the pod automatically. Or, if the kubeconfig is invalid, regenerate it using kubeadm or your cluster tooling.
### Leader Election Issues
In HA setups, only one scheduler instance is active; others wait. If leader election fails, scheduler pods might all be in CrashLoopBackOff or restarting frequently. Check logs for leader election messages:
kubectl logs -n kube-system kube-scheduler-master01 | grep -i leader
Expect lines like:
I1003 09:15:30.123456 1 leaderelection.go:258] successfully acquired lease kube-system/kube-scheduler
If you see repeated attempts to acquire the lease without success, check the kube-scheduler Lease object:
kubectl get lease -n kube-system kube-scheduler -o yaml
Ensure no stale holder and that endpoints are reachable. Sometimes deleting the lease helps if it is stuck:
kubectl delete lease -n kube-system kube-scheduler
Use this only as a last resort and after understanding the implications.
### Insufficient Resources
When pods cannot be scheduled because nodes lack CPU, memory, or other resources, the scheduler will emit Insufficient messages. Check node capacity and allocatable resources:
kubectl describe nodes
Look for lines under Allocatable :
Allocatable:
cpu: 4
memory: 8Gi
ephemeral-storage: 100Gi
Compare with pod requests. If requests are too high, adjust them. If nodes are genuinely full, add nodes or scale down other workloads.
### Node Selector/Affinity Mismatch
Pods with node selectors or affinity rules that do not match any node will remain Pending. Example event:
0/3 nodes are available: 3 node(s) didn't match node selector.
Check node labels:
kubectl get nodes --show-labels
Recovery: update the node labels if possible, or change the pod's nodeSelector/affinity to match actual nodes. For example, add a label to a node:
kubectl label node worker01 disktype=ssd
### Taints and Tolerations
Pods that do not tolerate node taints will not be scheduled there. See taints:
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints
If a node has a taint like node-role.kubernetes.io/master:NoSchedule , pods without matching tolerations will ignore it. Add tolerations to the pod spec or remove the taint if appropriate:
kubectl taint nodes master01 node-role.kubernetes.io/master:NoSchedule-
Be careful when removing taints on control-plane nodes in production.
## Operations Checklist
A concise checklist for daily operations and troubleshooting.
| Area | Action | Command | Expected Signal |
|---|
| Scheduler status | Verify pod is Running | kubectl get pod -n kube-system -l component=kube-scheduler | READY 1/1, STATUS Running |
| Logs | View last 50 lines | kubectl logs -n kube-system -l component=kube-scheduler --tail=50 | No error lines like FailedScheduling |
| Pending pods | List all Pending pods | kubectl get pods --all-namespaces --field-selector=status.phase=Pending | No pods, or known acceptable Pending |
| Events | Describe a stuck pod | kubectl describe pod <pod-name> | Events show successful scheduling or clear reason for failure |
| Config validation | Dry-run scheduler with config | kube-scheduler --config=/path/to/config --dry-run | No error output |
| Health endpoint | Check scheduler health | curl -k https://localhost:10259/healthz | Returns ok |
| Leader election | Verify lease holder | kubectl get lease -n kube-system kube-scheduler -o yaml | Holder identity matches a running pod |
Run these checks regularly or when troubleshooting. Always record outputs and timestamps for audit and rollback decisions.
## Conclusion
Kube Scheduler troubleshooting is a systematic process, not guesswork. Start with environment inventory to know the version and topology. Inspect configuration safely, making one change at a time with validation and rollback plans. Use logs and pod events to pinpoint failures, and apply recovery steps specific to the failure mode.
As a next step, choose one low-risk verification from the Operations Checklist, run it on your cluster, and compare the result against the expected signal. Review dependencies such as Scheduling Framework, Pod Priority and Preemption, and Node Affinity when appropriate.
A reliable workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision.
Master these practices, and you will turn scheduler troubleshooting from a fire drill into a routine, controlled procedure.