>
E-NO
EN FR
Get Started
Kube Controller Manager advanced concepts 7 Min Read

Kube Controller Manager Advanced Concepts Explained with Practical Examples

calendar_today Published: 2026-08-29
update Last Updated: 2026-08-29
analytics SEO Efficiency: 100%
Technical guide illustration for Kube Controller Manager Advanced Concepts Explained with Practical Examples.

Intro

Kube Controller Manager is the component of the Kubernetes control plane that runs the core control loops. It watches the state of the cluster through the API server and makes changes to move the current state toward the desired state. For operators and developers working with production clusters, understanding its advanced concepts is essential for reliable operation, effective troubleshooting, and performance tuning.

This article explains advanced concepts of kube-controller-manager with practical examples. We cover its architecture, how controllers work internally, key configuration options, monitoring and diagnostics, common failure modes, and recovery procedures. Every section includes commands, expected outputs, and best practices you can apply directly.

We assume you have a running Kubernetes cluster and kubectl access. All examples are version-agnostic but note where behavior changed. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached.

Version and Environment Inventory

Before making any changes or debugging issues, you need a clear picture of your environment. This means identifying the Kubernetes version, how the control plane is deployed, and the current state of the kube-controller-manager process.

Check Kubernetes Version

Run:

kubectl version --short

Expected output (example, may vary):

Client Version: v1.27.3
Kustomize Version: v5.0.1
Server Version: v1.27.3

Note the server version because some controller behaviors and flags change between releases. For example, the --use-service-account-credentials flag was deprecated in v1.24 and removed in v1.26.

Identify Control Plane Deployment Topology

kube-controller-manager runs as a static pod on control plane nodes in clusters installed with kubeadm, or as a systemd service in some self-managed clusters.

To see if it is a static pod, run:

kubectl get pods -n kube-system | grep controller-manager

Expected output (example):

kube-controller-manager-controlplane01            1/1     Running   0          5d3h

If you see such a pod, it is managed by the kubelet on that node via static pod manifests. The manifest is usually at /etc/kubernetes/manifests/kube-controller-manager.yaml.

To inspect the flags passed to the controller manager, describe the pod:

kubectl describe pod -n kube-system kube-controller-manager-controlplane01

Look for the Command section in the output. For example:

Command:
  kube-controller-manager
  --allocate-node-cidrs=true
  --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf
  --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf
  --bind-address=127.0.0.1
  --client-ca-file=/etc/kubernetes/pki/ca.crt
  --cluster-cidr=10.244.0.0/16
  --cluster-name=kubernetes
  --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
  --cluster-signing-key-file=/etc/kubernetes/pki/ca.key
  --controllers=*,bootstrapsigner,tokencleaner
  --kubeconfig=/etc/kubernetes/controller-manager.conf
  --leader-elect=true
  --node-cidr-mask-size=24
  --root-ca-file=/etc/kubernetes/pki/ca.crt
  --service-account-private-key-file=/etc/kubernetes/pki/sa.key
  --use-service-account-credentials=true

This output reveals important configuration details. For instance, --leader-elect=true means the controller manager is running in high-availability mode with leader election.

Check Controller Manager Health

The controller manager exposes a health endpoint at /healthz on its serving address (default 127.0.0.1:10257). On a control plane node, you can curl it locally:

curl -k https://127.0.0.1:10257/healthz

Expected output:

ok

If you get an error, the controller manager may be unhealthy. Check logs (see Verification and Diagnostics section).

Prerequisites for Changes

Before making any configuration change to kube-controller-manager, ensure:

  • You have a backup of the current configuration file or manifest.
  • You know the impact of changing each flag.
  • You have a rollback plan.
  • For static pod configuration, you can edit the manifest file on the node; the kubelet will restart the pod automatically.
  • For systemd service, edit the service file and restart.
  • Never put real credentials or tokens in documentation; use placeholders.

Quick check 1 of 2

What is the primary role of a controller in Kubernetes?

A controller is a control loop that watches the shared state of the cluster through the apiserver and makes changes attempting to move the current state towards the desired state.

Safe Configuration Path

Modifying kube-controller-manager settings can disrupt cluster operations if done incorrectly. Follow a safe configuration path: observe current state, make a minimal change, verify, and have a recovery plan.

Common Configuration Changes

1. Adjusting Controller Concurrency

Each controller in kube-controller-manager runs a number of concurrent workers. You can tune concurrency per controller via flags, but most controllers use defaults that are safe for production. For example, to increase concurrency for the deployment controller, you would need to modify the --concurrent-deployment-syncs flag (default 5). However, note that not all controllers have such flags; only a subset do. Check the official documentation for your Kubernetes version.

Example flag to add to the manifest:

--concurrent-deployment-syncs=10

Blast radius: affects only the deployment controller's sync speed; increased CPU and memory usage.

Verification: After restart, check the controller manager logs for the flag or use metrics (see Monitoring section).

2. Changing the Node Eviction Thresholds

The node lifecycle controller (formerly part of kube-controller-manager) evicts pods from unhealthy nodes based on thresholds. You can adjust --node-monitor-grace-period (default 40s) and --node-monitor-period (default 5s).

Example:

--node-monitor-grace-period=60s
--node-monitor-period=10s

These flags tell the controller how long to wait before considering a node unhealthy. Increasing the grace period can reduce false positives in flaky networks but delays pod eviction.

Blast radius: affects all nodes and workloads; may delay recovery from node failures.

Verification: Simulate a node failure (e.g., stop kubelet on a node) and observe eviction timing.

3. Configuring Leader Election

By default, kube-controller-manager uses leader election to ensure only one instance is active. You can tune --leader-elect-lease-duration (default 15s), --leader-elect-renew-deadline (default 10s), and --leader-elect-retry-period (default 2s). These values are usually fine for production, but in some environments with high network latency, you may need to increase them.

Example:

--leader-elect-lease-duration=30s
--leader-elect-renew-deadline=20s
--leader-elect-retry-period=5s

Blast radius: affects failover time if the active instance fails.

Verification: Check the leader election status in logs or metrics.

Safe Edit Procedure

  1. Back up the current manifest:
   sudo cp /etc/kubernetes/manifests/kube-controller-manager.yaml /etc/kubernetes/manifests/kube-controller-manager.yaml.bak
  1. Edit the manifest with your changes. For static pods, the kubelet will detect the change and restart the pod automatically.
  1. Watch the new pod come up:
   kubectl get pods -n kube-system -w | grep controller-manager
  1. If the pod fails to start, check logs:
   kubectl logs -n kube-system kube-controller-manager-controlplane01
  1. If necessary, roll back by restoring the backup:
   sudo cp /etc/kubernetes/manifests/kube-controller-manager.yaml.bak /etc/kubernetes/manifests/kube-controller-manager.yaml

Always verify that the controller manager becomes ready before considering the change successful.

Verification and Diagnostics

After changing configuration or when troubleshooting, you need to verify the controller manager is functioning correctly and diagnose any issues.

Checking Logs

The primary source for diagnostics is the controller manager logs. Access them via kubectl:

kubectl logs -n kube-system kube-controller-manager-controlplane01

Look for errors, warnings, or messages about leader election, controller sync failures, etc.

Example of a normal log snippet:

I0712 10:00:00.123456       1 controllermanager.go:246] "Starting" version="v1.27.3"
I0712 10:00:00.123789       1 leaderelection.go:248] attempting to acquire leader lease kube-system/kube-controller-manager...
I0712 10:00:00.234567       1 leaderelection.go:258] successfully acquired lease kube-system/kube-controller-manager
I0712 10:00:00.234999       1 controllermanager.go:256] "Started"

If the controller manager is stuck in leader election, you will see repeated "attempting to acquire" messages without success. This could indicate issues with the API server or etcd.

Metrics and Health Endpoints

kube-controller-manager exposes metrics and health endpoints. By default, they listen on 127.0.0.1:10257 for secure serving and 127.0.0.1:10252 for insecure (deprecated). To access metrics, you may need to port-forward:

kubectl port-forward -n kube-system kube-controller-manager-controlplane01 10257:10257

Then in another terminal:

curl -k https://127.0.0.1:10257/metrics

You will see a large set of metrics. Key metrics to watch include:

  • workqueue_depth for each controller (indicates backlog).
  • workqueue_adds_total and workqueue_retries_total.
  • leader_election_master_status (1 if this instance is leader).
  • rest_client_requests_total for API server interactions.

Example scrape of a specific metric:

curl -k https://127.0.0.1:10257/metrics | grep workqueue_depth

Output snippet:

# HELP workqueue_depth Current depth of workqueue
# TYPE workqueue_depth gauge
workqueue_depth{name="deployment"} 0
workqueue_depth{name="replicaset"} 0
workqueue_depth{name="statefulset"} 0

If any queue depth is continuously high, the controller may be struggling to keep up.

Using kubectl for Component Status

Older versions of Kubernetes had kubectl get componentstatuses (or cs), but it is deprecated and removed in v1.19+. Do not rely on it. Instead, check the pod status and logs.

Diagnostic Scenario: ReplicaSet Not Creating Pods

Suppose you create a Deployment, but the ReplicaSet is not creating pods. The issue may be in the replicaset controller. Check the controller manager logs for errors related to replicaset:

kubectl logs -n kube-system kube-controller-manager-controlplane01 | grep -i replicaset

Look for error messages like missing permissions, invalid spec, etc. If you see RBAC errors, check the service account permissions for the controller manager.

Quick check 2 of 2

What is the default secure serving port for kube-controller-manager metrics and health endpoints?

By default, kube-controller-manager exposes metrics and health endpoints on 127.0.0.1:10257 for secure serving.

Failure Modes and Recovery

Understanding common failure modes of kube-controller-manager and how to recover is critical for maintaining cluster health.

1. Leader Election Stuck

Symptom: Controller manager logs show repeated "attempting to acquire leader lease" but never succeed. The active instance may have died, but the lease is not released, or there is a network partition.

Diagnosis: Check the lease object:

kubectl get lease -n kube-system kube-controller-manager -o yaml

Look at the spec.holderIdentity and spec.renewTime. If the holder is a pod that no longer exists, the lease should expire, but sometimes cleanup is delayed.

Recovery: You can delete the lease to force a new election:

kubectl delete lease -n kube-system kube-controller-manager

Then watch logs to see a new leader elected. This is safe because the lease is recreated automatically.

2. Controller Manager CrashLoopBackOff

Symptom: The controller manager pod is in CrashLoopBackOff state.

Check: Describe the pod:

kubectl describe pod -n kube-system kube-controller-manager-controlplane01

Look for events and last termination reason. Common causes:

  • Invalid flag: the pod fails immediately with an error message.
  • Missing files: e.g., kubeconfig or certificate files not mounted.
  • Insufficient resources: unable to start due to OOM.

Example error:

Error: failed to create listener: failed to listen on 127.0.0.1:10257: listen tcp 127.0.0.1:10257: bind: address already in use

This indicates another process is using the port. You may need to kill that process or change the port.

Recovery: Fix the underlying issue, then restart the pod. For static pods, the kubelet will restart automatically after the manifest is corrected.

3. High CPU or Memory Usage

Symptom: Controller manager consumes excessive CPU or memory, possibly causing other components to degrade.

Diagnosis: Use resource monitoring (Prometheus, metrics-server) or kubectl top if available:

kubectl top pod -n kube-system kube-controller-manager-controlplane01

Output:

NAME                                            CPU(cores)   MEMORY(bytes)
kube-controller-manager-controlplane01          500m         300Mi

If usage is much higher than normal, investigate which controller is busy. Check metrics for workqueue depth and rate of processed items. High load may be due to a large number of objects churning, or a bug in a particular controller.

Mitigation: You can reduce concurrency for the most active controllers by adjusting flags (if available) or investigate the underlying workload causing churn. For example, a misbehaving controller that creates and deletes pods repeatedly will cause high CPU.

4. Controller Manager Not Starting Due to Certificate Issues

Symptom: The pod fails with TLS errors.

Example log:

E0712 10:00:00.123456       1 run.go:74] "command failed" err="failed to load config: open /etc/kubernetes/controller-manager.conf: no such file or directory"

Recovery: Ensure the kubeconfig file exists and is valid. You may need to regenerate certificates using kubeadm:

kubeadm init phase certs all

Then restart the controller manager.

Operations Checklist

Use this checklist for routine operations and incident response involving kube-controller-manager.

Routine Health Check

  • [ ] Verify the controller manager pod is Running and ready: kubectl get pods -n kube-system | grep controller-manager
  • [ ] Check leader election status: in logs or metrics (leader_election_master_status should be 1 on one instance).
  • [ ] Review logs for recent errors: kubectl logs -n kube-system kube-controller-manager-controlplane01 --tail=100
  • [ ] Monitor key metrics: workqueue depth, API request latency, goroutines.
  • [ ] Ensure no crash loops: kubectl get pods -n kube-system | grep controller-manager shows stable restart count.

Before Configuration Change

  • [ ] Backup current manifest: cp /etc/kubernetes/manifests/kube-controller-manager.yaml /tmp/kcm-backup-$(date +%Y%m%d).yaml
  • [ ] Review flag documentation for your Kubernetes version.
  • [ ] Estimate blast radius: which controllers are affected? Could this impact all workloads?
  • [ ] Prepare rollback: know how to restore the backup quickly.
  • [ ] Notify team if the change could cause brief control plane disruption.

After Configuration Change

  • [ ] Monitor pod startup: kubectl get pods -n kube-system -w | grep controller-manager
  • [ ] Verify logs show successful start and leader election.
  • [ ] Test a representative workload: create a test Deployment and ensure it scales correctly.
  • [ ] Check metrics for anomalies.
  • [ ] Document the change and its effect.

Incident Response: Controller Manager Down

  • [ ] Check pod status: kubectl get pods -n kube-system
  • [ ] If CrashLoopBackOff, inspect logs and describe pod.
  • [ ] If leader election stuck, consider deleting the lease.
  • [ ] If on a self-managed cluster, check systemd service status: systemctl status kube-controller-manager
  • [ ] If the API server is also down, focus first on etcd and API server recovery, as controller manager depends on them.
  • [ ] Once recovered, validate that controllers are working by checking that existing resources are reconciled (e.g., Deployments have correct replica counts).

Conclusion

kube-controller-manager is a core component that ensures the desired state of Kubernetes resources is maintained. Understanding its advanced concepts—architecture, controller loops, configuration, monitoring, and troubleshooting—is essential for running a healthy cluster.

We covered how to inventory your environment, safely modify configuration, diagnose issues, and recover from common failures. Always follow operational best practices: observe first, change minimally, verify, and have a rollback plan.

By applying the practical examples and checklists in this article, you can confidently manage kube-controller-manager in production and resolve issues before they impact your applications.

Next steps: choose one low-risk verification from this article, such as checking the health endpoint or viewing metrics, and run it in your cluster. Record the current state, compare with expected signals, and review dependencies like Kube API Server, ReplicaSet, and Deployment. This builds a reliable workflow that makes failures visible and recovery decisions clear.

Related Research

Article Quality Score

Reader usefulness 100%
  • check_circle Reader-ready guide
  • check_circle Practical examples included
  • check_circle Clean SEO article URL