Kube Controller Manager Capacity Planning: A Practical Sizing and Scaling Guide

Intro

Effective capacity planning for the Kubernetes Controller Manager (kube-controller-manager) helps platform engineers move from reactive troubleshooting to proactive sizing. This guide provides practical, hands-on techniques for measuring current resource consumption, determining appropriate CPU and memory allocations, and scaling the controller manager safely in production clusters.

Whether you run a small startup cluster or a large multi-tenant environment, the controller manager's health directly impacts core Kubernetes functionality: Deployments, ReplicaSets, StatefulSets, DaemonSets, Jobs, and more. Under-provisioning this component can lead to controller starvation, API server overload, and degraded cluster performance. Over-provisioning wastes resources and can mask design inefficiencies.

This article is written for SREs, DevOps engineers, and platform teams who need a practical, command-driven approach to capacity planning. We focus on Kubernetes version 1.28 and later, but the principles and most commands apply to earlier releases with minor adjustments. You will learn how to:

• Inventory your current controller manager configuration and resource usage
• Analyze metrics to identify bottlenecks and headroom
• Configure resource requests and limits appropriately
• Scale the controller manager horizontally or vertically when needed
• Diagnose common failure modes and recover from them
• Implement a repeatable operations checklist

By the end, you will have a tested methodology with concrete examples and expected outputs, enabling you to make informed decisions with minimal risk to cluster stability.

Version and Environment Inventory

Before changing anything, establish a clear picture of your environment. This section covers how to inspect the Kubernetes version, controller manager deployment topology, and current resource settings.

Check Kubernetes Version and Controller Manager Image

First, confirm your cluster's Kubernetes version and the controller manager image in use. This is critical because flags and metrics names change between versions.

# Get cluster version
kubectl version --short
# Output (example):
# Client Version: v1.28.2
# Server Version: v1.28.2

# If using a kubeadm cluster, check the static pod definition for the image
sudo grep -A5 'image:' /etc/kubernetes/manifests/kube-controller-manager.yaml
# Output may include:
#   image: registry.k8s.io/kube-controller-manager:v1.28.2

For managed Kubernetes services like EKS, GKE, or AKS, you cannot directly inspect the control plane. Instead, check the release channel or cluster version through the cloud provider's CLI, e.g.:

# AWS EKS
eksctl get cluster --name <cluster-name>
# Output shows Version: 1.28

# Google GKE
gcloud container clusters describe <cluster-name> --zone <zone> --format='value(currentMasterVersion)'

Record the exact version because certain resource-related flags (like --kube-api-qps and --kube-api-burst) were only added or changed in specific releases.

Inspect Current Resource Requests and Limits

For self-managed clusters, the controller manager is typically a static pod controlled by the kubelet. Its resource requests and limits are defined in the pod manifest.

# Read the static pod manifest
sudo cat /etc/kubernetes/manifests/kube-controller-manager.yaml

Look for the resources section in the container spec. Example snippet:

containers:
- name: kube-controller-manager
  image: registry.k8s.io/kube-controller-manager:v1.28.2
  resources:
    requests:
      cpu: 200m
      memory: 100Mi
    limits:
      cpu: 400m
      memory: 200Mi

If no resources are specified, the kubelet will not reserve any guarantee, and the controller manager can be evicted under pressure. This is a common pitfall.

For cloud-managed control planes, you cannot modify the controller manager directly. Instead, you rely on the provider's tuning and may need to adjust node sizes for the control plane (e.g., in EKS, the control plane runs on managed nodes with fixed resources). Thus, your capacity planning focuses on workload distribution and API priority.

Verify Controller Manager is Running and Healthy

Use kubectl to check the pod status (in self-managed clusters) or the controller manager's health endpoints.

# For self-managed, check pod status
kubectl get pods -n kube-system -l component=kube-controller-manager
# Output:
# NAME                         READY   STATUS    RESTARTS   AGE
# kube-controller-manager-node1 1/1     Running   0          5d2h

# Check health endpoints (from within control plane node)
curl -s http://localhost:10257/healthz
# Expected: ok
curl -s http://localhost:10257/healthz/ping
# Expected: ok

If the health check fails, investigate immediately using journalctl -u kubelet or the container logs.

Safe Configuration Path

When modifying the controller manager, always follow a safe process: snapshot current configuration, make one scoped change, verify, and have a rollback plan. This section provides concrete steps.

Step 1: Baseline Current Settings

Before any change, save the current manifest or configuration.

For self-managed clusters, copy the static pod manifest to a backup location:

sudo cp /etc/kubernetes/manifests/kube-controller-manager.yaml /root/kcm-backup-$(date +%Y%m%d).yaml

Also record current metrics observed over a baseline period (at least 24 hours) to compare after changes. Use metrics-server or Prometheus queries. Example with kubectl top:

kubectl top pod -n kube-system -l component=kube-controller-manager
# Output (example):
# NAME                         CPU(cores)   MEMORY(bytes)
# kube-controller-manager-node1 150m        180Mi

Document these numbers.

Step 2: Decide on Resource Changes Based on Evidence

Use historical metrics to set requests and limits. A general approach:

• Requests should cover P99 usage over the baseline period plus a safety margin (e.g., 25%).
• Limits should be set higher to allow bursts but not allow runaway usage. A common ratio is limits = 2 x requests for CPU, and limits = 1.5 to 2 x requests for memory.

For example, if P99 CPU usage is 180m and memory P99 is 150Mi, then set:

• requests: cpu=250m, memory=200Mi
• limits: cpu=500m, memory=400Mi

But always validate with workload-specific data.

Step 3: Apply Changes (Self-Managed Static Pod)

For a static pod managed by kubelet, editing the manifest file triggers a restart automatically.

# Edit manifest
sudo vi /etc/kubernetes/manifests/kube-controller-manager.yaml
# Add or modify the resources section as needed

# Save and exit; kubelet will restart the pod. Monitor with:
kubectl get pods -n kube-system -w

Verify the pod is running with new resources:

kubectl get pod -n kube-system -l component=kube-controller-manager -o json | jq '.items[0].spec.containers[0].resources'
# Output shows the new requests/limits

Step 4: Validate and Observe

After the change, monitor the controller manager's resource usage and ensure no evictions or OOM kills occur.

# Watch events in kube-system
kubectl get events -n kube-system --field-selector involvedObject.name=kube-controller-manager-node1 --watch
# Look for OOMKilled or eviction messages

If the controller manager restarts frequently or gets killed, revert to the backup:

sudo cp /root/kcm-backup-YYYYMMDD.yaml /etc/kubernetes/manifests/kube-controller-manager.yaml
# Wait for restart and verify health

Step 5: Adjust Flags for API Rate Limiting (Optional)

In large clusters, the controller manager may overwhelm the API server. Tune --kube-api-qps and --kube-api-burst based on observed throttling.

Default values are qps=20, burst=30. If you see many 429 responses in API server metrics, increase gradually.

# In the manifest, add flags:
--kube-api-qps=50
--kube-api-burst=100

Validate with kubectl get --raw /metrics on the controller manager, looking for rest_client_requests_total and throttling metrics.

Verification and Diagnostics

This section details how to use metrics and logs to verify capacity and diagnose issues.

Essential Metrics to Monitor

The controller manager exposes Prometheus metrics on port 10257 (HTTPS) at /metrics. Key metrics include:

• workqueue_depth for each controller (deployment, replicaset, etc.) - indicates backlog
• workqueue_adds_total, workqueue_retries_total - rates of processing
• process_cpu_seconds_total, process_resident_memory_bytes - resource usage
• rest_client_requests_total with code labels - API calls
• leader_election_master_status - indicates if this instance is the leader (in HA setups)

Sample PromQL queries:

# Controller manager CPU usage rate (per second)
rate(process_cpu_seconds_total{job="kube-controller-manager"}[5m])

# Current memory usage in bytes
process_resident_memory_bytes{job="kube-controller-manager"}

# Top 5 workqueue depths
sort_desc(topk(5, workqueue_depth{job="kube-controller-manager"}))

# API request throttling
rate(rest_client_requests_total{job="kube-controller-manager", code="429"}[5m])

If workqueue_depth for deployment controller stays high (>100) for extended periods, the controller manager may be under-resourced or API rate limited.

Using kubectl top and describe

For quick checks, kubectl top gives real-time resource usage (if metrics-server is installed):

kubectl top pod -n kube-system -l component=kube-controller-manager

kubectl describe can show recent events like OOMKilled:

kubectl describe pod -n kube-system kube-controller-manager-node1 | grep -A5 'Events:'
# Example event:
#   Warning   OOMKilled   kubelet   Container kube-controller-manager was OOMKilled

Analyzing Controller Manager Logs

On self-managed clusters, logs are available via journalctl (if using systemd) or container runtime logs.

# Using crictl (for containerd)
sudo crictl logs $(sudo crictl ps --name kube-controller-manager -q) --tail=100

# Or journalctl
journalctl -u kubelet --since "1 hour ago" | grep kube-controller-manager

Look for patterns like Failed to list *v1.Deployment (indicating API issues), the server has received too many requests (rate limiting), or repeated Back-off restarting failed container (resource crashes).

Verifying Leader Election (HA Clusters)

In a high-availability setup with multiple control plane nodes, only one controller manager is active. Check which pod is the leader:

# Query leader election metric (requires direct access to metrics endpoint)
kubectl exec -n kube-system <active-kube-controller-manager-pod> -- curl -s https://localhost:10257/metrics | grep leader_election_master_status
# Output: leader_election_master_status{name="kube-controller-manager"} 1

The pod with value 1 is the current leader. Non-leader pods should have 0. If none has 1, leader election is broken.

Failure Modes and Recovery

Understanding common failure modes helps you react quickly and prevent future occurrences.

OOMKilled (Out of Memory)

Cause: Memory limit too low for the controller manager's working set, often due to high object counts or large caches.

Detection: kubectl describe pod shows OOMKilled termination reason; pod restarts repeatedly.

Recovery: Increase memory limit gradually. For example, if current limit is 200Mi and OOM occurs, raise to 500Mi, observe, then adjust. Also consider reducing memory pressure by cleaning up old resources or tuning --concurrent-deployment-syncs (default 5) downward if too many concurrent reconciles.

Prevention: Set memory limit to at least 2x average usage and ensure requests cover P99.

CPU Starvation

Cause: CPU limit throttling prevents timely queue processing, causing backlog and delayed deployments.

Detection: Workqueue depth increases, kubectl top shows CPU usage at or near limit, and container CPU throttling metrics (container_cpu_cfs_throttled_seconds_total) are high.

Recovery: Increase CPU limit or remove it if the node has capacity. Alternatively, distribute controller load by running multiple controller managers with leader election (only one active) and using node anti-affinity for separation if you have custom controllers.

Prevention: Monitor CPU throttling; set CPU request equal to observed usage and limit to 2-3x request.

API Server Rate Limiting

Cause: Too many API requests from controller manager exceeding --kube-api-qps and --kube-api-burst limits, leading to 429 responses and retries.

Detection: Metrics rest_client_requests_total{code="429"} increasing; logs show "client rate limiter" messages.

Recovery: Increase qps and burst settings gradually. For example, from qps=20, burst=30 to qps=50, burst=100. Monitor effect; if problems persist, investigate root cause (e.g., too many objects in one namespace).

Prevention: Baseline API call rates in normal operation; set limits with headroom (e.g., 2x peak observed rate).

Leader Election Failure

Cause: Network partition between control plane nodes, clock skew, or etcd issues causing lost leases.

Detection: Multiple controller managers think they are leader (split-brain) or none does; logs show leader election timeouts.

Recovery: Ensure network connectivity, sync NTP, and restart the controller manager pods. Force a new lease by deleting the lease object (risky):

kubectl delete lease -n kube-system kube-controller-manager

Wait for re-election; do not do this during high load.

Prevention: Run control plane nodes on stable infrastructure, monitor clock skew, and have etcd redundancy.

Misconfigured Static Pod Manifest

Cause: Syntax error or invalid flag in /etc/kubernetes/manifests/kube-controller-manager.yaml after editing, causing pod creation failure.

Detection: Pod not appearing in kubectl get pods; kubelet logs show manifest parsing errors.

Recovery: Restore from backup manifest immediately.

Prevention: Validate YAML syntax before saving (yamllint or kubectl apply --dry-run=client -f on a dummy resource), test changes on a staging cluster first.

Operations Checklist

This checklist summarizes the key steps for safe capacity planning and scaling of the controller manager. For each item, we name an accountable owner (typical role) and suggested review frequency.

For example, Priya Shah, Engineering Lead, owns the monthly capacity forecast for the production cluster. She reviews controller manager metrics alongside etcd and API server projections every first Monday of the month. Her forecast includes expected resource headroom for the next quarter, triggering early scaling decisions.

Another example: Marcus Chen, SRE, is accountable for the weekly API rate limiting check. Every Friday, he runs a PromQL query on rest_client_requests_total{code="429"} for the controller manager. If the rate exceeds 0.1% (average 5-minute rate), he opens a ticket to adjust --kube-api-qps and --kube-api-burst after reviewing recent workload changes.

For the recovery plan test, Aisha Patel, DevOps Engineer, is the owner. She runs the test on the staging environment every quarter (e.g., on the second Tuesday of January, April, July, October). She simulates an OOMKill by temporarily lowering memory limits in a dedicated test cluster, documents the time from detection to recovery, and updates the runbook if the procedure drifts.

These ownership assignments ensure clear accountability, and the review frequencies keep capacity planning proactive.

Common Pitfalls and How to Avoid Them

This section addresses frequent mistakes in controller manager capacity planning.

1. Ignoring Baseline Data

Pitfall: Setting resource requests and limits arbitrarily (e.g., copy-pasting from another cluster) without observing actual usage.

Why it happens: Time pressure or lack of monitoring setup.

Impact: Under-provisioning leads to OOMKill or throttling; over-provisioning wastes resources and can cause node bin-packing issues.

How to avoid: Always collect at least one week of metrics before deciding. Use kubectl top and Prometheus queries as shown earlier.

2. Not Accounting for Bursts During Deployments or Scaling Events

Pitfall: Setting resource requests too close to average usage, ignoring peak loads during mass pod recreation.

Why it happens: Developers look at steady-state metrics only.

Impact: During rolling updates or node failures, controller manager CPU spikes, causing slow reconciliation and API timeouts.

How to avoid: Look at P99 or max usage over a period that includes scaling events. Add a buffer of 25-50% for requests and higher limits.

3. Forgetting to Tune API Rate Limits

Pitfall: Leaving default --kube-api-qps and --kube-api-burst in clusters with high object counts or many controllers.

Why it happens: These flags are hidden in the static pod manifest and often overlooked.

Impact: API throttling causes controller backlogs and delayed deployments.

How to avoid: Monitor 429 metrics; proactively tune based on cluster size. For large clusters (e.g., >10,000 pods), consider qps=100, burst=200 or higher, but test incrementally.

4. Misunderstanding Leader Election Implications

Pitfall: Assuming all controller manager pods process work simultaneously; in HA, only the leader is active, others are standby.

Why it happens: Misunderstanding of Kubernetes control plane design.

Impact: Horizontal scaling of controller manager beyond the standby instance does not increase throughput; only vertical scaling (more resources to leader) helps.

How to avoid: For scalability, focus on resource sizing of the leader and ensuring failover readiness. If you need parallel processing, consider building custom controllers with sharding, not the core controller manager.

5. Not Validating YAML Changes

Pitfall: Editing /etc/kubernetes/manifests/kube-controller-manager.yaml with an indentation error or mis-typed flag, causing the kubelet to ignore the pod or crash-loop it.

Why it happens: Direct editing without validation.

Impact: Controller manager down, cluster functionality impaired.

How to avoid: Use a YAML linter, dry-run with kubelet --config test, and always have a backup. Consider using a configuration management tool like Ansible to deploy changes with pre-validation.

Conclusion

Capacity planning for the Kubernetes Controller Manager is not a one-time task but a continuous process of observation, adjustment, and verification. By following the structured approach outlined in this article—inventory, baseline, safe configuration, diagnostics, failure recovery, and checklist—you can maintain a healthy control plane that responds predictably to workload changes.

Start with the low-risk steps: gather metrics, compare against current limits, and identify immediate over- or under-provisioning. Then implement gradual changes with rollback plans. Document your decisions and revisit them monthly.

A reliable operations workflow for the controller manager ensures that your cluster's core controllers can handle the demands of your applications without becoming a bottleneck. With the concrete commands and examples provided, you are equipped to perform effective capacity planning in your own environment.