Kubernetes Lease Troubleshooting: A Practical Field Guide

Intro

Kubernetes Leases are a coordination primitive that control plane components and custom controllers rely on for leader election, heartbeat signaling, and resource locking. When a Lease breaks, the blast radius can be wide: split-brain scenarios, stalled controllers, or failed node heartbeats. Troubleshooting these failures requires a systematic approach that examines logs, Lease object state, and cluster events. This guide provides practical steps, commands, and recovery workflows to diagnose and resolve common Kubernetes Lease problems. By following the methods described here, you can reduce downtime and prevent recurring issues.

Version and Environment Inventory

Before troubleshooting, establish the Kubernetes version, control plane topology, and the components involved in Lease usage. This information helps identify known issues and compatibility gaps. Run the following commands to gather environment details:

kubectl version --short
kubectl get nodes -o wide
kubectl get leases --all-namespaces

Typical output for kubectl version --short (example):

Client Version: v1.28.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.3

The kubectl get nodes -o wide output shows node status, roles, and Kubernetes version per node. The kubectl get leases --all-namespaces command lists all Lease objects across the cluster, including those used by kube-controller-manager, kube-scheduler, and custom controllers. For example:

NAMESPACE     NAME                      HOLDER                           AGE
kube-system   kube-controller-manager   node1                            3d
kube-system   kube-scheduler            node2                            3d
my-app        my-controller             my-controller-5f8b7c9d-abcde     1h

Identify which Lease is problematic and note its namespace, name, and holder. Also check the Kubernetes documentation for version-specific behavior, as Lease API changes can affect troubleshooting steps.

Safe Configuration Path

When modifying Lease-related configurations, make changes in a controlled manner. Start with a narrow scope, such as a single namespace or a non-critical controller, to validate the impact. This aligns with the principle of a narrow, measurable pilot before broader deployment.

Common configuration points include:

Lease duration and renew time for controllers: Many controllers expose flags like --leader-elect-lease-duration and --leader-elect-renew-deadline. Adjust these only after understanding the timing implications.
RBAC permissions: Controllers need get, create, and update permissions on Leases. A missing permission often causes failures.
Resource quotas: In constrained environments, Lease objects may be prevented from creation due to namespace quotas.

To inspect a specific Lease's configuration, use:

kubectl get lease <lease-name> -n <namespace> -o yaml

Example output (abbreviated):

apiVersion: coordination.k8s.io/v1
kind: Lease
metadata:
  name: my-controller
  namespace: my-app
spec:
  holderIdentity: my-controller-5f8b7c9d-abcde
  leaseDurationSeconds: 15
  acquireTime: "2025-03-10T10:00:00Z"
  renewTime: "2025-03-10T10:05:00Z"
  leaseTransitions: 0

If the renewTime is old relative to the current time, the holder has likely failed to renew. Before changing any configuration, back up the current state and plan a rollback path. For example, saving the Lease YAML allows restoration if needed.

Verification and Diagnostics

Start diagnostics by examining the status and events related to the Lease and its controller.

Check Lease status

kubectl describe lease <lease-name> -n <namespace>

Look for the Renew Time and Holder Identity. If the renew time is not updating, the holder is unhealthy.

Review controller logs

kubectl logs -n <namespace> deployment/<controller-deployment> | grep -i lease

Example log lines indicating failure:

E0310 10:05:12.345678       1 leaderelection.go:330] error retrieving resource lock kube-system/my-controller: leases.coordination.k8s.io "my-controller" is forbidden: User "system:serviceaccount:my-app:my-controller" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "kube-system"

This error points to an RBAC issue. Another common error:

W0310 10:05:12.345678       1 leaderelection.go:340] Failed to update lock: Operation cannot be fulfilled on leases.coordination.k8s.io "my-controller": the object has been modified; please apply your changes to the latest version and try again

This indicates a conflict due to multiple contenders or a stale cache.

Check events

kubectl get events -n <namespace> --field-selector involvedObject.name=<lease-name>

Output example:

LAST SEEN   TYPE     REASON                 OBJECT              MESSAGE
5m          Warning  FailedToUpdateLease   lease/my-controller  Failed to update lease my-controller: ...

Verify Lease API availability

kubectl api-resources | grep leases

Expected output:

leases   coordination.k8s.io/v1   true   Lease

If Leases are not listed, the API group may not be enabled (rare in standard distributions).

Simulate a Lease access check

kubectl auth can-i get leases -n <namespace> --as=system:serviceaccount:<namespace>:<serviceaccount>

Output yes or no indicates permission.

When diagnosing leader election issues, inspect the holder pod's logs for leader election messages:

I0310 10:05:12.345678       1 leaderelection.go:250] successfully acquired lease kube-system/my-controller

If the pod repeatedly logs "attempting to acquire leader lease" and fails, it suggests a problem with the Lease object or network.

Failure Modes and Recovery

Common failure modes and recovery steps:

1. Stale Lease Holder

Symptoms: The Lease holder pod is terminated or unresponsive, but the Lease still lists it as holder. Other replicas cannot acquire leadership.

Diagnosis:

kubectl get lease <lease-name> -n <namespace> -o jsonpath='{.spec.holderIdentity}{"\n"}'
kubectl get pods -n <namespace> | grep <holder-prefix>

If the holder pod no longer exists, the Lease is stale.

Recovery: Delete the Lease to allow re-election (ensure no active holder):

kubectl delete lease <lease-name> -n <namespace>

A new leader will be elected by a running replica. Alternatively, manually patch the Lease to clear the holder:

kubectl patch lease <lease-name> -n <namespace> --type merge -p '{"spec":{"holderIdentity":""}}'

2. RBAC Misconfiguration

Symptoms: Logs show "forbidden" errors when accessing Leases.

Diagnosis: Use kubectl auth can-i as above. Review the Role/RoleBinding or ClusterRole/ClusterRoleBinding for the service account.

Recovery: Correct the RBAC rules. Example minimal Role for Leases:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: lease-access
  namespace: my-app
rules:
- apiGroups: ["coordination.k8s.io"]
  resources: ["leases"]
  verbs: ["get", "create", "update", "patch"]

Apply and verify:

kubectl apply -f role.yaml
kubectl auth can-i update leases -n my-app --as=system:serviceaccount:my-app:my-controller

Expected output: yes

3. Clock Skew

Symptoms: Lease renews fail with "lease duration expired" or leaders flap.

Diagnosis: Check node time synchronization:

kubectl get nodes -o custom-columns=NAME:.metadata.name,STATUS:.status.conditions[-1].type,REASON:.status.conditions[-1].reason

Manually check timedatectl or chronyc tracking on nodes.

Recovery: Configure NTP properly on all nodes and restart affected components.

4. Resource Quota Exhaustion

Symptoms: Lease creation fails with exceeded quota.

Diagnosis:

kubectl describe resourcequota -n <namespace>

Check if the quota includes count/leases.

Recovery: Increase quota or delete unused Leases after confirming no active holders.

Rollback Guidance

Always have a rollback plan. For configuration changes, keep a backup of the original manifests. For Lease deletion, note the previous holder and ensure at least one replica is running to take over. In case of a bad change, restore the Lease from backup or revert the configuration and restart the controller.

Common Pitfalls and How to Avoid Them

Lease troubleshooting often uncovers recurring mistakes. Here are the most frequent ones and how to prevent or recover from them.

Pitfall 1: Assuming the Lease is the problem when it is a symptom.

The Lease may be stale because the holder pod is crash-looping, not because the Lease itself is corrupt. Always check the holder pod's status and logs before deleting the Lease.

How to avoid: Correlate Lease issues with pod health. Use kubectl describe pod and kubectl logs to understand why the holder failed. Deleting a healthy Lease without fixing the underlying pod will result in the same failure after re-election.

Pitfall 2: Misconfigured RBAC for custom controllers.

Many custom controllers use a service account in their own namespace but attempt to access Leases in kube-system or another namespace. A Role that works in my-app does not grant access in kube-system; you need a ClusterRole or a Role in the target namespace.

How to avoid: Verify the exact namespace in the error message. Use kubectl auth can-i --namespace=<target-ns> to confirm permissions before applying fixes.

Pitfall 3: Ignoring clock skew.

Lease timing depends on accurate clocks. Even a few seconds of drift can cause leaders to release the Lease prematurely or fail to renew. This is common on bare-metal clusters without NTP.

How to avoid: Implement NTP on all nodes as part of cluster provisioning. Monitor time drift with tools like chrony and alert on offsets beyond 100ms.

Pitfall 4: Deleting a Lease during an active leader election.

If you delete a Lease while a healthy leader is actively renewing it, you force a new election unnecessarily. This causes a brief period of no leader and can disrupt operations.

How to avoid: Check the renewTime and holder pod status before deleting. If the holder is running and the renew time is recent, do not delete the Lease; investigate the controller's logic instead.

Pitfall 5: Overly short lease durations.

Setting leaseDurationSeconds too low (e.g., 2 seconds) can cause flapping under load. The holder may not renew in time, causing frequent leader changes.

How to avoid: Use defaults (15 seconds for many controllers) unless you understand the timing implications. If you must tune, increase the renew deadline proportionally and test under load.

Operations Checklist

Use this checklist for routine Lease health checks and post-incident reviews. Assign an owner (e.g., Platform Engineer) and review frequency (monthly or after any cluster upgrade).

Check	Command or Action	Expected Result
All Leases listed	`kubectl get leases --all-namespaces`	No errors, expected Leases present
Lease renew time current	`kubectl get lease <name> -n <ns> -o jsonpath='{.spec.renewTime}'`	Timestamp within lease duration
Holder identity matches active pod	Compare holder with pod list	Holder pod is Running
Controller logs free of Lease errors	`kubectl logs <pod> \| grep -i lease \| grep -i error`	No output or only transient warnings
RBAC allows Lease operations	`kubectl auth can-i --list -n <ns> \| grep leases`	Rules include get, create, update
Events show no repeated failures	`kubectl get events -n <ns> --field-selector reason=FailedToUpdateLease`	No recent events
API resource available	`kubectl api-resources \| grep leases`	Line with coordination.k8s.io/v1
Clock sync on nodes	`kubectl get nodes -o jsonpath='{.items[*].status.addresses[?(@.type=="InternalIP")].address}'` then SSH check	No skew beyond a few ms

Perform these checks monthly and after any cluster upgrade or controller change. Store the results in a shared document and review with the team to track trends. For larger teams, consider automating these checks with a cron job or a monitoring tool like Prometheus alerts.

Conclusion

Kubernetes Lease troubleshooting is essential for maintaining cluster stability. By systematically gathering environment details, verifying configurations, using diagnostic commands, and applying targeted recovery steps, you can resolve Lease failures efficiently. Remember to start with narrow changes, verify outcomes, and maintain an operations checklist to prevent future issues. The practical examples in this guide provide a foundation for handling common Lease problems and can be adapted to your specific environment. Continue to monitor Lease health and stay updated with Kubernetes releases to ensure smooth operations.