Kubernetes Lease Advanced Concepts Explained with Practical Examples

Intro

Kubernetes Leases are a fundamental coordination primitive that often go unnoticed until something breaks. They power leader election for critical control plane components like kube-controller-manager and kube-scheduler, and they provide a lightweight way for applications to coordinate exclusive access. But what exactly is a Lease, how does it work under the hood, and how do you troubleshoot it when things go wrong?

This article is a deep dive into Kubernetes Lease advanced concepts, built for developers, DevOps consultants, and technical startup teams who need to move beyond the basics. We\'ll explore the Lease API, its architecture and internals, and then walk through practical examples with real commands and expected outputs. By the end, you\'ll be able to observe, verify, and recover Lease-related issues with confidence.

We\'ll focus on operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify results, and document recovery paths. Let\'s get started.

Version and Environment Inventory

Before touching any Lease object, you need a clear picture of your environment. This section covers the relevant components, supported versions, prerequisites, a read-only observation step, the smallest justified change, and verification.

Component Scope

Kubernetes Leases are part of the coordination.k8s.io API group. The Lease kind was introduced in Kubernetes 1.14 and has been stable (v1) since then. It is used by core components for leader election, and it is available to any application that wants to use it.

Key components that interact with Leases:

• kube-controller-manager: Uses a Lease named kube-controller-manager in the kube-system namespace to elect the active controller manager instance. Only the leader runs controllers; the standby instances wait.
• kube-scheduler: Uses a Lease named kube-scheduler in kube-system for leader election among multiple scheduler replicas.
• Custom controllers and operators: Many built with client-go or controller-runtime use Leases for leader election, often with lease names derived from the component name and namespace.
• kubelet: In some configurations, the kubelet uses a Lease for node heartbeat (replacing the older NodeStatus update mechanism).

Prerequisites

• A Kubernetes cluster (version 1.14 or later) with kubectl configured to access it.
• The coordination.k8s.io/v1 API must be enabled (it is by default in all current distributions).
• For leader election, you need at least two replicas of the component (for control plane components, this means a multi-master setup or a deployment with multiple replicas).
• Sufficient RBAC permissions to get, list, and describe leases in the relevant namespace.

Read-Only Observation

Start with observation. Run the following commands to see the current state of Leases in your cluster:

# List all leases in kube-system (where control plane components live)
kubectl get leases -n kube-system

Example output:

NAME                    HOLDER                            AGE
kube-controller-manager master-node-1_1a2b3c4d-5e6f-...   10d
kube-scheduler          master-node-1_9a8b7c6d-5e4f-...   10d

The HOLDER column shows which instance currently holds the lease (i.e., is the leader). In this example, both leases are held by master-node-1, meaning that node hosts the leader for both components.

To see the full lease details:

kubectl get lease kube-controller-manager -n kube-system -o yaml

This will show the lease spec: holderIdentity, leaseDurationSeconds, acquireTime, renewTime, leaseTransitions. We'll explain these fields later.

Smallest Justified Change

After observing, if a change is needed (e.g., forcing a leader election), the smallest change is often to delete the Lease object. This causes the current leader to lose its lease, and the other replicas to compete for leadership. But this is a disruptive action; only do it in a controlled manner during a maintenance window if you need to move leadership.

A safer approach is to scale down the component deployment to zero and then scale it back up, or to manually trigger a leader election if the component has an endpoint for that (most don't).

Verification

After any change, verify the new state:

kubectl get leases -n kube-system
kubectl describe lease kube-controller-manager -n kube-system

Check that the holder has changed to another instance (if that was the intent) and that the component's pods are running. For example:

kubectl get pods -n kube-system | grep kube-controller-manager

Expected output shows the new leader pod running on a different node.

Safe Configuration Path

Configuring Leases involves understanding their parameters and how they affect leader election behavior. The safe path is to avoid direct editing of system Leases and instead configure your own components correctly.

Lease Resource Definition

A Lease object is defined as follows (example):

apiVersion: coordination.k8s.io/v1
kind: Lease
metadata:
  name: my-app-leader
  namespace: default
spec:
  holderIdentity: my-app-pod-0
  leaseDurationSeconds: 15
  acquireTime: "2023-01-01T00:00:00Z"
  renewTime: "2023-01-01T00:00:10Z"
  leaseTransitions: 2

Key fields:

• holderIdentity: The identity of the current lease holder (often the pod name or a unique ID).
• leaseDurationSeconds: The time a lease is valid without renewal. If the holder fails to renew before this duration expires, the lease is considered lost and other contenders may acquire it.
• acquireTime: Timestamp when the lease was acquired.
• renewTime: Timestamp of the last renewal.
• leaseTransitions: Number of times the lease has changed hands.

Leader Election Algorithm

Kubernetes uses a simple algorithm for leader election based on Leases:

1. Each candidate attempts to create a Lease object with its own identity as holder. The first one to create it wins.
2. The leader periodically renews the lease by updating renewTime (typically every leaseDurationSeconds / 3 or so).
3. If the leader fails to renew, other candidates notice that renewTime is older than leaseDurationSeconds, and they attempt to acquire the lease by updating holderIdentity and renewTime.
4. The process repeats.

This is implemented in client-go's leaderelection package for custom controllers; control plane components use a similar built-in mechanism.

Configuring Lease Parameters

For your own applications, set leaseDurationSeconds based on your reliability requirements. A smaller duration means faster failover but more frequent renewals (more API calls). A larger duration reduces API load but increases the time to detect a failed leader.

A common pattern:

• leaseDurationSeconds = 15 (or 30)
• renewDeadline = 10 (the leader must renew within this time)
• retryPeriod = 2 (how often the leader attempts to renew)

These are typically set in the component's leader election configuration, not directly on the Lease object (the controller manages the Lease automatically).

Example: Configuring a Custom Controller with Leader Election

Assume you're using controller-runtime (Go). You would configure the leader election options like:

mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
    LeaderElection: true,
    LeaderElectionID: "my-app-leader",
    LeaseDuration: &leaseDuration,
    RenewDeadline: &renewDeadline,
    RetryPeriod: &retryPeriod,
})

This creates a Lease named my-app-leader in the namespace where the controller runs. The controller automatically handles renewal and failover.

Verification

After deploying your controller, verify the Lease was created and is being renewed:

kubectl get lease my-app-leader -n my-namespace -o yaml

Look for renewTime updating every few seconds. You can watch it with:

kubectl get lease my-app-leader -n my-namespace -w

Verification and Diagnostics

When something goes wrong with leader election or Lease-based coordination, you need to diagnose the issue systematically. This section outlines the steps and commands to verify healthy behavior and pinpoint failures.

Verifying Leader Election for Control Plane Components

Run the following to check the current leader and recent lease activity:

kubectl get lease -n kube-system -o custom-columns=NAME:.metadata.name,HOLDER:.spec.holderIdentity,AGE:.metadata.creationTimestamp,RENEW:.spec.renewTime

This gives a concise view of who is leader and when they last renewed. If RENEW is not recent (older than leaseDurationSeconds), the leader may be down or unable to renew.

Inspecting Lease Details

Use kubectl describe to see the full state:

kubectl describe lease kube-controller-manager -n kube-system

Output includes events (if any) and the spec fields. Note: Leases don't generate events by default, but you can see the current holder and renewal time.

Diagnostic Scenarios

Scenario 1: No leader elected

If multiple replicas are running but no one holds the lease, check the component logs for errors. For kube-controller-manager:

kubectl logs -n kube-system kube-controller-manager-<node-name> | grep -i leader

Look for messages like "attempting to acquire leader lease..." or errors related to API access or lease updates. Common causes:

• Insufficient RBAC permissions to update the lease.
• Network partition between control plane nodes.
• Misconfigured --leader-elect flag (must be true for multi-replica).

Scenario 2: Frequent leader changes (flapping)

If leaseTransitions increments rapidly, there may be a problem with renewals. Check the logs for renewal failures. Also, check clock skew between nodes; lease timing is based on wall clock time, and significant skew can cause premature lease expiry.

# Check clock sync status on each node (using timedatectl or ntpq)

Scenario 3: Lease not being renewed

If the holder's renewTime is stale, the leader may be stuck or resource-starved. Check the pod's resource usage:

kubectl top pod -n kube-system kube-controller-manager-<node-name>

High CPU or memory pressure could prevent timely renewals.

Using Events

While Leases themselves don't emit events, the component using the lease may log events. You can check the events in the namespace:

kubectl get events -n kube-system --sort-by=.lastTimestamp

Look for warnings related to leader election.

Failure Modes and Recovery

Understanding what can go wrong with Leases and how to recover is critical for maintaining highly available clusters. This section covers common failure modes and step-by-step recovery procedures.

Failure Mode 1: Leader Pod Crashes

Symptom: The holder pod is not running, but the lease has not been released (holder still shows the dead pod).

Cause: The pod crashed and did not have a chance to release the lease.

Recovery: The lease will expire after leaseDurationSeconds (default 15s for control plane components). The standby replicas will then compete for leadership. Generally, no manual intervention is needed; just wait for the lease to expire and a new leader to emerge.

Verify recovery:

kubectl get lease kube-controller-manager -n kube-system -w

Watch the holder change from the dead pod to a live one within ~20-30 seconds.

Failure Mode 2: Lease Expires but No New Leader

Symptom: Lease holder is empty, and all pods are running, but no leader is elected.

Cause: Possible issues: RBAC misconfiguration preventing lease updates, network partition between nodes, or all candidates are failing to acquire due to API errors.

Recovery Steps:

1. Check logs of candidate pods for errors:

   kubectl logs -n kube-system kube-controller-manager-<node1> | grep -i "leader\|lease"
   kubectl logs -n kube-system kube-controller-manager-<node2> | grep -i "leader\|lease"

1. Check RBAC permissions for the component's service account. For control plane components, the system:kube-controller-manager or system:kube-scheduler cluster roles should have permissions on leases. You can check with:

   kubectl auth can-i update leases -n kube-system --as=system:serviceaccount:kube-system:kube-controller-manager

Expected output: yes

1. If permissions are missing, apply the appropriate ClusterRole and ClusterRoleBinding (usually part of the Kubernetes installation, so a misconfiguration may indicate a broken cluster setup).
2. If all else fails, you can manually delete the Lease object; the next candidate will create it anew:

   kubectl delete lease kube-controller-manager -n kube-system

This should trigger immediate leader election.

Failure Mode 3: Split Brain Due to Clock Skew

Symptom: Two components believe they are leader simultaneously.

Cause: Large clock skew between nodes can cause the non-leader to think the lease has expired and acquire it, while the old leader continues to operate.

Recovery: Fix clock synchronization (e.g., ensure NTP is running on all nodes). Then, if necessary, force a new election by deleting the lease (as above). Going forward, ensure all nodes have synchronized clocks.

Failure Mode 4: Lease Object Deleted Accidentally

Symptom: Leader election fails or a new leader is elected unexpectedly.

Cause: Someone deleted the Lease object.

Recovery: The candidates will recreate it automatically. No manual action needed unless the component is not designed to retry creation (rare). If necessary, restart the component pods.

General Recovery Verification

After any recovery action, verify the cluster is functioning normally:

• Check component status: kubectl get componentstatuses (if available) or kubectl get pods -n kube-system.
• Check leader election: kubectl get leases -n kube-system.
• Ensure workloads are scheduled: kubectl get pods --all-namespaces and look for pending pods.

Operations Checklist

Use this checklist to ensure you've covered all aspects of Lease management and troubleshooting. Each item is concrete and actionable.

Pre-Change Checklist

• [ ] Verify cluster version: kubectl version (ensure >=1.14).
• [ ] List current leases: kubectl get leases --all-namespaces.
• [ ] Note down current holders for critical components (kube-controller-manager, kube-scheduler) in a secure notepad.
• [ ] Confirm you have kubectl access and permissions.
• [ ] Identify the blast radius of any potential change (e.g., deleting a lease affects leader election for that component).
• [ ] Set up a watch on the lease (in a separate terminal): kubectl get lease <name> -n <namespace> -w.

During Change

• [ ] If deleting a lease to force election, use:

  kubectl delete lease <name> -n <namespace>

Immediately watch the output of the watch command.

• [ ] For custom component configuration changes (e.g., lease duration), update the deployment spec or leader election flags, then apply:

  kubectl apply -f <updated-manifest.yaml>

• [ ] Monitor logs of the component to see leader election messages:

  kubectl logs -f deployment/<component> -n <namespace>

Post-Change Verification

• [ ] Verify new holder: kubectl get lease <name> -n <namespace> -o yaml and check holderIdentity.
• [ ] Verify renewTime is updating (run watch -n 5 kubectl get lease <name> -n <namespace> -o yaml and see the renewTime change).
• [ ] Check component pod status: kubectl get pods -n <namespace> | grep <component>.
• [ ] Test functionality: for kube-scheduler, try scheduling a test pod; for controllers, trigger a reconciliation.
• [ ] Document any changes in your operations log.

Troubleshooting Quick Reference

Conclusion

Kubernetes Leases are a powerful yet underappreciated mechanism for coordination and leader election. By understanding their internals—the spec fields, renewal process, and failure modes—you can operate your cluster and custom controllers with greater reliability.

This article walked through the version and environment inventory, safe configuration practices, verification and diagnostics, failure modes, and an operations checklist. The key takeaways are:

• Always observe before changing: use kubectl get leases and kubectl describe lease to understand the current state.
• Leader election relies on timely renewals; monitor renewTime to ensure health.
• Common failures include crashed leaders, clock skew, and RBAC issues; each has a clear recovery path.
• Use the operations checklist to systematically manage Lease-related changes.

As a next step, apply these concepts to a low-risk scenario: create a test Lease using the example manifest, observe its behavior, and simulate a failure by deleting the holder pod (if using a custom controller) or manually updating the lease. Record the results and compare them with the expected behavior described here.

Remember, a reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. With these Lease advanced concepts, you're better equipped to keep your Kubernetes environment stable and highly available.