Automating Kubernetes Lease Workflows with CI/CD: A Practical Implementation Guide

Intro

Kubernetes Leases are coordination primitives that let distributed components safely elect leaders, claim exclusive access, or heartbeat their liveness. The kube-controller-manager and kube-scheduler use Leases internally for leader election, and many custom controllers rely on Leases for similar purposes. Manually managing Lease objects is error-prone: names, namespaces, holder identities, and durations must match exactly, and an incorrect change can disrupt leader election or cause split-brain behavior. Automating Lease management through CI/CD pipelines brings repeatability, auditability, and safety to these sensitive objects.

This guide walks through a practical implementation: inventorying your environment, creating a scoped configuration, verifying deployments, handling failures, and establishing an operational checklist. By the end, you will be able to deploy Lease changes through a pipeline with confidence.

Understand Leases and Their Lifecycle

Before automating, take a moment to understand what a Lease object contains and how it behaves. A Lease has two main sections: spec and status. The spec defines the desired coordination parameters:

• holderIdentity: the current holder of the lease, typically a pod name or hostname plus a UUID. When empty, the lease is available.
• leaseDurationSeconds: how long a holder can retain the lease without renewing. Common values are 15, 30, or 60 seconds.
• acquireTime: when the current holder acquired the lease (set by the system).
• renewTime: when the current holder last renewed the lease (updated by the holder).
• leaseTransitions: how many times the lease has changed hands.

The status field is used by the system to track the observed state, but for most automation purposes, you only manage spec. When a component wants to become the leader, it attempts to create or update a Lease with its own holderIdentity and a renewTime. If the current holder fails to renew before leaseDurationSeconds elapses, another component can acquire the lease. This mechanism prevents two components from acting as leader simultaneously.

For CI/CD, you typically manage the existence and initial state of Leases, not their live updates. Your pipeline will apply a desired manifest that either creates a Lease with no holder (letting the runtime claim it) or updates a non-critical field like leaseDurationSeconds. Changing holderIdentity manually is risky and should be done only when you understand the component's behavior.

Version and Environment Inventory

Before automating Leases, establish exactly what you are working with. A typical environment includes:

• Kubernetes cluster version (e.g., v1.28.3)
• kubectl client version (must be within one minor version of the server)
• Namespace where Leases live (e.g., kube-node-lease for node heartbeats, kube-system for controller leader election)
• Existing Lease objects and their expected holder identities
• Access credentials and RBAC permissions for your CI/CD service account

Collect the cluster and client versions with:

kubectl version --short

Example output:

Client Version: v1.28.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.3

List existing Leases across all namespaces to see the landscape:

kubectl get leases --all-namespaces

Example output:

NAMESPACE        NAME                        HOLDER                                        AGE
kube-node-lease  ip-10-0-1-23               ip-10-0-1-23                                  5d
kube-system      kube-controller-manager     ip-10-0-0-10_1c9b8e2e-2f7a-4ad9-a3b1-9b1f0e6d6a6f   7d
kube-system      kube-scheduler              ip-10-0-0-10_2b4a1a5e-6d8e-4c4e-a5d4-6f9c8e4f1c2e   7d

Take note of the holder identity format: for leader election, it is typically <hostname>_<uuid>. Ensure your CI/CD service account has get, list, create, update, and patch permissions on Leases in the relevant namespaces. A minimal ClusterRole might look like:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: lease-manager
rules:
- apiGroups: ["coordination.k8s.io"]
  resources: ["leases"]
  verbs: ["get", "list", "create", "update", "patch"]

Do not grant delete unless your workflow specifically requires removing Leases, which can force an immediate leader election.

Additionally, document the responsible owner for Lease automation. This person reviews and approves changes to Lease manifests. For example, assign Priya Shah, Engineering Lead, as the owner. She reviews changes monthly or whenever a new Lease is added.

Design a Scoped and Safe Configuration

A safe Lease automation starts with a narrow, measurable pilot. Choose a single Lease that is not on the critical path, such as a test Lease in a development namespace. Define the Lease in a Git repository as a manifest, then apply it through your pipeline.

Here is an example manifest for a custom leader election Lease:

apiVersion: coordination.k8s.io/v1
kind: Lease
metadata:
  name: myapp-leader
  namespace: dev
spec:
  holderIdentity: ""
  leaseDurationSeconds: 15
  acquireTime: null
  renewTime: null
  leaseTransitions: 0

At creation, leave holderIdentity empty; the application runtime will claim it. For a static Lease that preassigns a holder, you can set holderIdentity, but be cautious: if that holder is not actually running, the Lease will be stale.

Store all Lease manifests in a dedicated directory in your Git repository, such as leases/. This makes it easy to review changes and apply them selectively.

Use Kubernetes namespaces to isolate Leases. If you manage multiple applications, consider using a naming convention like <app-name>-leader to avoid confusion.

Build the CI/CD Pipeline for Lease Deployments

Implement the pipeline in stages:

• Lint and validate: Use kubectl apply --dry-run=client to check syntax.
• Diff against live state: Use kubectl diff to see what would change.
• Apply to a staging namespace: If possible, test in a non-production cluster first.
• Apply to production with approval: Add a manual approval gate before the production apply.
• Post-apply verification: Run checks to confirm the Lease is correct.

Below are concrete examples for each stage using common CI tools.

Stage 1: Dry-run validation

Always validate the manifest before attempting any changes. This catches syntax errors and schema violations.

kubectl apply -f leases/myapp-leader.yaml --dry-run=client

If the command exits with a non-zero status, the manifest is invalid. In a pipeline, use:

if ! kubectl apply -f leases/myapp-leader.yaml --dry-run=client; then
  echo "Validation failed for leases/myapp-leader.yaml"
  exit 1
fi

Expected successful output:

lease.coordination.k8s.io/myapp-leader created (dry run)

Stage 2: Diff against live state

If the Lease already exists, use kubectl diff to see the exact changes. This is a critical safety check.

kubectl diff -f leases/myapp-leader.yaml || true

The || true prevents the pipeline from failing if there are differences (since kubectl diff returns 1 when differences are found). Review the diff output.

Example output when updating leaseDurationSeconds from 15 to 30:

--- live
+++ merged
@@ -7,7 +7,7 @@
 holderIdentity: myapp-instance-1
- leaseDurationSeconds: 15
+ leaseDurationSeconds: 30
 acquireTime: "2025-03-01T12:00:00Z"

Review the diff carefully; changing leaseDurationSeconds affects how quickly a new leader is elected after a failure.

Stage 3: Apply to staging

If you have a staging cluster or namespace, apply the manifest there first and run verification checks. This can be automated but should include some assertions.

kubectl apply -f leases/myapp-leader.yaml --namespace staging

Then verify the Lease exists and has expected fields:

kubectl get lease myapp-leader -n staging -o yaml

Stage 4: Manual approval gate for production

Before applying to production, require a human to approve the change. In GitHub Actions, you can use an environment with required reviewers. In GitLab CI, use a when: manual job.

Example GitLab CI configuration:

deploy_production:
  stage: deploy
  script:
    - kubectl apply -f leases/myapp-leader.yaml --namespace production
  environment:
    name: production
  when: manual
  only:
    - main

Stage 5: Post-apply verification

After applying to production, run the verification script described in the next section. If it fails, the pipeline should roll back or alert.

Example combined pipeline script for a simple shell-based CI job:

#!/bin/bash
set -e

# Dry-run validation
kubectl apply -f leases/myapp-leader.yaml --dry-run=client

# Show diff (only if object already exists)
kubectl diff -f leases/myapp-leader.yaml || true

# Apply
kubectl apply -f leases/myapp-leader.yaml

# Verify
./verify-lease.sh "myapp-leader" "dev" "myapp-instance-1"

In the script above, the owner (Priya Shah) receives an email or Slack notification when a production apply completes.

Verification and Diagnostics

After applying a Lease, verify its fields and status. Use kubectl get with output formatting:

kubectl get lease myapp-leader -n dev -o yaml

Expected output snippet:

apiVersion: coordination.k8s.io/v1
kind: Lease
metadata:
  name: myapp-leader
  namespace: dev
spec:
  holderIdentity: myapp-instance-1
  leaseDurationSeconds: 30
  acquireTime: "2025-03-01T12:30:00Z"
  renewTime: "2025-03-01T12:31:00Z"
  leaseTransitions: 1

Check that holderIdentity is as expected and that renewTime is recent (within leaseDurationSeconds). If the holder does not renew, a new election occurs.

For automated verification, write a script that queries the Lease and fails the pipeline if fields are out of bounds. Example:

#!/bin/bash
# verify-lease.sh
LEASE_NAME="$1"
NAMESPACE="$2"
EXPECTED_HOLDER="$3"

HOLDER=$(kubectl get lease "$LEASE_NAME" -n "$NAMESPACE" -o jsonpath='{.spec.holderIdentity}')
if [ "$HOLDER" != "$EXPECTED_HOLDER" ]; then
  echo "Unexpected holder: $HOLDER (expected $EXPECTED_HOLDER)"
  exit 1
fi

RENEW_TIME=$(kubectl get lease "$LEASE_NAME" -n "$NAMESPACE" -o jsonpath='{.spec.renewTime}')
if [ -z "$RENEW_TIME" ] || [ "$RENEW_TIME" == "null" ]; then
  echo "Renew time is null; lease may not be held"
  exit 1
fi

# Compare timestamps with a tolerance of 2 * leaseDurationSeconds (simplified using epoch).
LEASE_DURATION=$(kubectl get lease "$LEASE_NAME" -n "$NAMESPACE" -o jsonpath='{.spec.leaseDurationSeconds}')
TOLERANCE=$((LEASE_DURATION * 2))
CURRENT_EPOCH=$(date -u +%s)
RENEW_EPOCH=$(date -u -d "$RENEW_TIME" +%s)
DIFF=$((CURRENT_EPOCH - RENEW_EPOCH))
if [ "$DIFF" -gt "$TOLERANCE" ]; then
  echo "Renew time is too old: $RENEW_TIME (current: $(date -u +%Y-%m-%dT%H:%M:%SZ))"
  exit 1
fi

echo "Lease $LEASE_NAME in $NAMESPACE is healthy: holder=$HOLDER, renewTime=$RENEW_TIME"

Run it as:

./verify-lease.sh myapp-leader dev myapp-instance-1

Expected output:

Lease myapp-leader in dev is healthy: holder=myapp-instance-1, renewTime=2025-03-01T12:31:00Z

Additionally, watch events in the namespace for Lease-related issues:

kubectl get events -n dev --field-selector involvedObject.name=myapp-leader

Example output:

LAST SEEN   TYPE     REASON            OBJECT              MESSAGE
2m          Normal   LeaderElection    lease/myapp-leader  myapp-instance-1 became leader

No events may also be a good sign; it means the Lease is stable.

Set up alerting for critical Leases. For example, use a monitoring system like Prometheus to scrape the renewTime and alert if it is older than leaseDurationSeconds for more than two consecutive intervals.

Common Pitfalls and How to Avoid Them

Automating Leases introduces specific risks. Here are common pitfalls and how to handle them.

Pitfall 1: Accidental changes to holderIdentity

It is tempting to manually set holderIdentity to force a particular leader. However, if the specified holder is not running or is unhealthy, the cluster may enter a state where no component can acquire the lease because the holder field is non-empty but the holder never renews. This can stall leader election.

Why it happens: A developer may edit the manifest to set a holder during debugging or to "pre-assign" a leader.

Avoidance: Keep holderIdentity empty in version-controlled manifests unless you have a specific static use case. The application runtime should be responsible for setting and renewing the holder.

Recovery: If a stale holder is set, run kubectl patch lease <name> -n <ns> --type merge -p '{"spec":{"holderIdentity":""}}' to clear it, allowing a new election.

Pitfall 2: Misconfigured leaseDurationSeconds

A value that is too short may cause frequent leader changes under normal load, as the holder cannot renew in time. A value too long delays failover when the leader dies, causing downtime.

Why it happens: Copying a manifest from another application without adjusting the duration to match the application's renewal interval.

Avoidance: Understand the application's renewal interval. Typically, the holder renews at least every leaseDurationSeconds / 2. Set the duration to at least twice the renewal interval, plus buffer. For example, if the application renews every 5 seconds, set leaseDurationSeconds to 15 or 20.

Recovery: Update the manifest with the correct value and apply. The change takes effect for the next acquisition.

Pitfall 3: Applying to the wrong namespace

A Lease manifest may have a metadata.namespace field, but CI systems sometimes override it with a default or misconfigured kubeconfig context. This can create a Lease in an unintended namespace, leading to confusion and possibly a second leader.

Why it happens: Relying on the default namespace in the kubeconfig instead of explicitly setting the namespace in the manifest or the command.

Avoidance: Always specify namespace in the manifest. In pipeline commands, add --namespace explicitly. Use separate kubeconfig contexts per environment.

Recovery: Delete the incorrectly created Lease (if safe) and apply to the correct namespace. Review the pipeline configuration.

Pitfall 4: Deleting a Lease accidentally

A pipeline may include a kubectl delete step for cleanup, but without guards it could delete an active Lease, forcing an immediate election and potential disruption.

Why it happens: A generic cleanup job or a misconfigured manifest that sets metadata.deletionTimestamp.

Avoidance: Do not grant delete permission to the CI/CD service account. If deletion is necessary, add an explicit confirmation and a check that the Lease is not currently held (holderIdentity is empty or old).

Recovery: Recreate the Lease from the manifest; the application will re-elect a leader. Monitor for errors.

Pitfall 5: Not verifying after apply

Without post-apply checks, a misapplied manifest may go unnoticed until a failure occurs.

Why it happens: Assuming that a successful kubectl apply means the change is correct.

Avoidance: Always include a verification script as a pipeline step. The script should assert holder identity, renew time freshness, and optionally lease duration.

Recovery: Immediately run the verification script manually and roll back if needed.

Rollback and Recovery Procedures

Even with careful planning, a Lease change can cause problems. Here is a concrete rollback procedure.

1. Identify the previous good manifest from Git history. For example, use git log to find the commit before the change:

git log --oneline -- leases/myapp-leader.yaml

1. Revert the file to the previous version:

git checkout <previous-commit> -- leases/myapp-leader.yaml

This does not commit the change; it stages it. Then apply the old manifest:

kubectl apply -f leases/myapp-leader.yaml

1. Verify the Lease is healthy with the verification script.

1. Observe the application for a few lease cycles (e.g., 3 times leaseDurationSeconds) to ensure stability.

1. Commit the rollback to Git with a clear message.

1. The accountable owner (e.g., Priya Shah) reviews the rollback and updates the runbook with the incident details.

If the problem was caused by an external factor (e.g., the application misbehaving), address that separately.

Operations Checklist

Use this checklist for every Lease automation run and for periodic review. Store this checklist in the repository next to the Lease manifests. The owner for this checklist is the Engineering Lead, reviewed monthly.

Run this checklist for each Lease change. For critical Leases (e.g., kube-controller-manager), consider peer review of the manifest before applying.

Conclusion

Automating Kubernetes Lease workflows with CI/CD is a practical way to manage coordination primitives safely. By inventorying your environment, implementing a scoped pipeline with validation and diffing, verifying deployments, and preparing for failure, you reduce manual errors and gain repeatability. Start with a narrow pilot, as recommended for any new automation, then expand to more Leases as confidence grows. Remember to always review diffs, avoid accidental deletes, and have a rollback plan. With these practices, your Lease management becomes a controlled, auditable process.

As next steps, consider integrating your verification checks into a monitoring system and extending automation to other coordination resources like ConfigMaps used for leader election. Regularly revisit the owner assignment and checklist to adapt to changes.