## Intro Kubernetes Leases are coordination primitives that let distributed components safely elect leaders, claim exclusive access, or heartbeat their liveness. The kube-controller-manager and kube-scheduler use Leases internally for leader election, and many custom controllers rely on Leases for similar purposes. Manually managing Lease objects is error-prone: names, namespaces, holder identities, and durations must match exactly, and an incorrect change can disrupt leader election or cause split-brain behavior. Automating Lease management through CI/CD pipelines brings repeatability, auditability, and safety to these sensitive objects. This guide walks through a practical implementation: inventorying your environment, creating a scoped configuration, verifying deployments, handling failures, and establishing an operational checklist. By the end, you will be able to deploy Lease changes through a pipeline with confidence. ## Understand Leases and Their Lifecycle Before automating, take a moment to understand what a Lease object contains and how it behaves. A Lease has two main sections: `spec` and `status`. The `spec` defines the desired coordination parameters: - `holderIdentity`: the current holder of the lease, typically a pod name or hostname plus a UUID. When empty, the lease is available. - `leaseDurationSeconds`: how long a holder can retain the lease without renewing. Common values are 15, 30, or 60 seconds. - `acquireTime`: when the current holder acquired the lease (set by the system). - `renewTime`: when the current holder last renewed the lease (updated by the holder). - `leaseTransitions`: how many times the lease has changed hands. The `status` field is used by the system to track the observed state, but for most automation purposes, you only manage `spec`. When a component wants to become the leader, it attempts to create or update a Lease with its own `holderIdentity` and a `renewTime`. If the current holder fails to renew before `leaseDurationSeconds` elapses, another component can acquire the lease. This mechanism prevents two components from acting as leader simultaneously. For CI/CD, you typically manage the existence and initial state of Leases, not their live updates. Your pipeline will apply a desired manifest that either creates a Lease with no holder (letting the runtime claim it) or updates a non-critical field like `leaseDurationSeconds`. Changing `holderIdentity` manually is risky and should be done only when you understand the component's behavior. ## Version and Environment Inventory Before automating Leases, establish exactly what you are working with. A typical environment includes: - Kubernetes cluster version (e.g., v1.28.3) - kubectl client version (must be within one minor version of the server) - Namespace where Leases live (e.g., `kube-node-lease` for node heartbeats, `kube-system` for controller leader election) - Existing Lease objects and their expected holder identities - Access credentials and RBAC permissions for your CI/CD service account Collect the cluster and client versions with: ```bash kubectl version --short ``` Example output: ``` Client Version: v1.28.3 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.28.3 ``` List existing Leases across all namespaces to see the landscape: ```bash kubectl get leases --all-namespaces ``` Example output: ``` NAMESPACE NAME HOLDER AGE kube-node-lease ip-10-0-1-23 ip-10-0-1-23 5d kube-system kube-controller-manager ip-10-0-0-10_1c9b8e2e-2f7a-4ad9-a3b1-9b1f0e6d6a6f 7d kube-system kube-scheduler ip-10-0-0-10_2b4a1a5e-6d8e-4c4e-a5d4-6f9c8e4f1c2e 7d ``` Take note of the holder identity format: for leader election, it is typically `_`. Ensure your CI/CD service account has `get`, `list`, `create`, `update`, and `patch` permissions on Leases in the relevant namespaces. A minimal ClusterRole might look like: ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: lease-manager rules: - apiGroups: ["coordination.k8s.io"] resources: ["leases"] verbs: ["get", "list", "create", "update", "patch"] ``` Do not grant `delete` unless your workflow specifically requires removing Leases, which can force an immediate leader election. Additionally, document the responsible owner for Lease automation. This person reviews and approves changes to Lease manifests. For example, assign Priya Shah, Engineering Lead, as the owner. She reviews changes monthly or whenever a new Lease is added. ## Design a Scoped and Safe Configuration A safe Lease automation starts with a narrow, measurable pilot. Choose a single Lease that is not on the critical path, such as a test Lease in a development namespace. Define the Lease in a Git repository as a manifest, then apply it through your pipeline. Here is an example manifest for a custom leader election Lease: ```yaml apiVersion: coordination.k8s.io/v1 kind: Lease metadata: name: myapp-leader namespace: dev spec: holderIdentity: "" leaseDurationSeconds: 15 acquireTime: null renewTime: null leaseTransitions: 0 ``` At creation, leave `holderIdentity` empty; the application runtime will claim it. For a static Lease that preassigns a holder, you can set `holderIdentity`, but be cautious: if that holder is not actually running, the Lease will be stale. Store all Lease manifests in a dedicated directory in your Git repository, such as `leases/`. This makes it easy to review changes and apply them selectively. Use Kubernetes namespaces to isolate Leases. If you manage multiple applications, consider using a naming convention like `-leader` to avoid confusion. ## Build the CI/CD Pipeline for Lease Deployments Implement the pipeline in stages: - **Lint and validate**: Use `kubectl apply --dry-run=client` to check syntax. - **Diff against live state**: Use `kubectl diff` to see what would change. - **Apply to a staging namespace**: If possible, test in a non-production cluster first. - **Apply to production with approval**: Add a manual approval gate before the production apply. - **Post-apply verification**: Run checks to confirm the Lease is correct. Below are concrete examples for each stage using common CI tools. ### Stage 1: Dry-run validation Always validate the manifest before attempting any changes. This catches syntax errors and schema violations. ```bash kubectl apply -f leases/myapp-leader.yaml --dry-run=client ``` If the command exits with a non-zero status, the manifest is invalid. In a pipeline, use: ```bash if ! kubectl apply -f leases/myapp-leader.yaml --dry-run=client; then echo "Validation failed for leases/myapp-leader.yaml" exit 1 fi ``` Expected successful output: ``` lease.coordination.k8s.io/myapp-leader created (dry run) ``` ### Stage 2: Diff against live state If the Lease already exists, use `kubectl diff` to see the exact changes. This is a critical safety check. ```bash kubectl diff -f leases/myapp-leader.yaml || true ``` The `|| true` prevents the pipeline from failing if there are differences (since `kubectl diff` returns 1 when differences are found). Review the diff output. Example output when updating `leaseDurationSeconds` from 15 to 30: ```diff --- live +++ merged @@ -7,7 +7,7 @@ holderIdentity: myapp-instance-1 - leaseDurationSeconds: 15 + leaseDurationSeconds: 30 acquireTime: "2025-03-01T12:00:00Z" ``` Review the diff carefully; changing `leaseDurationSeconds` affects how quickly a new leader is elected after a failure. ### Stage 3: Apply to staging If you have a staging cluster or namespace, apply the manifest there first and run verification checks. This can be automated but should include some assertions. ```bash kubectl apply -f leases/myapp-leader.yaml --namespace staging ``` Then verify the Lease exists and has expected fields: ```bash kubectl get lease myapp-leader -n staging -o yaml ``` ### Stage 4: Manual approval gate for production Before applying to production, require a human to approve the change. In GitHub Actions, you can use an environment with required reviewers. In GitLab CI, use a `when: manual` job. Example GitLab CI configuration: ```yaml deploy_production: stage: deploy script: - kubectl apply -f leases/myapp-leader.yaml --namespace production environment: name: production when: manual only: - main ``` ### Stage 5: Post-apply verification After applying to production, run the verification script described in the next section. If it fails, the pipeline should roll back or alert. Example combined pipeline script for a simple shell-based CI job: ```bash #!/bin/bash set -e # Dry-run validation kubectl apply -f leases/myapp-leader.yaml --dry-run=client # Show diff (only if object already exists) kubectl diff -f leases/myapp-leader.yaml || true # Apply kubectl apply -f leases/myapp-leader.yaml # Verify ./verify-lease.sh "myapp-leader" "dev" "myapp-instance-1" ``` In the script above, the owner (Priya Shah) receives an email or Slack notification when a production apply completes. ## Verification and Diagnostics After applying a Lease, verify its fields and status. Use `kubectl get` with output formatting: ```bash kubectl get lease myapp-leader -n dev -o yaml ``` Expected output snippet: ```yaml apiVersion: coordination.k8s.io/v1 kind: Lease metadata: name: myapp-leader namespace: dev spec: holderIdentity: myapp-instance-1 leaseDurationSeconds: 30 acquireTime: "2025-03-01T12:30:00Z" renewTime: "2025-03-01T12:31:00Z" leaseTransitions: 1 ``` Check that `holderIdentity` is as expected and that `renewTime` is recent (within `leaseDurationSeconds`). If the holder does not renew, a new election occurs. For automated verification, write a script that queries the Lease and fails the pipeline if fields are out of bounds. Example: ```bash #!/bin/bash # verify-lease.sh LEASE_NAME="$1" NAMESPACE="$2" EXPECTED_HOLDER="$3" HOLDER=$(kubectl get lease "$LEASE_NAME" -n "$NAMESPACE" -o jsonpath='{.spec.holderIdentity}') if [ "$HOLDER" != "$EXPECTED_HOLDER" ]; then echo "Unexpected holder: $HOLDER (expected $EXPECTED_HOLDER)" exit 1 fi RENEW_TIME=$(kubectl get lease "$LEASE_NAME" -n "$NAMESPACE" -o jsonpath='{.spec.renewTime}') if [ -z "$RENEW_TIME" ] || [ "$RENEW_TIME" == "null" ]; then echo "Renew time is null; lease may not be held" exit 1 fi # Compare timestamps with a tolerance of 2 * leaseDurationSeconds (simplified using epoch). LEASE_DURATION=$(kubectl get lease "$LEASE_NAME" -n "$NAMESPACE" -o jsonpath='{.spec.leaseDurationSeconds}') TOLERANCE=$((LEASE_DURATION * 2)) CURRENT_EPOCH=$(date -u +%s) RENEW_EPOCH=$(date -u -d "$RENEW_TIME" +%s) DIFF=$((CURRENT_EPOCH - RENEW_EPOCH)) if [ "$DIFF" -gt "$TOLERANCE" ]; then echo "Renew time is too old: $RENEW_TIME (current: $(date -u +%Y-%m-%dT%H:%M:%SZ))" exit 1 fi echo "Lease $LEASE_NAME in $NAMESPACE is healthy: holder=$HOLDER, renewTime=$RENEW_TIME" ``` Run it as: ```bash ./verify-lease.sh myapp-leader dev myapp-instance-1 ``` Expected output: ``` Lease myapp-leader in dev is healthy: holder=myapp-instance-1, renewTime=2025-03-01T12:31:00Z ``` Additionally, watch events in the namespace for Lease-related issues: ```bash kubectl get events -n dev --field-selector involvedObject.name=myapp-leader ``` Example output: ``` LAST SEEN TYPE REASON OBJECT MESSAGE 2m Normal LeaderElection lease/myapp-leader myapp-instance-1 became leader ``` No events may also be a good sign; it means the Lease is stable. Set up alerting for critical Leases. For example, use a monitoring system like Prometheus to scrape the `renewTime` and alert if it is older than `leaseDurationSeconds` for more than two consecutive intervals. ## Common Pitfalls and How to Avoid Them Automating Leases introduces specific risks. Here are common pitfalls and how to handle them. ### Pitfall 1: Accidental changes to holderIdentity It is tempting to manually set `holderIdentity` to force a particular leader. However, if the specified holder is not running or is unhealthy, the cluster may enter a state where no component can acquire the lease because the holder field is non-empty but the holder never renews. This can stall leader election. **Why it happens**: A developer may edit the manifest to set a holder during debugging or to "pre-assign" a leader. **Avoidance**: Keep `holderIdentity` empty in version-controlled manifests unless you have a specific static use case. The application runtime should be responsible for setting and renewing the holder. **Recovery**: If a stale holder is set, run `kubectl patch lease -n --type merge -p '{"spec":{"holderIdentity":""}}'` to clear it, allowing a new election. ### Pitfall 2: Misconfigured leaseDurationSeconds A value that is too short may cause frequent leader changes under normal load, as the holder cannot renew in time. A value too long delays failover when the leader dies, causing downtime. **Why it happens**: Copying a manifest from another application without adjusting the duration to match the application's renewal interval. **Avoidance**: Understand the application's renewal interval. Typically, the holder renews at least every `leaseDurationSeconds / 2`. Set the duration to at least twice the renewal interval, plus buffer. For example, if the application renews every 5 seconds, set `leaseDurationSeconds` to 15 or 20. **Recovery**: Update the manifest with the correct value and apply. The change takes effect for the next acquisition. ### Pitfall 3: Applying to the wrong namespace A Lease manifest may have a `metadata.namespace` field, but CI systems sometimes override it with a default or misconfigured kubeconfig context. This can create a Lease in an unintended namespace, leading to confusion and possibly a second leader. **Why it happens**: Relying on the default namespace in the kubeconfig instead of explicitly setting the namespace in the manifest or the command. **Avoidance**: Always specify `namespace` in the manifest. In pipeline commands, add `--namespace` explicitly. Use separate kubeconfig contexts per environment. **Recovery**: Delete the incorrectly created Lease (if safe) and apply to the correct namespace. Review the pipeline configuration. ### Pitfall 4: Deleting a Lease accidentally A pipeline may include a `kubectl delete` step for cleanup, but without guards it could delete an active Lease, forcing an immediate election and potential disruption. **Why it happens**: A generic cleanup job or a misconfigured manifest that sets `metadata.deletionTimestamp`. **Avoidance**: Do not grant `delete` permission to the CI/CD service account. If deletion is necessary, add an explicit confirmation and a check that the Lease is not currently held (`holderIdentity` is empty or old). **Recovery**: Recreate the Lease from the manifest; the application will re-elect a leader. Monitor for errors. ### Pitfall 5: Not verifying after apply Without post-apply checks, a misapplied manifest may go unnoticed until a failure occurs. **Why it happens**: Assuming that a successful `kubectl apply` means the change is correct. **Avoidance**: Always include a verification script as a pipeline step. The script should assert holder identity, renew time freshness, and optionally lease duration. **Recovery**: Immediately run the verification script manually and roll back if needed. ## Rollback and Recovery Procedures Even with careful planning, a Lease change can cause problems. Here is a concrete rollback procedure. 1. Identify the previous good manifest from Git history. For example, use `git log` to find the commit before the change: ```bash git log --oneline -- leases/myapp-leader.yaml ``` 2. Revert the file to the previous version: ```bash git checkout -- leases/myapp-leader.yaml ``` This does not commit the change; it stages it. Then apply the old manifest: ```bash kubectl apply -f leases/myapp-leader.yaml ``` 3. Verify the Lease is healthy with the verification script. 4. Observe the application for a few lease cycles (e.g., 3 times `leaseDurationSeconds`) to ensure stability. 5. Commit the rollback to Git with a clear message. 6. The accountable owner (e.g., Priya Shah) reviews the rollback and updates the runbook with the incident details. If the problem was caused by an external factor (e.g., the application misbehaving), address that separately. ## Operations Checklist Use this checklist for every Lease automation run and for periodic review. Store this checklist in the repository next to the Lease manifests. The owner for this checklist is the Engineering Lead, reviewed monthly. | Step | Action | Expected Result | | :--- | :--- | :--- | | 1 | Verify cluster and client versions | Server and client within one minor version | | 2 | Check RBAC permissions for CI/CD service account | Can get, list, create, update, patch Leases in target namespaces | | 3 | Review manifest changes via diff | Only intended fields changed | | 4 | Dry-run apply | No syntax errors | | 5 | Apply to staging (if available) | Lease created or updated as expected | | 6 | Manual approval before production | Human sign-off recorded | | 7 | Apply to production | Successful apply message | | 8 | Verify holder identity and renew time | Holder matches expected; renew time recent | | 9 | Check events for leader election | Normal leader election events or none | | 10 | Document any anomalies | Log entry in runbook | Run this checklist for each Lease change. For critical Leases (e.g., kube-controller-manager), consider peer review of the manifest before applying. ## Conclusion Automating Kubernetes Lease workflows with CI/CD is a practical way to manage coordination primitives safely. By inventorying your environment, implementing a scoped pipeline with validation and diffing, verifying deployments, and preparing for failure, you reduce manual errors and gain repeatability. Start with a narrow pilot, as recommended for any new automation, then expand to more Leases as confidence grows. Remember to always review diffs, avoid accidental deletes, and have a rollback plan. With these practices, your Lease management becomes a controlled, auditable process. As next steps, consider integrating your verification checks into a monitoring system and extending automation to other coordination resources like ConfigMaps used for leader election. Regularly revisit the owner assignment and checklist to adapt to changes.