## Intro Kubernetes Pod Disruption Budgets (PDBs) define the minimum number of pods that must remain available during voluntary disruptions such as node drains, cluster upgrades, or maintenance operations. Automating PDB management in CI/CD pipelines ensures that availability policies are applied consistently across environments, reducing human error and preventing accidental downtime. This guide provides a practical, step-by-step approach to integrating PDB automation into your delivery workflow, from initial environment inventory to failure recovery. We focus on practitioners: developers, DevOps engineers, and technical teams running production Kubernetes clusters. By the end of this article, you will understand how to: - Discover current PDBs and their status - Safely modify PDBs using declarative manifests - Integrate PDB changes into CI/CD pipelines with automated validation - Diagnose common PDB-related issues - Implement rollback strategies when things go wrong The goal is operational safety: observe before changing, limit blast radius, use placeholders instead of secrets, verify results, and document recovery paths. ## Version and Environment Inventory Before automating PDBs, establish a clear picture of your cluster and tooling. PDBs are supported in Kubernetes v1.21+ (stable), but behavior may vary slightly by version. Check your cluster version with: kubectl version --short Expected output includes both client and server versions. For example: Client Version: v1.27.3 Server Version: v1.27.3 Next, inventory existing PDBs across all namespaces: kubectl get pdb --all-namespaces This command lists every PDB in the cluster, showing namespace, name, min available/max unavailable, allowed disruptions, and age. Example output: NAMESPACE NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE default app-pdb N/A 1 1 2d kube-system coredns-pdb 1 N/A 0 30d No PDBs? You will need to create them as part of your automation. Also note which controllers manage your workloads, as PDBs rely on labels selectors matching pods. For this article, we assume a typical Deployment-based application called webapp in the default namespace. Check current pod labels to ensure your PDB selector will match: kubectl get pods -l app=webapp --show-labels Expected output shows labels like app=webapp . If labels are inconsistent, fix them before proceeding. Keep your local testing environment small. Use a tool like minikube or kind for safe experimentation before touching production. Apply one manifest at a time and verify with kubectl port-forward or a local service before moving to cloud load balancers or ingress controllers.
Quick check 1 of 2 What does a PodDisruptionBudget (PDB) limit? - The number of replicas a Deployment can have - The number of Pods of a replicated application that can be down simultaneously from voluntary disruptions - The number of nodes in a cluster - The number of Pods that can be created in a namespace
According to the reference, a PDB limits the number of Pods of a replicated application that are down simultaneously from voluntary disruptions.
## Safe Configuration Path PDBs are defined using YAML manifests. A basic PDB for a Deployment with three replicas, requiring at least two pods to be available, looks like: apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: webapp-pdb spec: minAvailable: 2 selector: matchLabels: app: webapp Save this as webapp-pdb.yaml . Before applying, always review the live state: kubectl get deployment webapp -o wide kubectl get pods -l app=webapp This confirms the Deployment exists and pods are running. Then apply the PDB: kubectl apply -f webapp-pdb.yaml Verify the PDB was created correctly: kubectl get pdb webapp-pdb -o yaml Check the status section. It should show currentHealthy and desiredHealthy . For example: status: currentHealthy: 3 desiredHealthy: 2 disruptionsAllowed: 1 expectedPods: 3 currentHealthy must be greater than or equal to desiredHealthy for disruptions to be allowed. If disruptionsAllowed is 0, the PDB is blocking all voluntary disruptions; investigate immediately. In a versioned environment, store this manifest in Git and use CI/CD to apply changes. For example, in a GitHub Actions workflow, you might have a job that runs kubectl apply on every push to the main branch. Always use a dedicated service account with least privilege (only PDB create/update permissions) and store cluster credentials in secrets. Example CI/CD snippet (conceptual): - name: Apply PDB run: | kubectl apply -f deploy/webapp-pdb.yaml kubectl rollout status deployment/webapp --timeout=60s Do not hardcode sensitive values. Use environment variables or secret references. ## Verification and Diagnostics After applying a PDB, verify it is functioning as expected. Simulate a voluntary disruption using kubectl drain on a node (in a test cluster only): kubectl drain --ignore-daemonsets --delete-emptydir-data Watch the eviction process: kubectl get pods -l app=webapp -w The pods should be evicted one by one, but at least minAvailable pods remain running. If the drain command hangs, the PDB is blocking evictions because it cannot guarantee availability. Check PDB status: kubectl describe pdb webapp-pdb Look for events like Evicting pod webapp-xxxxx or Cannot evict pod as it would violate the pod's disruption budget. The latter indicates a configuration mismatch; for example, minAvailable might be set too high or selector matches no pods. Common diagnostic commands: - kubectl get pdb - list all PDBs and their allowed disruptions - kubectl describe pdb - detailed status and events - kubectl get events --field-selector involvedObject.name=- recent PDB-related events - kubectl logs --previous - check application logs if pods are crash-looping Use kubectl rollout status deployment/webapp to confirm the deployment is healthy after any changes. If the rollout is stalled, inspect with kubectl rollout history deployment/webapp and kubectl rollout undo deployment/webapp if needed. For local verification, use kubectl port-forward to expose the service and test traffic: kubectl port-forward svc/webapp 8080:80 Then curl http://localhost:8080 from another terminal. Expected response is your application's health endpoint. ## Failure Modes and Recovery PDB automation can fail in several ways. Let's examine common scenarios and how to recover. ### Scenario 1: PDB blocks all evictions due to incorrect selector Symptom: kubectl drain hangs, kubectl describe pdb shows disruptionsAllowed: 0 even though pods are healthy. Diagnosis: Check kubectl get pdb -o yaml and verify the selector matches actual pod labels. Use kubectl get pods --show-labels . Recovery: Fix the selector in the manifest, apply it, and retry. If immediate unblock is needed (e.g., critical maintenance), temporarily delete the PDB: kubectl delete pdb webapp-pdb Then re-apply corrected PDB after maintenance. ### Scenario 2: PDB missing during cluster upgrade Symptom: During a node upgrade, all pods of a service get evicted simultaneously, causing outage. Diagnosis: After the fact, kubectl get pdb shows no PDB for the workload. Review Git history to see if PDB manifest was accidentally removed. Recovery: Immediately re-apply the PDB from version control. Then scale up the deployment and verify health. Prevent recurrence by adding a CI check: if a namespace has workloads, ensure a PDB exists. ### Scenario 3: PDB minAvailable set higher than replicas Symptom: kubectl apply succeeds but kubectl describe pdb shows currentHealthy less than desiredHealthy , and disruptionsAllowed remains 0. Also, deployments cannot scale down. Diagnosis: Compare minAvailable in PDB spec with actual replicas in deployment. Recovery: Either increase replicas to meet minAvailable or lower minAvailable to a sensible value (e.g., replicas - 1 ). Apply the change and verify currentHealthy >= desiredHealthy . ### Rollback Strategies Since PDBs are Kubernetes objects, rollback is straightforward: - Identify the last known good PDB manifest from Git history. - Apply it using kubectl apply -f .yaml . - Verify status. For automation, consider storing PDB manifests with version tags in a Git repository. In CI/CD, use a deployment strategy that can revert to a previous tag on failure. For example, in a GitOps setup with Argo CD or Flux, PDB changes are part of the application configuration; rolling back to a previous commit in the Git repo reverts the PDB automatically. If a PDB was mistakenly deleted, recreate it immediately from source control. Do not rely on manual reconstruction.
Quick check 2 of 2 How is the intended number of pods for a PDB computed? - From the .spec.replicas of the workload resource - From the number of nodes in the cluster - From the .spec.minAvailable field in the PDB - From the .metadata.name of the PDB
The reference states that the intended number of pods is computed from the .spec.replicas of the workload resource that is managing those pods.
## Integration with CI/CD Pipelines Once you have tested PDB changes manually, integrate them into your pipeline. The goal is to ensure that any change to PDB manifests goes through automated validation before hitting production. ### Pipeline Steps - Lint and validate: Use kubectl apply --dry-run=client or server to validate manifests without applying. For example: kubectl apply -f webapp-pdb.yaml --dry-run=client Expected output: poddisruptionbudget.policy/webapp-pdb created (dry run) - Static analysis: Use tools like kubeval or kubeconform to validate YAML structure against Kubernetes schemas. Run these in CI. - Apply to staging: Deploy PDB to staging cluster and run integration tests that simulate disruptions. - Verify PDB status: After applying, run a script that checks: kubectl get pdb webapp-pdb -o jsonpath='{.status.disruptionsAllowed}' Ensure it returns a non-zero number (if disruptions should be allowed). - Approve and apply to production: Use a manual approval gate or progressive delivery. ### Example Pipeline (GitHub Actions) Here is a condensed workflow for applying PDB changes to a Kubernetes cluster using GitHub Actions. It uses azure/k8s-set-context to set cluster context and kubectl to apply manifests. name: PDB Deployment on: push: branches: [ main ] jobs: deploy: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - uses: azure/k8s-set-context@v3 with: kubeconfig: ${{ secrets.KUBE_CONFIG }} - name: Validate PDB manifest run: | kubectl apply -f deploy/webapp-pdb.yaml --dry-run=client - name: Apply PDB run: | kubectl apply -f deploy/webapp-pdb.yaml - name: Wait for PDB to be active run: | kubectl wait --for=condition=ready pdb/webapp-pdb --timeout=60s - name: Verify disruptions allowed run: | ALLOWED=$(kubectl get pdb webapp-pdb -o jsonpath='{.status.disruptionsAllowed}') if [ "$ALLOWED" -lt 1 ]; then echo "PDB not allowing disruptions"; exit 1; fi ### Testing in CI/CD Simulate a voluntary disruption in your staging environment after applying the PDB. You can use kubectl drain on a staging node (or a dedicated minikube) and verify pods remain available. Automate this in a separate test job if possible. For instance, in a kind cluster, you can run: kubectl drain kind-worker --ignore-daemonsets --delete-emptydir-data Then check the application's availability using a simple curl or a probe. ## Operations Checklist Use this checklist before and after any PDB change to ensure operational safety.
StepCommand/ActionExpected OutcomeNotes
1. Inventory current PDBskubectl get pdb --all-namespacesList of existing PDBs with statusCapture baseline
2. Check cluster versionkubectl version --shortServer version >= 1.21PDB stable since 1.21
3. Verify workload labelskubectl get pods -l app=webapp --show-labelsPods have expected labelsEnsure selector matches
4. Create/Update PDB manifestUse YAML file with minAvailable or maxUnavailableManifest validated with kubectl apply --dry-runStore in Git
5. Apply PDBkubectl apply -f webapp-pdb.yamlPDB created/updatedUse CI/CD for automation
6. Check PDB statuskubectl describe pdb webapp-pdbdisruptionsAllowed > 0If 0, investigate
7. Test disruption (staging)kubectl drain <node>Pods evicted but minAvailable remainsOnly in test environment
8. Validate application healthkubectl rollout status deployment/webappDeployment rolled out successfullyUse port-forward for local test
9. Document recoveryStore rollback commands in runbookEnsure team knows how to revertPDB manifests in Git
10. MonitorSet up alerts on PDB statusAlert if disruptionsAllowed becomes 0 unexpectedlyUse Prometheus metrics if available
Keep this checklist accessible in your team's runbook and update it as procedures evolve. ## Conclusion Automating Kubernetes Pod Disruption Budgets in CI/CD is not just about applying YAML files; it requires a disciplined approach to observation, safe configuration, verification, and recovery. By inventorying your environment, using version-controlled manifests, integrating validation into pipelines, and preparing for failure modes, you ensure that your applications maintain availability during voluntary disruptions without manual intervention. Start small: implement one PDB for a critical workload, test it thoroughly in staging, and integrate it into your CI/CD pipeline. As you gain confidence, expand to other services. Remember to keep blast radius small, use least privilege, and always have a rollback plan. The operational safety you gain far outweighs the setup effort. Next steps: choose a low-risk workload, create a PDB with appropriate minAvailable or maxUnavailable , apply it via your pipeline, and simulate a drain to verify behavior. Document your findings and share with your team to build a culture of reliability.