## Intro Kubernetes Pod Disruption Budgets (PDBs) are a critical safeguard for maintaining application availability during voluntary disruptions like node drains, cluster upgrades, or maintenance. However, misconfigured PDBs can block operations, cause unexpected downtime, or provide a false sense of security. This article provides a production-focused operations checklist with practical examples, commands, and failure signals. It is designed for developers, DevOps consultants, and technical startup teams who need to move from observing a problem to verifying a solution. We will cover five key operational areas: version and environment inventory, safe configuration path, verification and diagnostics, failure modes and recovery, and an operations checklist. For each, we provide concrete commands, expected outputs, and recovery decisions. The goal is operational safety: observe before changing, limit blast radius, use placeholders instead of secrets, verify results, and document recovery paths. ## Version and Environment Inventory Before touching any PDB, understand your environment. This section describes how to inventory your Kubernetes version, cluster topology, and existing PDBs to avoid compatibility surprises. ### Check Kubernetes Version Use `kubectl version` to see client and server versions: ```bash kubectl version --short ``` Expected output (example): ``` Client Version: v1.28.2 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.27.3 ``` PDB API version changed: `policy/v1beta1` was deprecated in Kubernetes 1.21 and removed in 1.25. For clusters 1.25+, use `policy/v1`. Verify with: ```bash kubectl api-versions | grep policy ``` Expected output includes `policy/v1`. ### Inspect Existing PDBs List PDBs in all namespaces: ```bash kubectl get pdb --all-namespaces ``` Example output: ``` NAMESPACE NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE default my-app-pdb N/A 1 1 45d kube-system coredns-pdb 1 N/A 0 120d ``` This shows the current state. Note the `ALLOWED DISRUPTIONS` column: it tells you how many pods can be voluntarily evicted simultaneously without violating the PDB. ### Verify Pod Labels Match Selectors PDBs select pods via labels. Use `kubectl describe pdb -n ` to see the selector and status: ```bash kubectl describe pdb my-app-pdb -n default ``` Excerpt: ``` Selector: app=my-app Status: Current Healthy: 3 Desired Healthy: 2 Disruptions Allowed: 1 ``` Ensure that all relevant pods have matching labels. Check with: ```bash kubectl get pods -l app=my-app -n default ``` If pods are missing labels, the PDB may not protect them. ### Prerequisites Check - Ensure you have `kubectl` access and appropriate RBAC permissions: `get`, `list`, `watch` on `poddisruptionbudgets`. - Confirm the workload controller (Deployment, StatefulSet, etc.) is present and healthy. For example, `kubectl rollout status deployment/my-app` should show `successfully rolled out`. ## Safe Configuration Path This section details how to safely create or modify a PDB, avoiding common pitfalls like invalid selectors or impossible budgets. ### Understanding PDB Parameters A PDB can specify either `minAvailable` or `maxUnavailable`, but not both. `minAvailable` is the minimum number of pods that must remain available after evictions; `maxUnavailable` is the maximum number of pods that can be unavailable. Use one or the other, expressed as an integer or percentage (e.g., 2 or 30%). Example PDB YAML using `minAvailable`: ```yaml apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: my-app-pdb spec: minAvailable: 2 selector: matchLabels: app: my-app ``` Example using `maxUnavailable`: ```yaml apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: my-app-pdb spec: maxUnavailable: 1 selector: matchLabels: app: my-app ``` Choose parameters based on your application's quorum or availability requirements. For a stateless app with 3 replicas, `maxUnavailable: 1` is common. For a stateful app requiring quorum, `minAvailable: 2` may be appropriate. ### Dry-Run Before Applying Always do a dry-run to validate the manifest: ```bash kubectl apply -f pdb.yaml --dry-run=client ``` Expected output: `poddisruptionbudget.policy/my-app-pdb created (dry run)` Check that the API version is accepted. If you get an error like `no matches for kind "PodDisruptionBudget" in version "policy/v1beta1"`, update to `policy/v1`. ### Apply and Verify Status Apply the PDB: ```bash kubectl apply -f pdb.yaml ``` Verify the PDB is active and check its current status: ```bash kubectl get pdb my-app-pdb -o yaml ``` Look for `status` fields: ```yaml status: currentHealthy: 3 desiredHealthy: 2 disruptionsAllowed: 1 expectedPods: 3 observedGeneration: 1 ``` If `disruptionsAllowed` is 0, the PDB is currently preventing all voluntary evictions, which could block maintenance. ### Test with a Simulated Drain To safely test the PDB's effect without affecting production, use `kubectl drain` with `--dry-run`: ```bash kubectl drain node-1 --dry-run=server ``` This simulates the eviction process and reports which pods would be blocked by PDBs. Example output: ``` node/node-1 cordoned evicting pod default/my-app-1 evicting pod default/my-app-2 error when evicting pod default/my-app-3: Cannot evict pod as it would violate the pod's disruption budget. ``` This confirms the PDB is working as intended. ## Verification and Diagnostics After configuring a PDB, you need to continuously verify its behavior and diagnose issues when disruptions are blocked or pods are unexpectedly unavailable. ### Routine Status Checks Periodically run: ```bash kubectl get pdb --all-namespaces ``` Watch for PDBs with `ALLOWED DISRUPTIONS` equal to 0 for extended periods, which may impede node maintenance. Also check that `CURRENT HEALTHY` matches expectations; if it's lower, pods may be failing or labels incorrect. ### Inspect Events for PDB Blocking When a drain or upgrade hangs, check events in the namespace: ```bash kubectl get events -n default --sort-by='.lastTimestamp' | grep -i pdb ``` Example output: ``` Warning EvictionBlocked Pod/my-app-3 Cannot evict pod as it would violate the pod's disruption budget. ``` This indicates the PDB is doing its job, but you may need to temporarily adjust it or address the application's health. ### Verify Pod Health and Readiness PDB relies on the readiness state to count healthy pods. If pods are not ready, they are not counted as healthy and may reduce `disruptionsAllowed`. Check readiness with: ```bash kubectl get pods -l app=my-app -o wide ``` Look for `READY 1/1` and `STATUS Running`. For deeper diagnosis, use `kubectl describe pod ` and check `Conditions` and `Events`. ### Check Controller Status If pods are not being created or are stuck, verify the owning Deployment or StatefulSet: ```bash kubectl rollout status deployment/my-app ``` Expected output: `deployment "my-app" successfully rolled out` If not, investigate with `kubectl describe deployment my-app` and pod logs: ```bash kubectl logs my-app- --previous ``` ## Failure Modes and Recovery Understanding common failure modes helps you act quickly. This section covers typical issues and recovery steps. ### Failure Mode: PDB Blocks All Drains Due to 0 Allowed Disruptions Symptom: `kubectl drain` fails with eviction errors for all pods in the PDB. Diagnosis: ```bash kubectl get pdb my-app-pdb -o jsonpath='{.status.disruptionsAllowed}' ``` If output is `0`, the PDB is too strict. Recovery options: - Temporarily increase `maxUnavailable` or decrease `minAvailable` (with change control). - Scale up replicas temporarily to provide more room for eviction. - If the workload is stateless and can tolerate downtime, delete the PDB with `kubectl delete pdb my-app-pdb` (ensure you recreate it later). Always communicate changes and revert after maintenance. ### Failure Mode: PDB Not Protecting Pods Due to Label Mismatch Symptom: During node drain, pods are evicted even though PDB exists. Diagnosis: Compare PDB selector with pod labels: ```bash kubectl get pdb my-app-pdb -o jsonpath='{.spec.selector.matchLabels}' kubectl get pods -l app=my-app --show-labels ``` If labels don't match, PDB has no effect. Recovery: Correct the selector or pod labels, apply, and verify with `kubectl get pdb -o yaml` that `expectedPods` equals the number of pods intended. ### Failure Mode: PDB With Percentage Causes Unexpected Blocking Symptom: `disruptionsAllowed` is calculated as a percentage, and rounding leads to 0 when replicas are low. Example: `minAvailable: 50%` with 3 replicas gives `desiredHealthy=2`, allowing 1 disruption. But with 1 replica (e.g., during scale down), `desiredHealthy=1`, so 0 disruptions allowed, blocking even a single maintenance eviction. Diagnosis: Check `kubectl get pdb` and `currentHealthy` vs `desiredHealthy`. Recovery: Use integer values for small replica counts, or ensure minimum replicas such that percentage yields a safe allowance. For example, with 3 replicas, use `minAvailable: 1` (absolute) to guarantee at least one disruption allowed. ### Failure Mode: PDB Prevents Node Upgrade Because Pods Stuck Terminating Symptom: Drain gets stuck with pods in `Terminating` state, and no eviction errors. Diagnosis: Check pod status: ```bash kubectl get pods -o wide ``` If pods are stuck, they may have finalizers or preStop hooks hanging. Recovery: Investigate pod details; force delete if necessary: ```bash kubectl delete pod --grace-period=0 --force ``` Then retry drain. Ensure PDB is not inadvertently blocking due to unhealthy pods. ## Operations Checklist Use this condensed checklist for regular PDB operations. Each item includes the command and expected result. | # | Operation | Command | Expected Result | |---|-----------|---------|-----------------| | 1 | Check Kubernetes version | `kubectl version --short` | Server >=1.21 for policy/v1beta1, >=1.25 for policy/v1 | | 2 | List all PDBs | `kubectl get pdb --all-namespaces` | PDBs present with correct allowed disruptions | | 3 | Verify PDB API version | `kubectl api-versions | grep policy` | Contains `policy/v1` (for modern clusters) | | 4 | Inspect PDB details | `kubectl describe pdb -n ` | Selector matches pods, status healthy | | 5 | Confirm pod labels match PDB selector | `kubectl get pods -l --show-labels` | All intended pods listed | | 6 | Dry-run PDB changes | `kubectl apply -f pdb.yaml --dry-run=client` | No errors, manifest accepted | | 7 | Apply PDB | `kubectl apply -f pdb.yaml` | PDB created/configured | | 8 | Check PDB status after apply | `kubectl get pdb -o yaml` | `disruptionsAllowed` > 0 if desired | | 9 | Simulate node drain | `kubectl drain --dry-run=server` | Eviction blocked for pods as expected | | 10 | Monitor events for PDB blocks | `kubectl get events -n | grep -i pdb` | EvictionBlocked warnings as expected during maintenance | | 11 | Verify pod readiness | `kubectl get pods -l ` | All pods Ready | | 12 | Check controller status | `kubectl rollout status deployment/` | Successfully rolled out | | 13 | Validate PDB after incident | `kubectl get pdb -o yaml` and compare with backup | Configuration matches expected | | 14 | Document recovery steps | Update runbook | Clear rollback plan in place | Regularly review PDBs against application requirements and cluster changes. Automate checks where possible using scripts or CI/CD. ## Conclusion A Kubernetes Pod Disruption Budget production operations checklist is only useful when each recommendation is version-scoped, observable, and reversible where technology permits. Copying a command without checking prerequisites and expected output is not an operations procedure. This article provided concrete commands, configuration examples, and failure scenarios to help you manage PDBs safely. As a next step, choose one low-risk verification from the checklist, such as a dry-run drain or PDB status check. Record the current state, run the documented check, compare with expected signals, and review dependencies like Deployments, StatefulSets, and Nodes. Keep failure visible, protect sensitive values, limit changes to intended resources, and define recovery verification before an incident forces the decision. By following this checklist, you can ensure that Pod Disruption Budgets remain a reliable tool for maintaining availability during voluntary disruptions without becoming an operational bottleneck.