## Intro Pod Disruption Budgets (PDBs) are a critical Kubernetes safeguard: they limit the number of pods that can be down simultaneously during voluntary disruptions such as node drains, cluster autoscaling, or rolling updates. When configured correctly, PDBs ensure your application remains available during maintenance. When misconfigured, however, they can block node drains, prevent deployments from progressing, or fail silently—leaving your workloads vulnerable. This article targets developers, DevOps engineers, and platform teams who operate Kubernetes clusters in production. It walks through the most common PDB errors, how to debug them with `kubectl`, and practical fixes validated against Kubernetes 1.28 to 1.30. You will learn: - How PDBs interact with evictions and voluntary disruptions - Which fields are mandatory and which are mutually exclusive - How to interpret `kubectl get pdb` output and Events - How to recover from a PDB that blocks maintenance - How to test PDB behavior safely with a local cluster The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached. ## Version and Environment Inventory Before debugging a PDB, confirm your cluster version and the PDB API version in use. PDBs have been stable since `policy/v1` in Kubernetes 1.21, but older clusters may still use `policy/v1beta1`, which is deprecated and removed in 1.25. Check your API resources: ```bash kubectl api-resources | grep poddisruptionbudget ``` Expected output on a modern cluster: ``` poddisruptionbudgets pdb policy/v1 true PodDisruptionBudget ``` If you see `policy/v1beta1`, plan an upgrade. The semantics are identical for the fields discussed here, but cluster administrators should migrate manifests to `policy/v1` to avoid future breakage. Prerequisites for this guide: - `kubectl` configured with access to a test namespace - A running cluster (kind, minikube, or cloud-based) with Kubernetes 1.25+ - Basic understanding of Deployments, ReplicaSets, and node operations Start with read-only observation. Never modify a PDB or its workloads until you understand the current state. Capture the following into a notes file: ```bash kubectl get pdb -n -o wide kubectl get pods -n -o wide kubectl get events -n --sort-by=.lastTimestamp | grep -i pdb ``` For example, a healthy PDB tied to a Deployment with 3 replicas might show: ``` NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE my-app-pdb 1 N/A 2 5m ``` If the Deployment has already lost pods due to node failure, `ALLOWED DISRUPTIONS` may drop to 0 or even show a negative number? No—PDB status never goes negative. If the current healthy pods are below the budget, `ALLOWED DISRUPTIONS` becomes 0, and further voluntary evictions are blocked. **Practical check:** On a local cluster, create a simple Deployment and PDB, then drain a node to observe the PDB in action. Use `kind` or `minikube`: ```bash kubectl create deployment web --image=nginx --replicas=3 kubectl apply -f - < --ignore-daemonsets --delete-emptydir-data ``` Watch the drain command—it will evict only one pod at a time, waiting for replacement pods to become ready before evicting the next. Keep the local test small: apply one manifest, inspect generated resources, and verify traffic before moving to a cloud load balancer or ingress controller. ## Safe Configuration Path PDB configuration errors often stem from a misunderstanding of the two mutually exclusive fields: `minAvailable` and `maxUnavailable`. You must specify exactly one of them. If you set both, the API server rejects the manifest with: ``` The PodDisruptionBudget "web-pdb" is invalid: spec: Invalid value: core.PodDisruptionBudgetSpec{...}: minAvailable and maxUnavailable cannot be both set ``` If you omit both, you'll see: ``` The PodDisruptionBudget "web-pdb" is invalid: spec: Invalid value: core.PodDisruptionBudgetSpec{...}: minAvailable or maxUnavailable must be specified ``` **Which one to use?** - `minAvailable` is an absolute number or percentage of pods that must remain available. Example: `minAvailable: 2` means at least 2 pods must be healthy and ready at all times. It directly expresses availability, making it easier to reason about service capacity. - `maxUnavailable` is the maximum number of pods that can be unavailable. Example: `maxUnavailable: 1` allows at most 1 pod to be down. It's convenient when you think in terms of disruption impact. For most stateless workloads, `minAvailable: 1` is a safe default if you have at least 2 replicas. For stateful applications with quorum requirements (e.g., etcd, databases), set `minAvailable` to the quorum size. Here is a complete, production-ready PDB manifest with sensible settings: ```yaml apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: api-pdb namespace: production spec: minAvailable: 50% selector: matchLabels: app: api tier: backend ``` Suppose the Deployment has 6 pods. A `minAvailable: 50%` means 3 pods must always be ready. The PDB allows at most 3 voluntary disruptions at any time (assuming all 6 are currently healthy). If one pod crashes due to an application bug, making only 5 healthy, then the allowed disruptions drop to 2. Node drains will only evict 2 pods and then block until the crashed pod recovers or is replaced. **The selector must match the pods.** If the PDB selector doesn't match any pods, the PDB status shows `ALLOWED DISRUPTIONS: 0` even though pods exist. For example, a common typo: ```yaml selector: matchLabels: app: web # Deployment labels pods with app: frontend ``` The PDB will have no effect, and you might not notice until a maintenance window fails. Always verify: ```bash kubectl get pdb web-pdb -o yaml ``` Look at `status.currentHealthy` and `status.desiredHealthy`. If `currentHealthy` is 0 but pods exist, your selector is wrong. **Another pitfall:** PDBs do not apply to bare pods or pods not controlled by a higher-level controller like Deployment or StatefulSet. They also do not apply to daemonset pods. Use Deployments or StatefulSets for pods you want protected. **Migration from v1beta1 to v1:** If your cluster still uses `policy/v1beta1`, change the `apiVersion` and ensure `spec` fields are the same. The only difference is the API group version. There is no field deprecation within PDB. Test in a staging environment first: ```bash kubectl apply --dry-run=client -f pdb.yaml kubectl apply -f pdb.yaml kubectl get pdb ``` ## Verification and Diagnostics Once a PDB is applied, you need to verify it is working as intended and debug any issues. The primary command is: ```bash kubectl get pdb -n ``` Output columns: - `NAME`: PDB name - `MIN AVAILABLE`: The `minAvailable` value - `MAX UNAVAILABLE`: The `maxUnavailable` value - `ALLOWED DISRUPTIONS`: How many more pods can be voluntarily disrupted right now - `AGE`: Age of the PDB For example: ``` NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE zookeeper 2 N/A 1 3d ``` This means 2 pods must remain available. Among the current healthy pods, one more can be evicted without violating the budget. To see more detail, including selector and status: ```bash kubectl describe pdb zookeeper -n ``` Look at the Events section for recent eviction attempts. You may see messages like: ``` Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning EvictionBlocked 10m disruption-controller Cannot evict pod as it would violate the pod's disruption budget. ``` If you see `EvictionBlocked`, that's not necessarily an error—it means the PDB is doing its job. However, if you expected an eviction to succeed and it didn't, you need to investigate why the PDB thinks the budget would be violated. Check the current healthy count: ```bash kubectl get pods -l app=zookeeper -o wide ``` If a pod is `CrashLoopBackOff` or `Pending`, it is not counted as healthy. Therefore, the PDB's `ALLOWED DISRUPTIONS` may be 0, and no further evictions can happen. In that case, fix the unhealthy pod first. **Debugging with `kubectl drain`:** When a node drain is stuck, you may see: ``` error when evicting pods/ -n (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. ``` To resolve, you have a few options: 1. **Wait and scale up:** If the application can handle more replicas, increase the Deployment replicas so that after eviction, the budget is still satisfied. ```bash kubectl scale deployment zookeeper --replicas=4 ``` Wait for the new pod to become ready, then retry the drain. 2. **Temporarily relax the PDB:** If you absolutely need to evict more pods than allowed (e.g., emergency maintenance), you can patch the PDB to a higher `maxUnavailable` or lower `minAvailable`. For example, set `minAvailable: 1` temporarily: ```bash kubectl patch pdb zookeeper -p '{"spec":{"minAvailable":1}}' ``` After maintenance, revert to the original value. **This should be a last resort and require a change approval.** 3. **Delete the PDB:** If the PDB is no longer needed or is misconfigured beyond repair, delete it: ```bash kubectl delete pdb zookeeper ``` This removes all protection. Recreate it with the correct configuration afterward. **Simulating a disruption locally:** To test PDB behavior without affecting production, use a tool like `kubectl drain` on a test node, or use the `evict` API directly via a small script. Here is a Python snippet using `kubernetes` client: ```python from kubernetes import client, config config.load_kube_config() v1 = client.PolicyV1Api() body = client.V1Eviction(metadata=client.V1ObjectMeta(name="my-pod", namespace="default")) try: v1.create_namespaced_pod_eviction("my-pod", "default", body) print("Eviction succeeded") except client.exceptions.ApiException as e: print(f"Eviction failed: {e}") ``` If the PDB blocks the eviction, you'll get a 429 Too Many Requests error from the API server. This confirms the PDB is working. **Common diagnostic mistake:** Assuming the PDB blocks evictions from the Eviction API but not from `kubectl delete pod`. In fact, `kubectl delete pod` is a direct deletion and bypasses the PDB. PDBs only apply to voluntary disruptions initiated through the Eviction API, which is used by `kubectl drain`, cluster autoscaler, and some operators. If you delete a pod manually, the PDB will not stop it. This is by design: the PDB protects against involuntary failures and maintenance, not administrative deletions. To enforce availability even on manual deletion, you need additional policies like Kyverno or OPA Gatekeeper. ## Failure Modes and Recovery Let's examine three common failure modes and how to recover from them. ### Failure Mode 1: PDB with zero allowed disruptions blocks all maintenance **Symptom:** `kubectl get pdb` shows `ALLOWED DISRUPTIONS: 0`, and node drains are stuck. `kubectl describe pdb` shows `EvictionBlocked` events for every pod. **Cause:** The PDB's `minAvailable` or `maxUnavailable` is set too strictly for the current replica count. For example, a Deployment with 3 replicas and a PDB `minAvailable: 3` means all 3 must be up at all times. If one pod fails, allowed disruptions become 0, and no evictions can proceed. This is often a misconfiguration: you intended `minAvailable: 2` but set 3. **Recovery:** 1. Identify the misconfigured PDB: ```bash kubectl get pdb -A ``` 2. Check the Deployment replica count and the PDB spec: ```bash kubectl get deploy -o yaml kubectl get pdb -o yaml ``` 3. Correct the PDB to a reasonable value. If you have 3 replicas, `minAvailable: 2` is typical: ```bash kubectl patch pdb --type merge -p '{"spec":{"minAvailable":2}}' ``` 4. Verify allowed disruptions is now > 0: ```bash kubectl get pdb ``` ### Failure Mode 2: PDB selector doesn't match any pods **Symptom:** PDB exists, but `ALLOWED DISRUPTIONS` is 0 even though pods are running. No eviction blocking events appear. **Cause:** The `selector` labels do not match the pod labels. For example, the Deployment labels pods with `app: frontend`, but the PDB selector looks for `app: web`. **Recovery:** 1. Compare pod labels and PDB selector: ```bash kubectl get pods --show-labels kubectl get pdb -o jsonpath='{.spec.selector}' ``` 2. Update the PDB selector to match. This often requires editing the PDB because `selector` is immutable after creation. Delete and recreate: ```bash kubectl delete pdb kubectl apply -f corrected-pdb.yaml ``` 3. Confirm `currentHealthy` in PDB status is now > 0: ```bash kubectl get pdb -o yaml ``` ### Failure Mode 3: PDB blocks rolling update or node drain during outage **Symptom:** During an incident, you need to drain a node immediately, but the PDB prevents eviction, causing delays. You might see errors like: ``` error when evicting pods/... : Cannot evict pod as it would violate the pod's disruption budget. ``` **Cause:** The PDB is fulfilling its purpose, but operational urgency requires temporary override. **Recovery (with caution):** 1. Document the incident and obtain approval to relax the PDB. 2. Temporarily adjust the PDB to allow more disruptions. For example, set `minAvailable: 0` (effectively disable protection) or delete the PDB. ```bash kubectl patch pdb --type merge -p '{"spec":{"minAvailable":0}}' # or kubectl delete pdb ``` 3. Perform the drain/maintenance. 4. Restore the PDB to its original configuration immediately. 5. Review why the urgency occurred and whether the PDB setting was too restrictive for real-world operations. **Prevention:** Set `minAvailable` appropriately for your availability targets. Test drain procedures in staging. Use `maxUnavailable` instead of `minAvailable` if you prefer to think in terms of how many pods can be down at once. For a 3-replica service, `maxUnavailable: 1` and `minAvailable: 2` are equivalent. ## Operations Checklist Use this checklist to ensure your PDBs are correctly deployed and maintained. ### PDB Pre-Deployment Checklist - [ ] Confirm cluster version supports `policy/v1` (Kubernetes 1.21+). - [ ] Identify the workload (Deployment, StatefulSet, Operator) and its labels. - [ ] Decide between `minAvailable` and `maxUnavailable` based on availability SLO. - [ ] Write the PDB manifest with a selector that matches the workload's pod template labels. - [ ] Set a reasonable value: for N replicas, `minAvailable: N-1` or `maxUnavailable: 1` for stateless apps; for stateful apps, use quorum size. - [ ] Apply with dry-run first: ```bash kubectl apply -f pdb.yaml --dry-run=client ``` - [ ] Apply the PDB and verify status: ```bash kubectl apply -f pdb.yaml kubectl get pdb ``` - [ ] Check that `ALLOWED DISRUPTIONS` is greater than 0 when all pods are healthy. ### Periodic PDB Audit - [ ] List all PDBs in the cluster: ```bash kubectl get pdb -A ``` - [ ] For each PDB, verify selector matches intended pods: ```bash kubectl get pods -l -n ``` - [ ] Confirm `currentHealthy` in PDB status equals the number of desired ready pods (or is within acceptable range). - [ ] Review if any PDB blocks maintenance: check `ALLOWED DISRUPTIONS` and node drain logs. - [ ] Remove obsolete PDBs that no longer have matching workloads. ### Troubleshooting Commands Cheat Sheet | Task | Command | |------|---------| | List PDBs with status | `kubectl get pdb -n ` | | Describe a PDB and events | `kubectl describe pdb -n ` | | View PDB YAML | `kubectl get pdb -n -o yaml` | | Check pods matching selector | `kubectl get pods -l = -n ` | | Simulate eviction (Python) | Use Eviction API snippet from above | | Temporarily disable PDB | `kubectl patch pdb --type merge -p '{"spec":{"minAvailable":0}}'` | | Delete PDB | `kubectl delete pdb -n ` | | Watch drain events | `kubectl drain --ignore-daemonsets --delete-emptydir-data` and observe | ## Conclusion Pod Disruption Budgets are a small but powerful mechanism to keep your applications available during voluntary cluster operations. The most common errors—misconfigured selectors, impossible availability targets, and using the wrong API version—can be avoided with careful manifest review and regular audits. Remember these key practices: - Always specify either `minAvailable` or `maxUnavailable`, not both. - Ensure the PDB selector exactly matches the labels of pods controlled by your Deployment or StatefulSet. - Check `kubectl get pdb` regularly; a sudden drop in `ALLOWED DISRUPTIONS` to 0 indicates an underlying issue. - Treat PDB relaxation as a last resort during incidents; document and revert promptly. - Test PDB behavior in a staging cluster using `kubectl drain` or the Eviction API. By following the verification and recovery steps in this guide, you can maintain a resilient Kubernetes environment where maintenance windows proceed smoothly and application uptime remains within SLO. As a next step, choose one low-risk verification from the Operations Checklist for a PDB in your cluster, record the current state, run the documented check, and compare the result with the expected signal. Then review dependencies such as Deployment replica counts and node health.