Intro
Kubernetes clusters are dynamic environments where pods are routinely evicted or terminated for many reasons, including node maintenance, cluster autoscaling, and resource pressure. While these disruptions are often necessary, they can hurt application availability if critical workloads lose too many replicas at the same time. A Pod Disruption Budget (PDB) is a Kubernetes policy object that limits the number of concurrent disruptions a workload can experience, ensuring a minimum number of pods remain available during voluntary disruptions.
This guide provides a practical deep dive into PDB architecture. You will learn how PDBs work under the hood, how to configure them safely, how to verify their behavior with concrete commands and expected outputs, how to diagnose and recover from common failure modes, and how to integrate PDBs into your operational workflows. Whether you are preparing for a node drain, cluster upgrade, or simply hardening your deployment, this guide gives you the tools to use PDBs effectively.
What Is a Pod Disruption Budget?
A Pod Disruption Budget is a Kubernetes API object that specifies the minimum number or percentage of pods of a given application that must remain available during voluntary disruptions. Voluntary disruptions are deliberate actions initiated by cluster administrators or automated controllers, such as:
- Draining a node for maintenance (kubectl drain)
- Upgrading a cluster node
- Removing a node from the cluster
- Autoscaling down a node pool
- Deleting a pod directly (though this is not always considered voluntary)
PDBs do not prevent involuntary disruptions like hardware failures, kernel panics, or node crashes. They only apply to voluntary disruptions that go through the Eviction API, which respects PDBs.
The PDB is not a controller; it does not actively manage pods. Instead, it is a policy object that the Kubernetes control plane consults when an eviction request is made. If evicting a pod would cause the number of available pods to fall below the budget, the eviction is rejected.
Version and Environment Inventory
Before implementing PDBs, establish a clear inventory of your Kubernetes environment and application topology.
Kubernetes Version
PDBs have been available since Kubernetes v1.5 as beta and became stable in v1.21. The PDB API version is policy/v1 for clusters v1.21 and later. For older clusters, use policy/v1beta1. Verify your cluster version:
kubectl version --short
Expected output (example):
Client Version: v1.27.3
Server Version: v1.27.3
If your cluster is older than v1.21, you may need to use the beta API version and adjust your manifests accordingly. For this guide, we assume a v1.21+ cluster with policy/v1.
Application Topology
Identify the workloads that require PDBs. Typically, these are Deployments, StatefulSets, or ReplicaSets running stateless or stateful applications. For each workload, note the desired replica count and the minimum number of pods that must remain available during disruptions. The following table illustrates an example inventory:
| Workload Name | Kind | Replicas | Minimum Available | Owner |
|---|---|---|---|---|
| web-frontend | Deployment | 5 | 3 | Priya Shah, Platform Lead |
| db-stateful | StatefulSet | 3 | 2 | Marcus Chen, DB Admin |
| cache | Deployment | 4 | 2 | Elena Rodriguez, SRE |
Assigning an owner for each PDB ensures accountability for its configuration and periodic review. We recommend reviewing PDBs quarterly or whenever replica counts change.
Prerequisites
- A running Kubernetes cluster (v1.21+ recommended).
- kubectl configured to access the cluster.
- Appropriate RBAC permissions to create PDBs (usually create and update on the
poddisruptionbudgetsresource). - Understanding of your application's availability requirements.
PDB Architecture and Core Components
To use PDBs effectively, it helps to understand the underlying architecture and the components involved.
The PDB Object
A PDB is defined by three main fields:
spec.minAvailable: The minimum number or percentage of pods that must remain available after evictions. Can be an integer or a percentage string (e.g., "60%").spec.maxUnavailable: The maximum number or percentage of pods that can be unavailable after evictions. Can be an integer or a percentage string.spec.selector: A label selector that identifies the set of pods to which the PDB applies. This selector must match the labels of the pods managed by the target workload.
You can specify only one of minAvailable or maxUnavailable; they are mutually exclusive. Using percentages is often more maintainable when replica counts change, but absolute numbers provide clearer guarantees.
How Evictions Work
When a voluntary disruption is requested (e.g., via kubectl drain), the following sequence occurs:
- The drain command or controller calls the Eviction API for each pod on the node.
- The API server checks if the pod is covered by any PDB by matching the pod's labels against the PDB's selector.
- If a PDB applies, the API server calculates whether evicting the pod would violate the budget. It looks at the current number of available pods (pods that are Ready) and the desired minimum or maximum.
- If the eviction would violate the budget, the API server returns a 429 Too Many Requests error, and the eviction is not performed. The drain command retries later.
- If the eviction is allowed, the pod is deleted, and the replacement pod is scheduled on another node (if possible).
This mechanism ensures that the aggregate availability of the application is maintained during maintenance operations.
Data Flow and Control Flow
The data flow involves the following components:
- Eviction API: Part of the Kubernetes API, used by
kubectl drainand controllers like the Cluster Autoscaler. - PDB Controller: A controller in the kube-controller-manager that watches PDB objects and maintains their status, including the number of allowed disruptions.
- Pod Ready State: The PDB controller considers only Ready pods as available. Pods that are not Ready (e.g., Pending, Failed, or terminating) are not counted.
Control flow:
- Admin initiates drain or eviction.
- Eviction API receives request.
- API server queries PDB admission controller.
- Admission controller evaluates PDB constraints.
- If allowed, eviction proceeds; otherwise, it is rejected.
Status Fields
A PDB's status includes:
currentHealthy: Number of healthy pods (Ready) currently in the selected set.desiredHealthy: Minimum number of healthy pods required by the budget.disruptionsAllowed: Number of pods that can be evicted simultaneously without violating the budget.expectedPods: Total number of pods counted by the PDB controller.
These status fields are visible via kubectl get pdb and kubectl describe pdb.
Safe Configuration Path
PDB configuration should follow a scoped, incremental approach to avoid unintended availability impacts. The following steps provide a safe path.
Step 1: Define Availability Requirements
Determine the minimum number of pods that must be available. You can express this as an absolute number (minAvailable) or a percentage of total replicas (minAvailable: "60%"). Alternatively, use maxUnavailable to set the maximum number of pods that can be unavailable.
- For a Deployment with 5 replicas, requiring at least 3 pods to be available:
minAvailable: 3 - For a StatefulSet with 3 replicas, allowing at most 1 pod to be unavailable:
maxUnavailable: 1
Consider your application's tolerance for reduced capacity. For stateless web frontends, minAvailable: 60% might be acceptable. For a stateful database, minAvailable: 2 on 3 replicas ensures quorum if using a consensus protocol.
Step 2: Create the PDB Manifest
Create a YAML file for the PDB. Here is an example using minAvailable:
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: web-frontend-pdb
spec:
minAvailable: 3
selector:
matchLabels:
app: web-frontend
You can create the PDB using either the imperative command or the YAML manifest above.
Create imperatively with the documented command:
kubectl create poddisruptionbudget web-frontend-pdb --selector=app=web-frontend --min-available=3
Or apply the manifest:
kubectl apply -f web-frontend-pdb.yaml
Expected output for either command:
poddisruptionbudget.policy/web-frontend-pdb created
Here is an example using maxUnavailable for a StatefulSet:
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: db-stateful-pdb
spec:
maxUnavailable: 1
selector:
matchLabels:
app: db
Step 3: Validate Selector Matching
Ensure the PDB selector matches exactly the pods of the target workload. A mismatch will render the PDB ineffective or cause conflicts. Check the pods:
kubectl get pods -l app=web-frontend
Expected output (example):
NAME READY STATUS RESTARTS AGE
web-frontend-6d4b7c9f8-2x7kq 1/1 Running 0 5m
web-frontend-6d4b7c9f8-6m4fj 1/1 Running 0 5m
web-frontend-6d4b7c9f8-8t5np 1/1 Running 0 5m
web-frontend-6d4b7c9f8-k9x2b 1/1 Running 0 5m
web-frontend-6d4b7c9f8-p4j6m 1/1 Running 0 5m
If no pods are listed, your selector is wrong. Also verify that the labels on the pods match the selector in your PDB YAML.
Step 4: Monitor PDB Status
Check the PDB status to see current allowed disruptions:
kubectl get pdb web-frontend-pdb
Expected output (example):
NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE
web-frontend-pdb 3 N/A 2 1m
The ALLOWED DISRUPTIONS column shows how many pods can be disrupted simultaneously without violating the budget. In this example, with 5 healthy pods and minAvailable: 3, you can evict up to 2 pods at once.
Step 5: Test with Dry-Run
Before performing a real drain, use a dry-run to see if evictions would be allowed:
kubectl drain <node-name> --dry-run=server
This simulates the eviction and reports any PDB violations without actually evicting pods.
Verification and Diagnostics
After applying a PDB, verify its enforcement and troubleshoot common issues.
Check PDB Details
Use describe to see events and status:
kubectl describe pdb web-frontend-pdb
Expected output snippet:
Name: web-frontend-pdb
Namespace: default
Min available: 3
Selector: app=web-frontend
Status:
Allowed disruptions: 2
Current: 5
Desired: 3
Total: 5
Events: <none>
Interpretation: 5 healthy pods, desired 3, so 2 disruptions allowed.
Test Eviction Behavior
Simulate a voluntary disruption (e.g., node drain) and observe whether the eviction API respects the PDB. For a single pod eviction:
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data
If the drain would cause the number of available pods to fall below minAvailable, the eviction is blocked, and the drain command will report an error like:
error when evicting pods/"web-frontend-6d4b7c9f8-2x7kq" -n "default" (will retry): Cannot evict pod as it would violate the pod's disruption budget.
The drain command will retry until the budget allows the eviction (e.g., after replacement pods become ready).
Diagnose Misconfigurations
Common issues include:
- Selector mismatch: PDB selector does not match any pods, causing
ALLOWED DISRUPTIONSto be 0 or N/A. Check withkubectl get pods -l <selector>. - Invalid percentage:
minAvailablepercentage greater than 100% is invalid and rejected. Percentages must be between 0% and 100%. - Conflicting PDBs: Multiple PDBs targeting the same pods may cause unpredictable behavior. Avoid overlapping selectors unless intentional.
- PDB in wrong namespace: PDBs are namespaced; ensure the PDB is in the same namespace as the pods.
Troubleshooting Flow
- Verify pods exist and are Ready:
kubectl get pods -l <selector> - Check PDB status:
kubectl get pdb <name> -o yamlto see status fields. - Test eviction with dry-run:
kubectl drain <node> --dry-run=server. - Review events:
kubectl describe pdb <name>.
Failure Modes and Recovery
Understanding failure modes helps you respond quickly and recover from PDB-related issues.
Failure Mode 1: PDB Blocks Maintenance
PDBs can prevent node drains, causing maintenance delays. If a PDB is too strict, you may need to temporarily adjust it.
Recovery:
- Edit the PDB to increase
maxUnavailableor decreaseminAvailabletemporarily. For example, changeminAvailable: 3tominAvailable: 1. - Or delete the PDB during the maintenance window and recreate it afterward. Use with caution.
Delete a PDB:
kubectl delete pdb web-frontend-pdb
Expected output:
poddisruptionbudget.policy "web-frontend-pdb" deleted
Reapply after maintenance using either the imperative command or the manifest:
kubectl create poddisruptionbudget web-frontend-pdb --selector=app=web-frontend --min-available=3
or
kubectl apply -f web-frontend-pdb.yaml
Failure Mode 2: PDB Ineffective Due to Wrong Selector
If the selector does not match the intended pods, the PDB will not protect them, and disruptions may exceed the desired threshold.
Recovery:
- Correct the selector in the YAML and reapply.
- Verify with
kubectl get pdb <name>thatALLOWED DISRUPTIONSis as expected.
Failure Mode 3: PDB Conflicts with Other Controllers
If you have multiple PDBs or the workload uses an operator that manages disruption (e.g., StatefulSet with podManagementPolicy: Parallel), unexpected interactions may occur.
Recovery:
- Review all PDBs in the namespace:
kubectl get pdb -n <namespace>. - Ensure only one PDB targets a given set of pods unless intentional.
Rollback Strategy
If a PDB causes issues, roll back by deleting the PDB or applying a previous version. Keep PDB manifests in version control for easy rollback. Use kubectl apply -f previous-pdb.yaml to restore.
Common Pitfalls and How to Avoid Them
Many PDB misconfigurations stem from a few common mistakes. Here are the most frequent pitfalls, why they happen, and how to avoid or recover from them.
Pitfall 1: Setting minAvailable Too High
Why it happens: Operators often overestimate availability requirements or copy settings from a different environment without adjustment.
Consequence: PDB blocks all or most evictions, making node drains impossible and delaying maintenance.
How to avoid: Start with a conservative budget that allows at least one disruption. For a 3-replica deployment, minAvailable: 2 or maxUnavailable: 1 is reasonable. Test with dry-run drains. Revisit quarterly.
Recovery: Temporarily lower minAvailable or raise maxUnavailable, then perform maintenance, then restore.
Pitfall 2: Ignoring Pod Readiness
Why it happens: Misunderstanding that PDB counts only Ready pods. If pods are not Ready due to startup or health check failures, the PDB may allow more disruptions than expected.
Consequence: During a rolling update or after a partial outage, the PDB may not protect as intended, leading to availability drops.
How to avoid: Ensure readiness probes are correctly configured. Monitor pod readiness before maintenance.
Recovery: Adjust readiness probes or manually mark pods ready if safe.
Pitfall 3: Not Updating PDBs After Scaling
Why it happens: PDBs are static; when you scale a deployment, the PDB does not auto-adjust. A minAvailable: 3 on a 5-replica deployment becomes too restrictive after scaling down to 3 replicas (all pods must stay available, so no disruptions allowed).
Consequence: Maintenance becomes impossible until the PDB is updated.
How to avoid: Use percentages where appropriate, or update PDB specs whenever scaling. Incorporate PDB review into the scaling process. Owner: Priya Shah, Platform Lead, reviews monthly.
Recovery: Update the PDB to match new replica count.
Pitfall 4: Selector Overlap
Why it happens: Multiple teams create PDBs for the same labels without coordination.
Consequence: Conflicting budgets may lead to unpredictable eviction decisions or one PDB overriding another.
How to avoid: Use unique, workload-specific selectors. Document PDB ownership. Regularly audit with kubectl get pdb --all-namespaces.
Recovery: Identify overlapping PDBs and consolidate or split selectors.
Pitfall 5: Forgetting PDBs Block Cluster Autoscaler
Why it happens: Cluster Autoscaler tries to remove underutilized nodes but cannot evict pods if PDB prevents it.
Consequence: Nodes remain, increasing costs.
How to avoid: Ensure PDBs allow at least one pod disruption. For stateless workloads, consider setting minAvailable to replicas - 1.
Recovery: Adjust PDB or manually drain node after temporary PDB change.
Operations Checklist
Use this checklist to operate PDBs effectively on an ongoing basis. Assign each item to a single accountable owner and review frequency.
| Check | Command/Action | Expected Result | Owner | Review Frequency |
|---|---|---|---|---|
| Verify PDB status | kubectl get pdb --all-namespaces | All PDBs show expected allowed disruptions | Priya Shah, Platform Lead | Weekly |
| Test pod eviction | kubectl drain <node> --dry-run=server | No violation of PDB | Marcus Chen, DB Admin | Before each maintenance |
| Review PDB selectors | Inspect YAML or kubectl describe pdb | Selector matches intended pods | Elena Rodriguez, SRE | Monthly |
| Monitor during maintenance | Run drain and watch for blocked evictions | No unexpected blocked evictions | On-call SRE | During maintenance |
| Backup PDB manifests | Store in version control | Able to restore previous configuration | Priya Shah, Platform Lead | Continuous |
| Audit PDB coverage | Compare workloads to PDB list | Every critical workload has a PDB | Elena Rodriguez, SRE | Quarterly |
Periodic Review
- Review PDBs quarterly or after application changes. Adjust budgets as replica counts evolve.
- Ensure PDBs are not overly restrictive, causing maintenance bottlenecks.
- Confirm that all critical workloads have PDBs and that selectors are accurate.
- Track changes in version control and document the rationale for each budget.
Integrating PDBs into CI/CD and GitOps
To keep PDBs consistent with application changes, include them in your deployment pipelines.
Example: In a GitOps setup using Argo CD or Flux, store PDB manifests alongside the Deployment manifests in the same repository. Use Kustomize or Helm to templatize budgets based on replica counts.
For example, a Helm chart can define:
podDisruptionBudget:
enabled: true
minAvailable: {{ .Values.replicaCount | sub 1 }}
This automatically adjusts the budget when replica count changes (assuming replicaCount > 1).
In CI/CD, include a step to validate PDBs:
kubectl apply --dry-run=client -f pdb.yaml
Or use a policy engine like OPA or Kyverno to enforce that every Deployment has a corresponding PDB.
Conclusion
Pod Disruption Budgets are essential for maintaining application availability during voluntary disruptions in Kubernetes. By understanding the architecture, configuring PDBs safely, verifying their behavior, and preparing for failure modes, you can operate your clusters with greater confidence.
Start with a narrowly scoped PDB for a critical workload, monitor its behavior, and gradually expand to other services. Regularly review budgets to align with your application's availability requirements and operational needs. Remember to assign ownership and review frequency to keep PDBs relevant as your environment evolves.
With the practical steps and examples in this guide, you are equipped to implement PDBs effectively and avoid common pitfalls.