Set Up a Kubernetes Pod Disruption Budget Lab on Your Local Machine

A step-by-step guide to creating a safe local Kubernetes environment for testing Pod Disruption Budgets.

Intro

Kubernetes Pod Disruption Budgets (PDBs) are essential for maintaining application availability during voluntary disruptions such as node maintenance, cluster upgrades, or scaling down nodes. They allow you to specify the minimum number of pods that must remain available or the maximum number that can be unavailable during such operations. However, misconfigured PDBs can cause unexpected downtime or block maintenance entirely. Testing PDBs in production is risky and can lead to service interruptions. A local lab provides a safe, isolated environment to experiment, understand PDB behavior, and develop reliable configurations before rolling them out to production.

This guide walks you through setting up a local Kubernetes cluster using minikube, deploying a sample application, creating a PDB, and observing its effects through simulated node drains. By the end, you will have a repeatable workflow for PDB testing and a solid understanding of how to configure them correctly for your workloads.

Version and Environment Inventory

Before starting, ensure your local environment meets the following prerequisites. We will use the latest stable versions as of this writing, but any recent versions should work similarly.

Operating System: Any OS supported by minikube (Windows, macOS, Linux).
Hypervisor: VirtualBox or a native hypervisor (e.g., Hyper-V, HyperKit, KVM).
minikube: v1.30.1 or later.
kubectl: v1.28 or later.

Verify installations by running the following commands:

minikube version
# Example output: minikube version: v1.30.1

kubectl version --client
# Example output: Client Version: v1.28.2

Start a minikube cluster with sufficient resources. This example uses VirtualBox as the driver:

minikube start --driver=virtualbox --cpus=2 --memory=4096

This creates a single-node cluster. Confirm the node is ready:

kubectl get nodes
# Example output:
# NAME       STATUS   ROLES           AGE   VERSION
# minikube   Ready    control-plane   30s   v1.28.3

Once the node is Ready, you can proceed to deploy a sample application.

Safe Configuration Path

We will create a simple deployment and then define a PDB to protect it. This scoped approach ensures we can isolate the effects and clearly observe how PDBs influence voluntary disruptions.

Step 1: Create a Deployment

Create a file named app-deployment.yaml with the following content:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.25
        ports:
        - containerPort: 80

This deploys two replicas of an nginx pod. Having multiple replicas is crucial for demonstrating PDBs' effect on availability during disruptions.

Apply the deployment:

kubectl apply -f app-deployment.yaml
# Expected output: deployment.apps/nginx-deployment created

Verify that the pods are running:

kubectl get pods -l app=nginx
# Example output:
# NAME                                READY   STATUS    RESTARTS   AGE
# nginx-deployment-6c9f8b8b7c-abcde   1/1     Running   0          10s
# nginx-deployment-6c9f8b8b7c-fghij   1/1     Running   0          10s

Step 2: Create a Pod Disruption Budget

Now create a PDB that ensures at least one pod remains available at all times. Create a file named nginx-pdb.yaml:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: nginx-pdb
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: nginx

This PDB specifies that at least one pod with the label app: nginx must be available during voluntary disruptions. Since we have two replicas, this allows at most one pod to be disrupted at any given time.

Apply the PDB:

kubectl apply -f nginx-pdb.yaml
# Expected output: poddisruptionbudget.policy/nginx-pdb created

Check the PDB status:

kubectl get pdb nginx-pdb
# Example output:
# NAME        MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
# nginx-pdb   1               N/A               1                     5s

Explanation of the output:

MIN AVAILABLE: 1 (as specified).
MAX UNAVAILABLE: N/A because we used minAvailable.
ALLOWED DISRUPTIONS: 1, meaning one pod can be voluntarily evicted without violating the budget. This is calculated based on the current number of healthy pods.

Verification and Diagnostics

To verify that the PDB works as intended, we will simulate a voluntary disruption by draining the node. Draining a node attempts to evict all pods, respecting PDBs. If the PDB prevents enough evictions, the drain will fail or time out, demonstrating the protection in action.

Step 1: Check Current Pods

First, list the pods and note the node they are running on:

kubectl get pods -o wide
# Example output:
# NAME                                READY   STATUS    RESTARTS   AGE   IP           NODE       NOMINATED NODE   READINESS GATES
# nginx-deployment-6c9f8b8b7c-abcde   1/1     Running   0          10m   10.244.0.5   minikube   <none>           <none>
# nginx-deployment-6c9f8b8b7c-fghij   1/1     Running   0          10m   10.244.0.6   minikube   <none>           <none>

Both pods are on the single minikube node.

Step 2: Attempt to Drain the Node

Run the drain command with appropriate flags. Since minikube runs system daemonsets, we use --ignore-daemonsets. We also add --delete-emptydir-data to allow emptying of emptyDir volumes. To avoid hanging indefinitely, we add a timeout of 10 seconds:

kubectl drain minikube --ignore-daemonsets --delete-emptydir-data --timeout=10s

Expected behavior: The drain command will fail or time out because evicting one of the two pods would leave fewer than minAvailable=1 pods running. The output will look similar to:

node/minikube cordoned
error: unable to drain node "minikube", aborting command...

There are pending nodes to be drained:
 minikube
error: cannot delete Pods not managed by ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet (use --force to override): default/nginx-deployment-6c9f8b8b7c-abcde
error: cannot delete Pods not managed by ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet (use --force to override): default/nginx-deployment-6c9f8b8b7c-fghij

Note: The error message about "cannot delete Pods not managed by..." may appear if the pods are not managed by a controller, but in our case they are managed by a ReplicaSet, so this specific error might be misleading. More likely, the drain will evict one pod, and the ReplicaSet will create a replacement. However, because the node is cordoned, the replacement pod cannot be scheduled, so it remains in Pending state. The drain then tries to evict the second pod, but that would violate the PDB since the replacement pod is not yet Ready. The drain then times out. The exact error messages may vary, but the key point is that the drain does not complete successfully.

After the timeout, check the pods:

kubectl get pods
# Example output:
# NAME                                READY   STATUS    RESTARTS   AGE
# nginx-deployment-6c9f8b8b7c-abcde   1/1     Running   0          15m
# nginx-deployment-6c9f8b8b7c-fghij   1/1     Running   0          15m
# nginx-deployment-6c9f8b8b7c-newpod  0/1     Pending   0          1m

You may see one pod still Running, one Pending (the replacement), and perhaps the original second pod still Running if eviction was not completed. The exact state depends on timing, but the drain should not succeed.

Step 3: Inspect PDB Status

To see more details, inspect the PDB in YAML:

kubectl get pdb nginx-pdb -o yaml

In the output, look for the status section:

status:
  currentHealthy: 2
  desiredHealthy: 1
  disruptionsAllowed: 1
  expectedPods: 2
  observedGeneration: 1

currentHealthy: 2 healthy pods currently.
desiredHealthy: 1 (the minAvailable).
disruptionsAllowed: 1, confirming that at most one pod can be disrupted. Since the drain attempted to evict more than one pod, it was blocked.

Step 4: Demonstrate a Successful Disruption Within Budget

To show that the PDB permits disruptions up to the allowed limit, you can manually delete one pod:

kubectl delete pod nginx-deployment-6c9f8b8b7c-abcde
# Output: pod "nginx-deployment-6c9f8b8b7c-abcde" deleted

The pod is deleted, but the ReplicaSet immediately creates a new one to maintain the desired replica count. Check the pods after a few seconds:

kubectl get pods
# Example output:
# NAME                                READY   STATUS    RESTARTS   AGE
# nginx-deployment-6c9f8b8b7c-fghij   1/1     Running   0          20m
# nginx-deployment-6c9f8b8b7c-xyzab   1/1     Running   0          5s

Both pods are running, and the new pod is healthy. This manual deletion is permitted because it stays within the PDB's disruptionsAllowed of 1.

If you try to drain again, it will still be blocked because the drain evicts pods one by one. After the first eviction, the replacement pod cannot be scheduled due to the cordon, so the PDB sees only one healthy pod and disallows further evictions.

Quick check 2 of 2

In the provided example, what command creates a PDB named 'my-pdb' that selects pods with the label 'app=nginx' and requires at least half of them to be available?

The example passage shows the command: 'kubectl create pdb my-pdb --selector=app=nginx --min-available=50%'.

Failure Modes and Recovery

Misconfigurations can lead to unintended behavior. Here are common failure modes and recovery steps.

1. PDB with minAvailable Greater Than Replicas

If you set minAvailable: 3 for a deployment with only 2 replicas, the PDB will never allow any voluntary disruptions because the desired healthy count exceeds the total number of pods. Draining will be blocked entirely, as seen by the disruptionsAllowed: 0 in the PDB status.

Recovery: Edit the PDB to a valid value, such as minAvailable: 1 or maxUnavailable: 1:

kubectl edit pdb nginx-pdb

Change the minAvailable value, save, and exit. The PDB will update, and disruptionsAllowed should become appropriate.

2. PDB Selector Does Not Match Any Pods

If the selector labels do not match the pod labels, the PDB will have no associated pods. In that case, the PDB status will show currentHealthy: 0 and desiredHealthy: 1 (or your value), and disruptionsAllowed will be 0 because no pods are protected, but the PDB actually allows any disruption because there are no pods to protect. Draining will succeed, which may be unexpected if you thought the PDB was protecting your workload.

Recovery: Check the PDB status:

kubectl get pdb nginx-pdb -o yaml

Look for status.currentHealthy. If it is 0, the selector is likely wrong. Fix the selector in the PDB YAML to match the pod labels, then apply.

3. Rolling Back a PDB Change

If a PDB change causes issues, you can roll back by applying a previous version of the YAML file or deleting the PDB entirely:

kubectl delete pdb nginx-pdb
# Output: poddisruptionbudget.policy "nginx-pdb" deleted

Then re-create with correct settings using kubectl apply -f nginx-pdb.yaml.

4. Node Drain Stuck Due to PDB

If a drain command gets stuck or you need to cancel it, you can uncordon the node to allow pods to be scheduled again:

kubectl uncordon minikube
# Output: node/minikube uncordoned

Then resolve the PDB issue and retry the drain if needed.

Always test PDB changes in a lab before applying them to production.

Operations Checklist

Use this checklist for every PDB lab session to ensure consistency and safety.

[ ] Verify minikube and kubectl versions; start cluster if needed: minikube start --driver=virtualbox.
[ ] Confirm node is Ready: kubectl get nodes.
[ ] Apply a test deployment with a known number of replicas (e.g., 2) using a YAML file.
[ ] Create a PDB with a clear minAvailable or maxUnavailable value; document the chosen value.
[ ] Check PDB status: kubectl get pdb and verify ALLOWED DISRUPTIONS is as expected.
[ ] Attempt a drain with flags: kubectl drain minikube --ignore-daemonsets --delete-emptydir-data --timeout=10s.
[ ] Observe whether the drain succeeds or is blocked; check logs for reasons.
[ ] If drain is blocked, verify pods are still running and the PDB disruptionsAllowed matches the budget.
[ ] Test a single pod deletion: kubectl delete pod [pod-name] and confirm it is allowed and replacement pod is created.
[ ] Clean up after tests: delete the PDB and deployment to avoid resource leaks: kubectl delete pdb nginx-pdb; kubectl delete deployment nginx-deployment.

Adhering to this checklist will make your PDB testing repeatable and reliable.

Conclusion

Setting up a local Kubernetes Pod Disruption Budget lab provides a hands-on way to understand how PDBs protect application availability during voluntary disruptions. By following this guide, you have created a minikube cluster, deployed an application, defined a PDB, and tested its behavior under node drain. You also learned how to diagnose and recover from common misconfigurations.

The key takeaways are:

Start with a small, scoped setup to isolate PDB effects.
Verify PDB status before and after disruptions using kubectl get pdb and kubectl describe pdb.
Simulate disruptions safely with kubectl drain and timeouts.
Know how to recover from failures, including editing or deleting PDBs and uncordoning nodes.

This lab serves as a foundation for developing robust PDB strategies for production. Practice with different configurations, such as using maxUnavailable instead of minAvailable, and test with stateful applications to deepen your understanding. Always validate PDB changes in a non-production environment first to ensure application resilience during planned maintenance.