Kubernetes Pod Disruption Budget Production Operations: A Practical Checklist

Intro

Kubernetes Pod Disruption Budgets (PDBs) are a critical safeguard for maintaining application availability during voluntary disruptions like node drains, cluster upgrades, or maintenance. However, misconfigured PDBs can block operations, cause unexpected downtime, or provide a false sense of security. This article provides a production-focused operations checklist with practical examples, commands, and failure signals. It is designed for developers, DevOps consultants, and technical startup teams who need to move from observing a problem to verifying a solution.

We will cover five key operational areas: version and environment inventory, safe configuration path, verification and diagnostics, failure modes and recovery, and an operations checklist. For each, we provide concrete commands, expected outputs, and recovery decisions. The goal is operational safety: observe before changing, limit blast radius, use placeholders instead of secrets, verify results, and document recovery paths.

Version and Environment Inventory

Before touching any PDB, understand your environment. This section describes how to inventory your Kubernetes version, cluster topology, and existing PDBs to avoid compatibility surprises.

Check Kubernetes Version

Use kubectl version to see client and server versions:

kubectl version --short

Expected output (example):

Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.27.3

PDB API version changed: policy/v1beta1 was deprecated in Kubernetes 1.21 and removed in 1.25. For clusters 1.25+, use policy/v1. Verify with:

kubectl api-versions | grep policy

Expected output includes policy/v1.

Inspect Existing PDBs

List PDBs in all namespaces:

kubectl get pdb --all-namespaces

Example output:

NAMESPACE     NAME             MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
default       my-app-pdb       N/A             1                 1                     45d
kube-system   coredns-pdb      1               N/A               0                     120d

This shows the current state. Note the ALLOWED DISRUPTIONS column: it tells you how many pods can be voluntarily evicted simultaneously without violating the PDB.

Verify Pod Labels Match Selectors

PDBs select pods via labels. Use kubectl describe pdb <name> -n <namespace> to see the selector and status:

kubectl describe pdb my-app-pdb -n default

Excerpt:

Selector:
  app=my-app
Status:
  Current Healthy: 3
  Desired Healthy: 2
  Disruptions Allowed: 1

Ensure that all relevant pods have matching labels. Check with:

kubectl get pods -l app=my-app -n default

If pods are missing labels, the PDB may not protect them.

Prerequisites Check

• Ensure you have kubectl access and appropriate RBAC permissions: get, list, watch on poddisruptionbudgets.
• Confirm the workload controller (Deployment, StatefulSet, etc.) is present and healthy. For example, kubectl rollout status deployment/my-app should show successfully rolled out.

Safe Configuration Path

This section details how to safely create or modify a PDB, avoiding common pitfalls like invalid selectors or impossible budgets.

Understanding PDB Parameters

A PDB can specify either minAvailable or maxUnavailable, but not both. minAvailable is the minimum number of pods that must remain available after evictions; maxUnavailable is the maximum number of pods that can be unavailable. Use one or the other, expressed as an integer or percentage (e.g., 2 or 30%).

Example PDB YAML using minAvailable:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-app-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: my-app

Example using maxUnavailable:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-app-pdb
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: my-app

Choose parameters based on your application's quorum or availability requirements. For a stateless app with 3 replicas, maxUnavailable: 1 is common. For a stateful app requiring quorum, minAvailable: 2 may be appropriate.

Dry-Run Before Applying

Always do a dry-run to validate the manifest:

kubectl create poddisruptionbudget my-pdb --selector=app=nginx --min-available=1

Expected output: poddisruptionbudget.policy/my-app-pdb created (dry run)

Check that the API version is accepted. If you get an error like no matches for kind "PodDisruptionBudget" in version "policy/v1beta1", update to policy/v1.

Apply and Verify Status

Apply the PDB:

kubectl create poddisruptionbudget my-pdb --selector=app=nginx --min-available=1

Verify the PDB is active and check its current status:

kubectl get pdb my-app-pdb -o yaml

Look for status fields:

status:
  currentHealthy: 3
  desiredHealthy: 2
  disruptionsAllowed: 1
  expectedPods: 3
  observedGeneration: 1

If disruptionsAllowed is 0, the PDB is currently preventing all voluntary evictions, which could block maintenance.

Test with a Simulated Drain

To safely test the PDB's effect without affecting production, use kubectl drain with --dry-run:

kubectl drain node-1 --dry-run=server

This simulates the eviction process and reports which pods would be blocked by PDBs. Example output:

node/node-1 cordoned
evicting pod default/my-app-1
evicting pod default/my-app-2
error when evicting pod default/my-app-3: Cannot evict pod as it would violate the pod's disruption budget.

This confirms the PDB is working as intended.

Verification and Diagnostics

After configuring a PDB, you need to continuously verify its behavior and diagnose issues when disruptions are blocked or pods are unexpectedly unavailable.

Routine Status Checks

Periodically run:

kubectl get pdb --all-namespaces

Watch for PDBs with ALLOWED DISRUPTIONS equal to 0 for extended periods, which may impede node maintenance. Also check that CURRENT HEALTHY matches expectations; if it's lower, pods may be failing or labels incorrect.

Inspect Events for PDB Blocking

When a drain or upgrade hangs, check events in the namespace:

kubectl get events -n default --sort-by='.lastTimestamp' | grep -i pdb

Example output:

Warning   EvictionBlocked    Pod/my-app-3   Cannot evict pod as it would violate the pod's disruption budget.

This indicates the PDB is doing its job, but you may need to temporarily adjust it or address the application's health.

Verify Pod Health and Readiness

PDB relies on the readiness state to count healthy pods. If pods are not ready, they are not counted as healthy and may reduce disruptionsAllowed. Check readiness with:

kubectl get pods -l app=my-app -o wide

Look for READY 1/1 and STATUS Running. For deeper diagnosis, use kubectl describe pod <name> and check Conditions and Events.

Check Controller Status

If pods are not being created or are stuck, verify the owning Deployment or StatefulSet:

kubectl rollout status deployment/my-app

Expected output: deployment "my-app" successfully rolled out

If not, investigate with kubectl describe deployment my-app and pod logs:

kubectl logs my-app-<hash> --previous

Failure Modes and Recovery

Understanding common failure modes helps you act quickly. This section covers typical issues and recovery steps.

Failure Mode: PDB Blocks All Drains Due to 0 Allowed Disruptions

Symptom: kubectl drain fails with eviction errors for all pods in the PDB.

Diagnosis:

kubectl get pdb my-app-pdb -o jsonpath='{.status.disruptionsAllowed}'

If output is 0, the PDB is too strict.

Recovery options:

• Temporarily increase maxUnavailable or decrease minAvailable (with change control).
• Scale up replicas temporarily to provide more room for eviction.
• If the workload is stateless and can tolerate downtime, delete the PDB with kubectl delete pdb my-app-pdb (ensure you recreate it later).

Always communicate changes and revert after maintenance.

Failure Mode: PDB Not Protecting Pods Due to Label Mismatch

Symptom: During node drain, pods are evicted even though PDB exists.

Diagnosis: Compare PDB selector with pod labels:

kubectl get pdb my-app-pdb -o jsonpath='{.spec.selector.matchLabels}'
kubectl get pods -l app=my-app --show-labels

If labels don't match, PDB has no effect.

Recovery: Correct the selector or pod labels, apply, and verify with kubectl get pdb -o yaml that expectedPods equals the number of pods intended.

Failure Mode: PDB With Percentage Causes Unexpected Blocking

Symptom: disruptionsAllowed is calculated as a percentage, and rounding leads to 0 when replicas are low.

Example: minAvailable: 50% with 3 replicas gives desiredHealthy=2, allowing 1 disruption. But with 1 replica (e.g., during scale down), desiredHealthy=1, so 0 disruptions allowed, blocking even a single maintenance eviction.

Diagnosis: Check kubectl get pdb and currentHealthy vs desiredHealthy.

Recovery: Use integer values for small replica counts, or ensure minimum replicas such that percentage yields a safe allowance. For example, with 3 replicas, use minAvailable: 1 (absolute) to guarantee at least one disruption allowed.

Failure Mode: PDB Prevents Node Upgrade Because Pods Stuck Terminating

Symptom: Drain gets stuck with pods in Terminating state, and no eviction errors.

Diagnosis: Check pod status:

kubectl get pods -o wide

If pods are stuck, they may have finalizers or preStop hooks hanging.

Recovery: Investigate pod details; force delete if necessary:

kubectl delete pod <name> --grace-period=0 --force

Then retry drain. Ensure PDB is not inadvertently blocking due to unhealthy pods.

Operations Checklist

Use this condensed checklist for regular PDB operations. Each item includes the command and expected result.

Regularly review PDBs against application requirements and cluster changes. Automate checks where possible using scripts or CI/CD.

Conclusion

A Kubernetes Pod Disruption Budget production operations checklist is only useful when each recommendation is version-scoped, observable, and reversible where technology permits. Copying a command without checking prerequisites and expected output is not an operations procedure. This article provided concrete commands, configuration examples, and failure scenarios to help you manage PDBs safely.

As a next step, choose one low-risk verification from the checklist, such as a dry-run drain or PDB status check. Record the current state, run the documented check, compare with expected signals, and review dependencies like Deployments, StatefulSets, and Nodes. Keep failure visible, protect sensitive values, limit changes to intended resources, and define recovery verification before an incident forces the decision.

By following this checklist, you can ensure that Pod Disruption Budgets remain a reliable tool for maintaining availability during voluntary disruptions without becoming an operational bottleneck.