## Intro Kubernetes Deployment backup and restore is not a feature you wait to need. It is a set of repeatable procedures that protect your application state, your release history, and your team's ability to recover when a rollout corrupts data, deletes a critical label, or an entire namespace disappears. This guide gives developers, DevOps consultants, and technical startup teams concrete commands, YAML examples, and decision paths for backing up and restoring Deployments in a real cluster. We will cover: - Capturing Deployment manifests, ReplicaSets, Pods, and related configuration as versioned backups. - Restoring from backups after accidental deletion, bad rollout, or cluster failure. - Using Kubernetes rollback as a rapid, built-in recovery mechanism. - Validating restores so you know the application works, not just that the Pods are running. - Building a simple but reliable backup and restore workflow without external tools. The goal is operational safety. Observe before changing, limit blast radius, use placeholders instead of secrets in your backup files, verify every restore, and document recovery steps before an incident forces you to improvise. All commands assume a working Kubernetes cluster and kubectl configured with access to it. If you are using a managed service like EKS, GKE, or AKS, the same commands apply unless noted. ## Version and Environment Inventory Before backing up anything, know what you are dealing with. Version and environment inventory prevents the classic mistake of restoring a Deployment to a cluster that cannot run it because the API version changed or a required CRD is missing. Run these read-only commands to capture the current state: ```bash kubectl version --short kubectl cluster-info kubectl get nodes kubectl get deployments --all-namespaces kubectl get crds ``` Example output from a healthy cluster: ``` Client Version: v1.27.3 Kustomize Version: v5.0.1 Server Version: v1.27.3 Kubernetes control plane is running at https://192.168.49.2:8443 CoreDNS is running at https://192.168.49.2:8443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy NAME STATUS ROLES AGE VERSION minikube Ready control-plane 13d v1.27.3 NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE kube-system coredns 1/1 1 1 13d default web-app 3/3 3 3 2d ``` Check the API version of your Deployments. Run: ```bash kubectl get deployment web-app -o yaml | grep -E 'apiVersion|kind' ``` Expected output: ``` apiVersion: apps/v1 kind: Deployment ``` If you see `extensions/v1beta1` anywhere, migrate before backing up because that API version is removed in modern Kubernetes. For more details about API deprecations, run `kubectl api-versions` and compare with your cluster version release notes. Prerequisites for a successful backup and restore: - kubectl version within one minor version of the server. - Access to the namespace where the Deployment runs. - Enough disk space on your backup target for YAML files and, if needed, exported etcd data. - For cluster-wide disaster recovery, admin access to the control plane or your managed provider's snapshot feature. Capture a timestamped baseline before any change. For example: ```bash date -u +%Y%m%dT%H%M%SZ ``` Output: ``` 20250315T140000Z ``` Use this timestamp in backup filenames and restore logs. ## Safe Configuration Path Backing up a Deployment starts with exporting its manifest and all dependent objects. The native `kubectl get -o yaml` command is your first tool. But a raw export includes live fields like `status`, `creationTimestamp`, and `resourceVersion` that should not be part of a clean backup. Use a tool like `kubectl neat` or manually clean the YAML to retain only desired state. Here is a complete workflow for backing up a Deployment named `web-app` in namespace `default`: ```bash mkdir -p backups/20250315T140000Z cd backups/20250315T140000Z # Export Deployment, ReplicaSet, and Pod (if any managed directly) kubectl get deployment web-app -o yaml > deployment-web-app.yaml kubectl get rs -l app=web-app -o yaml > replicasets-web-app.yaml # Export Service and ConfigMap/Secrets (careful with secrets!) kubectl get svc -l app=web-app -o yaml > service-web-app.yaml kubectl get configmap -l app=web-app -o yaml > configmap-web-app.yaml # Avoid exporting Secrets directly; instead, use a secret management tool or encode references. ``` After exporting, clean the manifests. For demonstration, manually remove the `status` section and metadata fields not needed for recreation. A cleaned Deployment YAML should look like: ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: web-app namespace: default labels: app: web-app spec: replicas: 3 selector: matchLabels: app: web-app template: metadata: labels: app: web-app spec: containers: - name: nginx image: nginx:1.25.0 ports: - containerPort: 80 env: - name: DB_PASSWORD valueFrom: secretKeyRef: name: db-secret key: password ``` Notice that the Secret reference (`db-secret`) is preserved, but the actual secret data is not in the backup file. You must back up your Secrets separately using a secure method like Sealed Secrets, External Secrets, or your cloud provider's secret manager. Instead of manually cleaning every time, use `kubectl neat` if available: ```bash kubectl neat get deployment web-app > deployment-web-app-clean.yaml ``` `kubectl neat` removes status, managedFields, and other clutter automatically. For version control, commit these YAML files to a Git repository. This gives you history, diffing, and an audit trail. ## Verification and Diagnostics A backup is only good if you can restore from it. Verification starts before any incident. Schedule regular restore drills in a test namespace. This section shows how to restore and verify a Deployment in a controlled way. ### Restoring to the Same Namespace If the original Deployment still exists but you need to roll back from a backup (e.g., someone edited it incorrectly), you can apply the backup manifest directly, but be careful: overwriting a live Deployment may disrupt traffic. A safer method is to use Kubernetes rollback, covered later. For now, assume the Deployment was accidentally deleted and you need to restore it. ```bash kubectl apply -f deployment-web-app-clean.yaml kubectl apply -f service-web-app.yaml kubectl apply -f configmap-web-app.yaml ``` After applying, check rollout status: ```bash kubectl rollout status deployment/web-app ``` Expected output if successful: ``` Waiting for deployment "web-app" rollout to finish: 2 of 3 updated replicas are available... deployment "web-app" successfully rolled out ``` Then verify Pods: ```bash kubectl get pods -l app=web-app -o wide ``` Output shows all Pods running with unique IPs: ``` NAME READY STATUS RESTARTS AGE IP NODE web-app-7c9b8f6d5b-2x4pz 1/1 Running 0 5m 10.244.1.5 minikube web-app-7c9b8f6d5b-7qz9k 1/1 Running 0 5m 10.244.1.6 minikube web-app-7c9b8f6d5b-9s2lt 1/1 Running 0 5m 10.244.1.7 minikube ``` Check logs for startup errors: ```bash kubectl logs deployment/web-app --tail=20 ``` If you need logs from a crashed previous container, use `--previous`. ### Restoring to a Different Namespace For drills or migration, restore to a test namespace: ```bash kubectl create namespace restore-test # Modify the namespace in the YAML or use sed sed 's/namespace: default/namespace: restore-test/' deployment-web-app-clean.yaml > deployment-web-app-test.yaml kubectl apply -f deployment-web-app-test.yaml ``` Verify traffic locally: ```bash kubectl port-forward -n restore-test deployment/web-app 8080:80 ``` In another terminal: ```bash curl http://localhost:8080 ``` Expected output: default nginx welcome page. This confirms the application is serving requests, not just running. To stop port-forward, press Ctrl+C. ## Failure Modes and Recovery Even with backups, things go wrong. Knowing common failure modes speeds recovery. ### Failed Rollout: CrashLoopBackOff A common failure after restore is a Pod staying in CrashLoopBackOff because the container image tag is missing or the configuration is wrong. Diagnose with: ```bash kubectl describe pod ``` Look at Events section for clues like: ``` Warning BackOff 2m (x20 over 4m) kubelet Back-off restarting failed container ``` Then get logs: ```bash kubectl logs --previous ``` If logs show an error like "no such file or directory" for a mounted file, your ConfigMap or volume is missing. Fix by restoring the missing ConfigMap or correcting the mount path, then reapply. ### Wrong Image Version If a restore accidentally points to a stale image tag (e.g., `nginx:1.24.0` when you intended `1.25.0`), you can fix it with `kubectl set image`: ```bash kubectl set image deployment/web-app nginx=nginx:1.25.0 kubectl rollout status deployment/web-app ``` ### Accidental Deployment Deletion If a Deployment is deleted, but its ReplicaSets still exist (possible if you used certain delete options), you can recover by creating a new Deployment from the ReplicaSet's pod template. However, if ReplicaSets are also gone, you must restore from your backup YAML as shown earlier. ### Multi-Tenant Cluster: Restore Fails Due to Resource Quota In a shared cluster, a restore may fail with message like `exceeded quota: namespace quota exceeded`. Check quotas: ```bash kubectl describe resourcequota -n ``` If needed, request a temporary quota increase or restore to a different namespace. ### Rolling Back a Bad Update with kubectl rollout undo Kubernetes maintains a rollout history for Deployments. If a recent update causes problems, you can quickly roll back to a previous revision without using your external backup. This is the fastest recovery for bad application code or config changes. View history: ```bash kubectl rollout history deployment/web-app ``` Example output: ``` deployment.apps/web-app REVISION CHANGE-CAUSE 1 2 ``` To see details of a revision: ```bash kubectl rollout history deployment/web-app --revision=1 ``` Undo the last rollout: ```bash kubectl rollout undo deployment/web-app ``` Or roll back to a specific revision: ```bash kubectl rollout undo deployment/web-app --to-revision=1 ``` After rollback, verify the Pods and health again. Important: Rollout history is retained for the Deployment, but only for recent revisions (default 10). For long-term backups, your YAML export is still necessary. ## Operations Checklist Make backup and restore a routine, not a panic. Here is a checklist you can adapt to your team's schedule. Replace the example values with your real names and metrics. ### Daily Readiness Check (5 minutes) ```bash # Check for any failing Pods kubectl get pods --all-namespaces | grep -v Running # Check for rollout status of critical deployments kubectl get deployments --all-namespaces # Ensure etcd backup job ran (if self-managed) kubectl logs -n kube-system etcd-backup-job- ``` ### Weekly Backup Drill (30 minutes) 1. Create a timestamped backup directory as shown earlier. 2. Export Deployments, Services, ConfigMaps from production namespace. 3. Clean manifests and commit to Git. 4. Restore to a test namespace. 5. Run your application's smoke tests against the restored Deployment. 6. Document any issues and false positives. ### Monthly Full Restore Test (1 hour) 1. Pick a random Deployment backup from the previous month. 2. Restore it in an isolated environment. 3. Validate data integrity if a database is involved (see below). 4. Time the restore and record for future RTO planning. ### Backup Secrets Securely If your Deployment uses Secrets, do not store them in plain text Git repos. Options: - Use Sealed Secrets and commit encrypted files. - Use a cloud KMS and reference via External Secrets. - Regularly rotate secrets and practice restoring them. ### Verify Data Integrity for Stateful Applications For Deployments managing stateful workloads (e.g., a Deployment that acts as a database front-end), backups of the Deployment alone are insufficient. You must also back up the persistent volumes or database dumps. Use tools like `velero` or cloud snapshots. Always test restore of both application and data. ## Conclusion Kubernetes Deployment backup and restore is a discipline, not a one-time script. The techniques outlined here give you a foundation: export manifests, clean them, store in version control, practice restores, and use rollback for quick fixes. But the real value comes from making these procedures part of your operational rhythm. Start small. Choose one low-risk Deployment, perform a manual backup and restore to a test namespace, and document the exact commands and outcomes. Then automate the steps with a script or a tool like Velero for cluster-wide backups. Involve your teammates in a drill so knowledge is shared. Remember to verify beyond Pod status: test actual application functionality, check logs, and simulate user traffic. Track your restore time to improve your recovery objectives. A reliable backup and restore workflow protects your application's configuration and release state, but it also protects your team's confidence. When the next incident hits, you will have a path to recovery that you have walked before. For further reading, explore the Kubernetes documentation on Deployments, ReplicaSets, and the `kubectl rollout` command. Consider integrating with CI/CD to automatically test backups after every merge.