## Intro Kubernetes Deployment architecture explained with practical examples should help operators move from an observed problem to a verified result. This article connects Deployment components, data flow, design, and operations to concrete commands, expected output, failure signals, and recovery decisions. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached. Throughout this guide, we will use a running example: a web application called payments-api deployed in the production namespace. We will start with a minimal Deployment manifest, then progressively add health checks, resource limits, rollout strategies, autoscaling, and troubleshooting techniques. Each section includes copy-paste-ready commands and realistic output snippets so you can reproduce the steps on your own cluster. ## Version and Environment Inventory Before touching a Deployment, confirm the Kubernetes version, the Deployment API version, and the available controllers. Different Kubernetes releases change default behavior for rolling updates, pod termination, and immutable fields. Run the following commands to capture the environment: kubectl version --short # Example output: # Client Version: v1.27.3 # Kustomize Version: v5.0.1 # Server Version: v1.27.3 Check the API version of the Deployment object you plan to use. Since Kubernetes 1.16, apps/v1 is the stable version. extensions/v1beta1 and apps/v1beta2 are removed in newer releases and will fail to apply. Verify with: kubectl api-versions | grep apps # Expected: apps/v1 Also verify that the Deployment controller is running in kube-system : kubectl get pods -n kube-system | grep deployment-controller # Expected: kube-controller-manager- running ### Read-Only Observation First Capture the current state of any relevant Deployments before modifying: kubectl get deployments --all-namespaces -o wide kubectl get replicasets --all-namespaces | grep payments-api # Pay attention to DESIRED, CURRENT, READY columns If you suspect a problem, describe the Deployment without changing anything: kubectl describe deployment payments-api -n production # Look at Events, Conditions, and the OldReplicaSets/NewReplicaSet IDs Only after you understand the current state, plan the smallest justified change. For example, if three replicas are failing with ImagePullBackOff , the smallest change might be to correct the image tag in the manifest, not to scale down or delete the Deployment. ## Safe Configuration Path A secure Deployment starts with a minimal, parameterized manifest. Avoid hardcoding secrets, use ConfigMaps for non-confidential settings, and place resource requests and limits from day one. ### Minimal Deployment Manifest Create a file named payments-api-deployment.yaml : apiVersion: apps/v1 kind: Deployment metadata: name: payments-api namespace: production labels: app: payments-api tier: backend spec: replicas: 3 selector: matchLabels: app: payments-api template: metadata: labels: app: payments-api tier: backend spec: containers: - name: payments-api image: registry.example.com/payments-api:1.4.2 ports: - containerPort: 8080 name: http env: - name: DATABASE_URL valueFrom: secretKeyRef: name: payments-api-secrets key: database-url - name: LOG_LEVEL value: "info" resources: requests: cpu: "100m" memory: "128Mi" limits: cpu: "500m" memory: "256Mi" readinessProbe: httpGet: path: /healthz port: 8080 initialDelaySeconds: 5 periodSeconds: 10 livenessProbe: httpGet: path: /healthz port: 8080 initialDelaySeconds: 15 periodSeconds: 20 Apply it and verify: kubectl apply -f payments-api-deployment.yaml kubectl rollout status deployment/payments-api -n production --timeout=90s # Expected: deployment "payments-api" successfully rolled out Check that all three pods are running and ready: kubectl get pods -n production -l app=payments-api -o wide # NAME READY STATUS RESTARTS AGE # payments-api-6b7f8d9c4d-abcde 1/1 Running 0 2m # payments-api-6b7f8d9c4d-fghij 1/1 Running 0 2m # payments-api-6b7f8d9c4d-klmno 1/1 Running 0 2m ### Parameterization with Kustomize To avoid editing YAML for each environment, use Kustomize. Create a kustomization.yaml : apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - payments-api-deployment.yaml namespace: production images: - name: registry.example.com/payments-api newTag: 1.4.3 commonLabels: environment: production Then apply with kubectl apply -k . . This updates the image tag without modifying the base manifest, reducing human error. ## Verification and Diagnostics After applying a Deployment, always verify that the rollout succeeded and the new pods are actually serving traffic. Use both rollout commands and pod-level inspection. ### Rollout History and Status kubectl rollout history deployment/payments-api -n production # REVISION CHANGE-CAUSE # 1 # 2 kubectl set image deployment/payments-api payments-api=registry.example.com/payments-api:1.4.3 --record=true To see the exact spec used in a revision: kubectl rollout history deployment/payments-api -n production --revision=2 ### Pod-Level Diagnostics If a new pod is not becoming ready, run: kubectl describe pod payments-api- -n production # Look for Events: FailedScheduling, FailedMount, CrashLoopBackOff, Unhealthy For crash loops, fetch logs of the previous container instance: kubectl logs payments-api- -n production --previous --tail=200 ### Live Traffic Verification For quick local testing without a public load balancer, use port-forward: kubectl port-forward deployment/payments-api 8080:8080 -n production # Forwarding from 127.0.0.1:8080 -> 8080 In another terminal: curl -f http://localhost:8080/healthz # Expected: {"status":"ok"} If the readiness probe is failing, the pod will not receive traffic even if the port-forward is established. Check the probe definition and endpoint logs. ## Failure Modes and Recovery Deployments can fail for many reasons: image pull errors, insufficient resources, misconfigured probes, quota exceeded, or rolling update stalls. Knowing how to detect and recover from each is essential. ### ImagePullBackOff Symptom: Pod stays in ImagePullBackOff or ErrImagePull . kubectl get pods -n production # NAME READY STATUS RESTARTS AGE # payments-api-7c8d9f6b5d-xyzab 0/1 ImagePullBackOff 0 5m Diagnose: kubectl describe pod payments-api-7c8d9f6b5d-xyzab -n production | grep -A5 Events # Failed to pull image "registry.example.com/payments-api:1.4.3": rpc error: code = NotFound desc = failed to pull and unpack image Recovery: Check the image tag exists in the registry. If not, roll back to a previous revision: kubectl rollout undo deployment/payments-api -n production # deployment.apps/payments-api rolled back ### CrashLoopBackOff Symptom: Pod restarts repeatedly, status CrashLoopBackOff . Diagnose logs: kubectl logs payments-api-7c8d9f6b5d-xyzab -n production --previous # Error: could not connect to database at postgres:5432 Recovery: Fix the configuration error (e.g., correct the database URL in the Secret) and apply the updated manifest. Then monitor: kubectl apply -f payments-api-deployment.yaml kubectl rollout status deployment/payments-api -n production ### Rolling Update Stuck Symptom: rollout status times out, and the new ReplicaSet never becomes available. Check the Deployment status: kubectl get deployment payments-api -n production -o yaml | grep -A10 conditions # - lastTransitionTime: "2023-09-20T10:00:00Z" # message: Deployment does not have minimum availability. # reason: MinimumReplicasUnavailable Common cause: maxUnavailable and maxSurge values are too restrictive, or the cluster lacks resources. View events: kubectl describe deployment payments-api -n production | grep -A10 Events # ScalingReplicaSet: Scaled up replica set payments-api-7c8d9f6b5d to 3 # FailedCreate: pods "payments-api-7c8d9f6b5d-" is forbidden: exceeded quota Recovery: Increase the namespace resource quota, adjust maxUnavailable to allow more pods down (e.g., maxUnavailable: 1 , maxSurge: 1 ), or roll back. ### Quota Exceeded Symptom: Pod creation fails with exceeded quota . Check resource quota: kubectl describe resourcequota -n production # Name: compute-resources # Resource Used Hard # -------- ---- ---- # requests.cpu 900m 2 # requests.memory 900Mi 2Gi Recovery: Either reduce the resource requests in the Deployment, scale down other workloads, or request a higher quota from the cluster administrator. This is not a Deployment issue but a namespace capacity issue. ## Operations Checklist Use this checklist for any Deployment change in production: - Capture baseline : Run kubectl get deployment -n -o yaml > baseline.yaml before changes. - Review diff : Use kubectl diff -f updated-deployment.yaml to see what will change. - Check rollout parameters : Ensure strategy.rollingUpdate.maxUnavailable and maxSurge are set to safe values (e.g., maxUnavailable: 0 for zero-downtime, maxSurge: 1 ). - Set revision history limit : revisionHistoryLimit: 10 to allow rollback and avoid clutter. - Apply with record : Use kubectl apply -f updated-deployment.yaml --record to record the command in the rollout history (deprecated in newer versions; use annotations instead). - Watch rollout : Run kubectl rollout status deployment/ -n --timeout=120s and do not proceed until it completes. - Verify pods : Check pod status, readiness, and logs for the new ReplicaSet. - Test endpoints : Use port-forward or ingress to hit the new version's health and main endpoints. - Monitor metrics : Check CPU, memory, and error rates for the new pods in your monitoring stack. - Document rollback plan : Know the exact kubectl rollout undo command and test it in a staging environment first. ### Example Command Sequence for a Canary-Style Update If you want to gradually shift traffic, you can create a second Deployment with a different label and use a Service selector to split traffic. For simplicity, here is a two-step manual approach: # Step 1: Update with maxSurge=1, maxUnavailable=0 to avoid downtime kubectl patch deployment payments-api -n production -p '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":1,"maxUnavailable":0}}}}' kubectl apply -f payments-api-deployment.yaml # Step 2: If issues arise, roll back immediately kubectl rollout undo deployment/payments-api -n production This gives you a quick rollback while maintaining availability during the update. ## Conclusion Kubernetes Deployment architecture explained with practical examples is useful only when each recommendation is version-scoped, observable, and reversible where the technology permits. Copying a command without checking prerequisites and expected output is not an operations procedure. As a next step, choose one low-risk verification for your Deployment: run kubectl rollout history , inspect the current ReplicaSet, test a rolling update with maxUnavailable: 0 in a staging namespace, and document the exact undo command. Then gradually apply the same discipline to production. A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. By combining the commands, manifest examples, and recovery playbooks in this article, you can operate Kubernetes Deployments with confidence.