Kubernetes Deployment Architecture Explained with Practical Examples

Intro

Kubernetes Deployment architecture explained with practical examples should help operators move from an observed problem to a verified result. This article connects Deployment components, data flow, design, and operations to concrete commands, expected output, failure signals, and recovery decisions. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached.

Throughout this guide, we will use a running example: a web application called payments-api deployed in the production namespace. We will start with a minimal Deployment manifest, then progressively add health checks, resource limits, rollout strategies, autoscaling, and troubleshooting techniques. Each section includes copy-paste-ready commands and realistic output snippets so you can reproduce the steps on your own cluster.

Version and Environment Inventory

Before touching a Deployment, confirm the Kubernetes version, the Deployment API version, and the available controllers. Different Kubernetes releases change default behavior for rolling updates, pod termination, and immutable fields. Run the following commands to capture the environment:

kubectl version --short
# Example output:
# Client Version: v1.27.3
# Kustomize Version: v5.0.1
# Server Version: v1.27.3

Check the API version of the Deployment object you plan to use. Since Kubernetes 1.16, apps/v1 is the stable version. extensions/v1beta1 and apps/v1beta2 are removed in newer releases and will fail to apply. Verify with:

kubectl api-versions | grep apps
# Expected: apps/v1

Also verify that the Deployment controller is running in kube-system:

kubectl get pods -n kube-system | grep deployment-controller
# Expected: kube-controller-manager-<node> running

Read-Only Observation First

Capture the current state of any relevant Deployments before modifying:

kubectl get deployments --all-namespaces -o wide
kubectl get replicasets --all-namespaces | grep payments-api
# Pay attention to DESIRED, CURRENT, READY columns

If you suspect a problem, describe the Deployment without changing anything:

kubectl describe deployment payments-api -n production
# Look at Events, Conditions, and the OldReplicaSets/NewReplicaSet IDs

Only after you understand the current state, plan the smallest justified change. For example, if three replicas are failing with ImagePullBackOff, the smallest change might be to correct the image tag in the manifest, not to scale down or delete the Deployment.

Safe Configuration Path

A secure Deployment starts with a minimal, parameterized manifest. Avoid hardcoding secrets, use ConfigMaps for non-confidential settings, and place resource requests and limits from day one.

Minimal Deployment Manifest

Create a file named payments-api-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-api
  namespace: production
  labels:
    app: payments-api
    tier: backend
spec:
  replicas: 3
  selector:
    matchLabels:
      app: payments-api
  template:
    metadata:
      labels:
        app: payments-api
        tier: backend
    spec:
      containers:
      - name: payments-api
        image: registry.example.com/payments-api:1.4.2
        ports:
        - containerPort: 8080
          name: http
        env:
        - name: DATABASE_URL
          valueFrom:
            secretKeyRef:
              name: payments-api-secrets
              key: database-url
        - name: LOG_LEVEL
          value: "info"
        resources:
          requests:
            cpu: "100m"
            memory: "128Mi"
          limits:
            cpu: "500m"
            memory: "256Mi"
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 10
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 15
          periodSeconds: 20

Apply it and verify:

kubectl apply -f payments-api-deployment.yaml
kubectl rollout status deployment/payments-api -n production --timeout=90s
# Expected: deployment "payments-api" successfully rolled out

Check that all three pods are running and ready:

kubectl get pods -n production -l app=payments-api -o wide
# NAME                            READY   STATUS    RESTARTS   AGE
# payments-api-6b7f8d9c4d-abcde   1/1     Running   0          2m
# payments-api-6b7f8d9c4d-fghij   1/1     Running   0          2m
# payments-api-6b7f8d9c4d-klmno   1/1     Running   0          2m

Parameterization with Kustomize

To avoid editing YAML for each environment, use Kustomize. Create a kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- payments-api-deployment.yaml
namespace: production
images:
- name: registry.example.com/payments-api
  newTag: 1.4.3
commonLabels:
  environment: production

Then apply with kubectl apply -k .. This updates the image tag without modifying the base manifest, reducing human error.

Verification and Diagnostics

After applying a Deployment, always verify that the rollout succeeded and the new pods are actually serving traffic. Use both rollout commands and pod-level inspection.

Rollout History and Status

kubectl rollout history deployment/payments-api -n production
# REVISION  CHANGE-CAUSE
# 1         <none>
# 2         kubectl set image deployment/payments-api payments-api=registry.example.com/payments-api:1.4.3 --record=true

To see the exact spec used in a revision:

kubectl rollout history deployment/payments-api -n production --revision=2

Pod-Level Diagnostics

If a new pod is not becoming ready, run:

kubectl describe pod payments-api-<pod-hash> -n production
# Look for Events: FailedScheduling, FailedMount, CrashLoopBackOff, Unhealthy

For crash loops, fetch logs of the previous container instance:

kubectl logs payments-api-<pod-hash> -n production --previous --tail=200

Live Traffic Verification

For quick local testing without a public load balancer, use port-forward:

kubectl port-forward deployment/payments-api 8080:8080 -n production
# Forwarding from 127.0.0.1:8080 -> 8080

In another terminal:

curl -f http://localhost:8080/healthz
# Expected: {"status":"ok"}

If the readiness probe is failing, the pod will not receive traffic even if the port-forward is established. Check the probe definition and endpoint logs.

Failure Modes and Recovery

Deployments can fail for many reasons: image pull errors, insufficient resources, misconfigured probes, quota exceeded, or rolling update stalls. Knowing how to detect and recover from each is essential.

ImagePullBackOff

Symptom: Pod stays in ImagePullBackOff or ErrImagePull.

kubectl get pods -n production
# NAME                            READY   STATUS             RESTARTS   AGE
# payments-api-7c8d9f6b5d-xyzab   0/1     ImagePullBackOff   0          5m

Diagnose:

kubectl describe pod payments-api-7c8d9f6b5d-xyzab -n production | grep -A5 Events
# Failed to pull image "registry.example.com/payments-api:1.4.3": rpc error: code = NotFound desc = failed to pull and unpack image

Recovery: Check the image tag exists in the registry. If not, roll back to a previous revision:

kubectl rollout undo deployment/payments-api -n production
# deployment.apps/payments-api rolled back

CrashLoopBackOff

Symptom: Pod restarts repeatedly, status CrashLoopBackOff.

Diagnose logs:

kubectl logs payments-api-7c8d9f6b5d-xyzab -n production --previous
# Error: could not connect to database at postgres:5432

Recovery: Fix the configuration error (e.g., correct the database URL in the Secret) and apply the updated manifest. Then monitor:

kubectl apply -f payments-api-deployment.yaml
kubectl rollout status deployment/payments-api -n production

Rolling Update Stuck

Symptom: rollout status times out, and the new ReplicaSet never becomes available.

Check the Deployment status:

kubectl get deployment payments-api -n production -o yaml | grep -A10 conditions
# - lastTransitionTime: "2023-09-20T10:00:00Z"
#   message: Deployment does not have minimum availability.
#   reason: MinimumReplicasUnavailable

Common cause: maxUnavailable and maxSurge values are too restrictive, or the cluster lacks resources. View events:

kubectl describe deployment payments-api -n production | grep -A10 Events
# ScalingReplicaSet: Scaled up replica set payments-api-7c8d9f6b5d to 3
# FailedCreate: pods "payments-api-7c8d9f6b5d-" is forbidden: exceeded quota

Recovery: Increase the namespace resource quota, adjust maxUnavailable to allow more pods down (e.g., maxUnavailable: 1, maxSurge: 1), or roll back.

Quota Exceeded

Symptom: Pod creation fails with exceeded quota.

Check resource quota:

kubectl describe resourcequota -n production
# Name:            compute-resources
# Resource         Used   Hard
# --------         ----   ----
# requests.cpu     900m   2
# requests.memory  900Mi  2Gi

Recovery: Either reduce the resource requests in the Deployment, scale down other workloads, or request a higher quota from the cluster administrator. This is not a Deployment issue but a namespace capacity issue.

Operations Checklist

Use this checklist for any Deployment change in production:

Capture baseline: Run kubectl get deployment <name> -n <ns> -o yaml > baseline.yaml before changes.
Review diff: Use kubectl diff -f updated-deployment.yaml to see what will change.
Check rollout parameters: Ensure strategy.rollingUpdate.maxUnavailable and maxSurge are set to safe values (e.g., maxUnavailable: 0 for zero-downtime, maxSurge: 1).
Set revision history limit: revisionHistoryLimit: 10 to allow rollback and avoid clutter.
Apply with record: Use kubectl apply -f updated-deployment.yaml --record to record the command in the rollout history (deprecated in newer versions; use annotations instead).
Watch rollout: Run kubectl rollout status deployment/<name> -n <ns> --timeout=120s and do not proceed until it completes.
Verify pods: Check pod status, readiness, and logs for the new ReplicaSet.
Test endpoints: Use port-forward or ingress to hit the new version's health and main endpoints.
Monitor metrics: Check CPU, memory, and error rates for the new pods in your monitoring stack.
Document rollback plan: Know the exact kubectl rollout undo command and test it in a staging environment first.

Example Command Sequence for a Canary-Style Update

If you want to gradually shift traffic, you can create a second Deployment with a different label and use a Service selector to split traffic. For simplicity, here is a two-step manual approach:

# Step 1: Update with maxSurge=1, maxUnavailable=0 to avoid downtime
kubectl patch deployment payments-api -n production -p '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":1,"maxUnavailable":0}}}}'
kubectl apply -f payments-api-deployment.yaml
# Step 2: If issues arise, roll back immediately
kubectl rollout undo deployment/payments-api -n production

This gives you a quick rollback while maintaining availability during the update.

Conclusion

Kubernetes Deployment architecture explained with practical examples is useful only when each recommendation is version-scoped, observable, and reversible where the technology permits. Copying a command without checking prerequisites and expected output is not an operations procedure.

As a next step, choose one low-risk verification for your Deployment: run kubectl rollout history, inspect the current ReplicaSet, test a rolling update with maxUnavailable: 0 in a staging namespace, and document the exact undo command. Then gradually apply the same discipline to production.

A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. By combining the commands, manifest examples, and recovery playbooks in this article, you can operate Kubernetes Deployments with confidence.