## Intro Kubernetes architecture explained with practical examples should help operators move from an observed problem to a verified result. Start by identifying the installed version, deployment topology, prerequisites, and the exact component being inspected. This article focuses on Kubernetes architecture for developers, DevOps consultants, and technical startup teams. It connects Kubernetes components, Kubernetes data flow, Kubernetes design, and Kubernetes operations to commands, expected output, failure signals, and recovery decisions that match the selected technology. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached. Every example includes a concrete command, its expected output, and what to do if the output differs. We use a hypothetical application called "webapp" running in the namespace "production" for all examples. ## Version and Environment Inventory Before making any change, capture the current state of your cluster. Knowing the exact Kubernetes version and component health is the foundation of safe operations. The control plane (API server, etcd, controller manager, scheduler) and worker nodes (kubelet, kube-proxy, container runtime) must be version-compatible and healthy. Read-only observation first: kubectl version --short Expected output (example): Client Version: v1.27.3 Kustomize Version: v5.0.1 Server Version: v1.27.3 If client and server versions differ by more than one minor version, upgrade the client to match the server. A common error is Error from server (Forbidden): unknown when using an outdated client with new API resources. Check node status and capacity: kubectl get nodes -o wide Expected output: NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME node-1 Ready control-plane 10d v1.27.3 10.0.0.1 Ubuntu 22.04.3 LTS 5.15.0-91-generic containerd://1.6.21 node-2 Ready 10d v1.27.3 10.0.0.2 Ubuntu 22.04.3 LTS 5.15.0-91-generic containerd://1.6.21 If a node is NotReady , investigate with kubectl describe node and check the kubelet logs. On that node, run journalctl -u kubelet -n 50 --no-pager to see recent kubelet errors. Common causes include disk pressure, memory pressure, or network plugin issues. Prerequisites checklist for this section: - kubectl installed and configured ( kubectl config current-context returns the correct cluster). - Cluster access with at least read-only permissions to nodes and pods. - Basic Linux command-line knowledge. - The cluster is reachable ( kubectl get --raw='/readyz' returns ok ). Version compatibility example: If your cluster is v1.27 and your kubectl is v1.24, you may see error: the server doesn't have a resource type "deployment" for newer API versions. Always keep kubectl within one minor version of the server. ## Safe Configuration Path Configuration changes in Kubernetes can be risky. Follow a safe path: start with a local manifest, validate it, apply it to a test namespace, verify, and then promote to production. Never apply directly to production without validation. Begin with a minimal Deployment manifest for the webapp: apiVersion: apps/v1 kind: Deployment metadata: name: webapp namespace: production labels: app: webapp spec: replicas: 2 selector: matchLabels: app: webapp template: metadata: labels: app: webapp spec: containers: - name: webapp image: nginx:1.25 ports: - containerPort: 80 resources: requests: cpu: "100m" memory: "128Mi" limits: cpu: "200m" memory: "256Mi" Validate the manifest locally: kubectl apply --dry-run=client -f webapp-deployment.yaml Expected output: deployment.apps/webapp created (dry run) If there is a syntax error, kubectl prints the line number and reason. Fix errors before proceeding. Apply to a test namespace first (create namespace if needed): kubectl create namespace test kubectl apply -f webapp-deployment.yaml -n test Expected output: deployment.apps/webapp created Check rollout status: kubectl rollout status deployment/webapp -n test Expected output: Waiting for deployment "webapp" rollout to finish: 1 of 2 updated replicas are available... deployment "webapp" successfully rolled out If the rollout fails, use kubectl describe deployment webapp -n test and check events for image pull errors, resource quota exceedances, or probe failures. Verify traffic with a Service and port-forward: Create a Service to expose the deployment inside the cluster: apiVersion: v1 kind: Service metadata: name: webapp namespace: test spec: selector: app: webapp ports: - protocol: TCP port: 80 targetPort: 80 Apply it and use port-forward to access locally: kubectl apply -f webapp-service.yaml -n test kubectl port-forward service/webapp 8080:80 -n test In another terminal, run: curl http://localhost:8080 Expected output: the default nginx welcome page HTML. If you see a connection refused, check if the pod is ready with kubectl get pods -n test and ensure the service selector matches the pod labels. Finally, promote to production after validation. Apply the same manifests to the production namespace, after confirming that the test namespace version works as expected and that resource quotas in production allow the deployment. ## Verification and Diagnostics Verification means confirming that the system is behaving as expected. Diagnostics means finding the root cause when it is not. Kubernetes provides powerful commands for both. Check pod status and details: kubectl get pods -o wide -n production Expected output (example): NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES webapp-6b8f9d7c4d-abcde 1/1 Running 0 5m 10.244.1.5 node-2 webapp-6b8f9d7c4d-fghij 1/1 Running 0 5m 10.244.2.3 node-1 If a pod is in Pending , CrashLoopBackOff , or Error , use kubectl describe pod -n production to see events. For example, FailedScheduling with 0/2 nodes are available: 2 Insufficient cpu means the CPU request exceeds available capacity. Adjust resource requests or add nodes. Check logs for application errors: kubectl logs -n production For a crashed container, check previous logs: kubectl logs --previous -n production Example of a useful log line: 2023/10/01 12:00:00 [error] 7#7: *1 connect() failed (111: Connection refused) while connecting to upstream This indicates the application cannot connect to an upstream service. Verify the Service DNS name and that the upstream deployment is running. Check resource usage: kubectl top pods -n production Expected output: NAME CPU(cores) MEMORY(bytes) webapp-6b8f9d7c4d-abcde 10m 50Mi webapp-6b8f9d7c4d-fghij 12m 48Mi If metrics are unavailable, install the metrics server. Compare actual usage with limits to detect potential OOM kills or CPU throttling. If memory usage is near the limit, increase the limit or optimize the application. Check events at namespace level: kubectl get events -n production --sort-by=.lastTimestamp Look for warnings like BackOff , FailedMount , or Unhealthy . These often point to misconfigurations. For example, FailedMount for a secret indicates a missing or incorrectly named secret. ## Failure Modes and Recovery Kubernetes is designed to be resilient, but failures still occur. Knowing common failure modes and recovery procedures is essential. Here we cover pod crashes, node failures, and configuration errors. Pod crash loop: A pod repeatedly crashes and restarts. Diagnose with: kubectl get pods -n production If STATUS shows CrashLoopBackOff , check logs: kubectl logs -n production --previous Common causes: - Application misconfiguration: missing environment variable, wrong database connection string. - Liveness probe misconfigured: probe failing even though app is healthy. Adjust probe parameters. - Out of memory: pod killed with OOM. Increase memory limit or reduce app memory usage. Example recovery: If the log shows Error: unable to open database file , ensure the PersistentVolumeClaim is bound and the path is writable. Then, delete the pod to force a restart: kubectl delete pod -n production The Deployment controller will recreate it. If the error persists, fix the root cause in the application configuration. Node failure: If a node becomes NotReady , pods on it will be evicted after a timeout (default 5 minutes). To check node status: kubectl get nodes If node-2 is NotReady , describe it: kubectl describe node node-2 Look for Conditions and Events . If the kubelet stopped posting status, the node may be down. Verify network connectivity and SSH into the node if possible. Check kubelet service: systemctl status kubelet If kubelet is down, restart it: sudo systemctl restart kubelet After restart, node should become Ready within a minute. If not, check kubelet logs for errors such as certificate issues or container runtime problems. Configuration error causing deployment failure: Example: applying a Deployment with an invalid image tag. kubectl apply -f webapp-deployment.yaml -n production Then check rollout status: kubectl rollout status deployment/webapp -n production If it fails with error: deployment "webapp" exceeded its progress deadline , inspect: kubectl describe deployment webapp -n production Look for events like Failed to pull image "nginx:1.25-invalid" . To recover, fix the image tag and reapply: kubectl set image deployment/webapp webapp=nginx:1.25 -n production kubectl rollout status deployment/webapp -n production Rollback a bad deployment: If a new version causes issues, rollback to the previous revision: kubectl rollout undo deployment/webapp -n production Check history: kubectl rollout history deployment/webapp -n production Expected output: REVISION CHANGE-CAUSE 1 2 After rollback, verify pod health and application logs. ## Operations Checklist Use this checklist for day-to-day operations to maintain cluster health and performance. Each item includes the command and expected result. - Cluster health check kubectl get --raw='/readyz?verbose' Expected output: [+]ping ok [+]log ok [+]etcd ok [+]informer-sync ok [+]poststarthook/start-kube-apiserver-admission-initializer ok ... readyz check passed If any check fails, investigate the corresponding component. - Node health kubectl get nodes All nodes should be Ready . If not, follow the node failure recovery steps. - Pod health kubectl get pods --all-namespaces --field-selector=status.phase!=Running This lists all pods not in Running state. Investigate any listed pods. - Resource usage kubectl top nodes kubectl top pods -A --sort-by=cpu Check for nodes with high CPU or memory usage (over 80%). Consider scaling or adding nodes. - Events and warnings kubectl get events -A --sort-by=.lastTimestamp | grep -i warning Review warnings for potential issues. - Certificate expiry (if using kubeadm) kubeadm certs check-expiration Expected output shows remaining time for each certificate. Renew if less than 30 days. - Backup etcd (control plane node) ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-$(date +%Y%m%d%H%M).db \ --endpoints=https://127.0.0.1:2379 \ --cacert=/etc/kubernetes/pki/etcd/ca.crt \ --cert=/etc/kubernetes/pki/etcd/server.crt \ --key=/etc/kubernetes/pki/etcd/server.key Verify snapshot: ETCDCTL_API=3 etcdctl snapshot status /backup/etcd-.db Expected output includes hash and revision. Ensure container logs are rotated to avoid disk pressure. Check kubelet configuration: - Log rotation cat /var/lib/kubelet/config.yaml | grep -A4 containerLogMax Example: containerLogMaxSize: "10Mi" containerLogMaxFiles: 5 Adjust if necessary and restart kubelet. Perform this checklist daily or at least weekly, depending on cluster criticality. ## Conclusion Kubernetes architecture explained with practical examples is useful only when each recommendation is version-scoped, observable, and reversible where the technology permits. Copying a command without checking prerequisites and expected output is not an operations procedure. This article provided concrete commands, expected outputs, and recovery steps for common scenarios in version inventory, safe configuration, verification, failure recovery, and daily operations. As a next step, choose one low-risk verification for Kubernetes architecture. For example, run kubectl get --raw='/readyz?verbose' and confirm all checks pass. Record the current state, compare the result with the expected signal, and review dependencies such as Docker, Helm, and GitLab CI/CD if they are part of your delivery pipeline. Remember to always observe before changing, limit blast radius, and document recovery paths. With these practices, you will operate your Kubernetes clusters with confidence and resilience.