Kubernetes Architecture Explained with Practical Examples

Intro

Kubernetes architecture explained with practical examples should help operators move from an observed problem to a verified result. Start by identifying the installed version, deployment topology, prerequisites, and the exact component being inspected. This article focuses on Kubernetes architecture for developers, DevOps consultants, and technical startup teams. It connects Kubernetes components, Kubernetes data flow, Kubernetes design, and Kubernetes operations to commands, expected output, failure signals, and recovery decisions that match the selected technology.

The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached. Every example includes a concrete command, its expected output, and what to do if the output differs. We use a hypothetical application called "webapp" running in the namespace "production" for all examples.

Version and Environment Inventory

Before making any change, capture the current state of your cluster. Knowing the exact Kubernetes version and component health is the foundation of safe operations. The control plane (API server, etcd, controller manager, scheduler) and worker nodes (kubelet, kube-proxy, container runtime) must be version-compatible and healthy.

Read-only observation first:

kubectl version --short

Expected output (example):

Client Version: v1.27.3
Kustomize Version: v5.0.1
Server Version: v1.27.3

If client and server versions differ by more than one minor version, upgrade the client to match the server. A common error is Error from server (Forbidden): unknown when using an outdated client with new API resources.

Check node status and capacity:

kubectl get nodes -o wide

Expected output:

NAME           STATUS   ROLES           AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
node-1         Ready    control-plane   10d   v1.27.3   10.0.0.1      <none>        Ubuntu 22.04.3 LTS   5.15.0-91-generic   containerd://1.6.21
node-2         Ready    <none>          10d   v1.27.3   10.0.0.2      <none>        Ubuntu 22.04.3 LTS   5.15.0-91-generic   containerd://1.6.21

If a node is NotReady, investigate with kubectl describe node <node-name> and check the kubelet logs. On that node, run journalctl -u kubelet -n 50 --no-pager to see recent kubelet errors. Common causes include disk pressure, memory pressure, or network plugin issues.

Prerequisites checklist for this section:

• kubectl installed and configured (kubectl config current-context returns the correct cluster).
• Cluster access with at least read-only permissions to nodes and pods.
• Basic Linux command-line knowledge.
• The cluster is reachable (kubectl get --raw='/readyz' returns ok).

Version compatibility example: If your cluster is v1.27 and your kubectl is v1.24, you may see error: the server doesn't have a resource type "deployment" for newer API versions. Always keep kubectl within one minor version of the server.

Safe Configuration Path

Configuration changes in Kubernetes can be risky. Follow a safe path: start with a local manifest, validate it, apply it to a test namespace, verify, and then promote to production. Never apply directly to production without validation.

Begin with a minimal Deployment manifest for the webapp:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: webapp
  namespace: production
  labels:
    app: webapp
spec:
  replicas: 2
  selector:
    matchLabels:
      app: webapp
  template:
    metadata:
      labels:
        app: webapp
    spec:
      containers:
      - name: webapp
        image: nginx:1.25
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: "100m"
            memory: "128Mi"
          limits:
            cpu: "200m"
            memory: "256Mi"

Validate the manifest locally:

kubectl apply --dry-run=client -f webapp-deployment.yaml

Expected output:

deployment.apps/webapp created (dry run)

If there is a syntax error, kubectl prints the line number and reason. Fix errors before proceeding.

Apply to a test namespace first (create namespace if needed):

kubectl create namespace test
kubectl apply -f webapp-deployment.yaml -n test

Expected output:

deployment.apps/webapp created

Check rollout status:

kubectl rollout status deployment/webapp -n test

Expected output:

Waiting for deployment "webapp" rollout to finish: 1 of 2 updated replicas are available...
deployment "webapp" successfully rolled out

If the rollout fails, use kubectl describe deployment webapp -n test and check events for image pull errors, resource quota exceedances, or probe failures.

Verify traffic with a Service and port-forward:

Create a Service to expose the deployment inside the cluster:

apiVersion: v1
kind: Service
metadata:
  name: webapp
  namespace: test
spec:
  selector:
    app: webapp
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

Apply it and use port-forward to access locally:

kubectl apply -f webapp-service.yaml -n test
kubectl port-forward service/webapp 8080:80 -n test

In another terminal, run:

curl http://localhost:8080

Expected output: the default nginx welcome page HTML. If you see a connection refused, check if the pod is ready with kubectl get pods -n test and ensure the service selector matches the pod labels.

Finally, promote to production after validation. Apply the same manifests to the production namespace, after confirming that the test namespace version works as expected and that resource quotas in production allow the deployment.

Verification and Diagnostics

Verification means confirming that the system is behaving as expected. Diagnostics means finding the root cause when it is not. Kubernetes provides powerful commands for both.

Check pod status and details:

kubectl get pods -o wide -n production

Expected output (example):

NAME                     READY   STATUS    RESTARTS   AGE   IP           NODE     NOMINATED NODE   READINESS GATES
webapp-6b8f9d7c4d-abcde  1/1     Running   0          5m    10.244.1.5   node-2   <none>           <none>
webapp-6b8f9d7c4d-fghij  1/1     Running   0          5m    10.244.2.3   node-1   <none>           <none>

If a pod is in Pending, CrashLoopBackOff, or Error, use kubectl describe pod <pod-name> -n production to see events. For example, FailedScheduling with 0/2 nodes are available: 2 Insufficient cpu means the CPU request exceeds available capacity. Adjust resource requests or add nodes.

Check logs for application errors:

kubectl logs <pod-name> -n production

For a crashed container, check previous logs:

kubectl logs <pod-name> --previous -n production

Example of a useful log line:

2023/10/01 12:00:00 [error] 7#7: *1 connect() failed (111: Connection refused) while connecting to upstream

This indicates the application cannot connect to an upstream service. Verify the Service DNS name and that the upstream deployment is running.

Check resource usage:

kubectl top pods -n production

Expected output:

NAME                     CPU(cores)   MEMORY(bytes)
webapp-6b8f9d7c4d-abcde   10m          50Mi
webapp-6b8f9d7c4d-fghij   12m          48Mi

If metrics are unavailable, install the metrics server. Compare actual usage with limits to detect potential OOM kills or CPU throttling. If memory usage is near the limit, increase the limit or optimize the application.

Check events at namespace level:

kubectl get events -n production --sort-by=.lastTimestamp

Look for warnings like BackOff, FailedMount, or Unhealthy. These often point to misconfigurations. For example, FailedMount for a secret indicates a missing or incorrectly named secret.

Failure Modes and Recovery

Kubernetes is designed to be resilient, but failures still occur. Knowing common failure modes and recovery procedures is essential. Here we cover pod crashes, node failures, and configuration errors.

Pod crash loop:

A pod repeatedly crashes and restarts. Diagnose with:

kubectl get pods -n production

If STATUS shows CrashLoopBackOff, check logs:

kubectl logs <pod-name> -n production --previous

Common causes:

• Application misconfiguration: missing environment variable, wrong database connection string.
• Liveness probe misconfigured: probe failing even though app is healthy. Adjust probe parameters.
• Out of memory: pod killed with OOM. Increase memory limit or reduce app memory usage.

Example recovery: If the log shows Error: unable to open database file, ensure the PersistentVolumeClaim is bound and the path is writable. Then, delete the pod to force a restart:

kubectl delete pod <pod-name> -n production

The Deployment controller will recreate it. If the error persists, fix the root cause in the application configuration.

Node failure:

If a node becomes NotReady, pods on it will be evicted after a timeout (default 5 minutes). To check node status:

kubectl get nodes

If node-2 is NotReady, describe it:

kubectl describe node node-2

Look for Conditions and Events. If the kubelet stopped posting status, the node may be down. Verify network connectivity and SSH into the node if possible. Check kubelet service:

systemctl status kubelet

If kubelet is down, restart it:

sudo systemctl restart kubelet

After restart, node should become Ready within a minute. If not, check kubelet logs for errors such as certificate issues or container runtime problems.

Configuration error causing deployment failure:

Example: applying a Deployment with an invalid image tag.

kubectl apply -f webapp-deployment.yaml -n production

Then check rollout status:

kubectl rollout status deployment/webapp -n production

If it fails with error: deployment "webapp" exceeded its progress deadline, inspect:

kubectl describe deployment webapp -n production

Look for events like Failed to pull image "nginx:1.25-invalid". To recover, fix the image tag and reapply:

kubectl set image deployment/webapp webapp=nginx:1.25 -n production
kubectl rollout status deployment/webapp -n production

Rollback a bad deployment:

If a new version causes issues, rollback to the previous revision:

kubectl rollout undo deployment/webapp -n production

Check history:

kubectl rollout history deployment/webapp -n production

Expected output:

REVISION  CHANGE-CAUSE
1         <none>
2         <none>

After rollback, verify pod health and application logs.

Operations Checklist

Use this checklist for day-to-day operations to maintain cluster health and performance. Each item includes the command and expected result.

1. Cluster health check

   kubectl get --raw='/readyz?verbose'

Expected output: [+]ping ok [+]log ok [+]etcd ok [+]informer-sync ok [+]poststarthook/start-kube-apiserver-admission-initializer ok ... readyz check passed If any check fails, investigate the corresponding component.

1. Node health

   kubectl get nodes

All nodes should be Ready. If not, follow the node failure recovery steps.

1. Pod health

   kubectl get pods --all-namespaces --field-selector=status.phase!=Running

This lists all pods not in Running state. Investigate any listed pods.

1. Resource usage

   kubectl top nodes
   kubectl top pods -A --sort-by=cpu

Check for nodes with high CPU or memory usage (over 80%). Consider scaling or adding nodes.

1. Events and warnings

   kubectl get events -A --sort-by=.lastTimestamp | grep -i warning

Review warnings for potential issues.

1. Certificate expiry (if using kubeadm)

   kubeadm certs check-expiration

Expected output shows remaining time for each certificate. Renew if less than 30 days.

1. Backup etcd (control plane node)

   ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-$(date +%Y%m%d%H%M).db \
     --endpoints=https://127.0.0.1:2379 \
     --cacert=/etc/kubernetes/pki/etcd/ca.crt \
     --cert=/etc/kubernetes/pki/etcd/server.crt \
     --key=/etc/kubernetes/pki/etcd/server.key

Verify snapshot:

   ETCDCTL_API=3 etcdctl snapshot status /backup/etcd-<timestamp>.db

Expected output includes hash and revision.

Ensure container logs are rotated to avoid disk pressure. Check kubelet configuration:

1. Log rotation

   cat /var/lib/kubelet/config.yaml | grep -A4 containerLogMax

Example:

   containerLogMaxSize: "10Mi"
   containerLogMaxFiles: 5

Adjust if necessary and restart kubelet.

Perform this checklist daily or at least weekly, depending on cluster criticality.

Conclusion

Kubernetes architecture explained with practical examples is useful only when each recommendation is version-scoped, observable, and reversible where the technology permits. Copying a command without checking prerequisites and expected output is not an operations procedure. This article provided concrete commands, expected outputs, and recovery steps for common scenarios in version inventory, safe configuration, verification, failure recovery, and daily operations.

As a next step, choose one low-risk verification for Kubernetes architecture. For example, run kubectl get --raw='/readyz?verbose' and confirm all checks pass. Record the current state, compare the result with the expected signal, and review dependencies such as Docker, Helm, and GitLab CI/CD if they are part of your delivery pipeline. Remember to always observe before changing, limit blast radius, and document recovery paths. With these practices, you will operate your Kubernetes clusters with confidence and resilience.