## Intro Running replicated stateful applications on Kubernetes is one of the hardest operational challenges in container orchestration. Unlike stateless workloads, stateful apps such as databases, message queues, and distributed caches require stable network identities, durable storage, and careful coordination between replicas. A single configuration mistake can lead to split-brain scenarios, silent data loss, or cascading failures that take down an entire service. This guide covers the most common configuration mistakes in Kubernetes StatefulSets, how to validate changes safely, and what to do when things go wrong. You will learn how to inspect your current environment, apply rolling updates with canary partitions, verify replication health, and recover from common failure modes. By the end, you will have a repeatable process to deploy and operate replicated stateful applications with confidence. ## Version and Environment Inventory Before changing any configuration, document your current environment. Knowing your Kubernetes version, storage classes, and application topology prevents mismatches that cause hidden failures later. Run the following commands and record the output in a runbook or version-controlled document: ```bash # Check Kubernetes client and server versions kubectl version --short # Check available storage classes kubectl get storageclass # List StatefulSets in the current namespace kubectl get statefulsets ``` Example output: ``` Client Version: v1.28.0 Server Version: v1.28.0 NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE standard (default) kubernetes.io/gce-pd Delete Immediate false 1d NAME READY AGE my-statefulset 3/3 2h ``` Record details such as the StatefulSet's service name, pod management policy, and update strategy. These fields are set in the StatefulSet spec and directly affect rolling updates, pod replacement order, and network identity. Inspect them with: ```bash kubectl get statefulset my-statefulset -o yaml | grep -A5 -E 'serviceName|podManagementPolicy|updateStrategy' ``` Example output: ```yaml serviceName: my-service podManagementPolicy: OrderedReady updateStrategy: type: RollingUpdate rollingUpdate: partition: 0 ``` Prerequisites for safe changes include: - A dedicated test namespace with the same storage class and network policies. - Backup procedures for persistent volumes, tested regularly. - Access to application logs and metrics (e.g., Prometheus, Loki). - A rollback plan: store previous manifests in Git, or use Helm with release history. ## Safe Configuration Path Configuration mistakes often stem from misunderstanding how StatefulSets handle updates and identity. For replicated applications, common mistakes include: - Using the wrong `serviceName`, breaking stable DNS names like `myapp-0.my-service`. - Misconfiguring `podManagementPolicy`, causing unexpected parallel operations when ordered startup is required. - Using `RollingUpdate` with incorrect `partition` values, updating too many replicas at once and risking quorum loss. - Ignoring `volumeClaimTemplates`, leading to shared storage or no persistence for stateful data. - Setting resource limits too low, causing `OOMKilled` during startup or normal operation. To avoid these pitfalls, adopt a scoped implementation path. Instead of updating all replicas simultaneously, use a rolling update with a `partition` to create a canary. For example, with three replicas (`myapp-0`, `myapp-1`, `myapp-2`), set `partition: 2`. This means only pods with an ordinal greater than or equal to 2 (i.e., `myapp-2`) will be updated. The other two pods remain on the previous version, allowing you to validate the change on a single replica before rolling it out further. Here is a complete StatefulSet manifest with safe defaults: ```yaml apiVersion: apps/v1 kind: StatefulSet metadata: name: my-statefulset spec: serviceName: "my-service" replicas: 3 podManagementPolicy: OrderedReady updateStrategy: type: RollingUpdate rollingUpdate: partition: 2 # Only update pod with ordinal >= 2 (myapp-2) volumeClaimTemplates: - metadata: name: data spec: accessModes: [ "ReadWriteOnce" ] storageClassName: standard resources: requests: storage: 1Gi template: spec: containers: - name: app image: myapp:1.0 ports: - containerPort: 8080 resources: requests: memory: "512Mi" cpu: "250m" limits: memory: "1Gi" cpu: "500m" ``` Apply changes incrementally and verify each step. Use `kubectl apply` with the modified manifest: ```bash kubectl apply -f updated-statefulset.yaml ``` After applying, monitor the rollout status: ```bash kubectl rollout status statefulset/my-statefulset ``` Expected output during a canary update: ``` Waiting for 1 pods to be ready... Waiting for 1 pod to be ready... statefulset rolling update complete 1 pods at revision my-statefulset-7d9f9d6f7 ``` If the rollout stalls, inspect pod events and logs before proceeding. Do not force the rollout; investigate the cause first. ## Verification and Diagnostics After applying configuration changes, verify the health of each replica. For replicated stateful applications, consistency and data integrity are critical. Use the following commands and checks. **Check pod status and readiness:** ```bash kubectl get pods -l app=myapp -o wide ``` Expected output: ``` NAME READY STATUS RESTARTS AGE IP NODE myapp-0 1/1 Running 0 10m 10.0.0.1 node1 myapp-1 1/1 Running 0 10m 10.0.0.2 node2 myapp-2 1/1 Running 0 10m 10.0.0.3 node3 ``` **Check PVC bound to each pod:** ```bash kubectl get pvc -l app=myapp ``` Expected output: ``` NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE data-myapp-0 Bound pvc-0d1f7c42-1a5a-4f4d-9a5b-0f0a5e5f0a5a 1Gi RWO standard 10m data-myapp-1 Bound pvc-5d7e3e1f-8b3c-4c1d-8d0a-2b1e2f1d2e1f 1Gi RWO standard 10m data-myapp-2 Bound pvc-9f8e7c6d-2c3b-4d5e-6f7a-3c4d5e6f7a8b 1Gi RWO standard 10m ``` **Check logs for replication health:** ```bash kubectl logs myapp-0 ``` Example output for a database replica: ``` 2024-01-01T12:00:00Z [Replication] INFO: Successfully connected to primary 2024-01-01T12:00:01Z [Replication] INFO: Replication lag: 0 ms ``` Diagnostic checks to perform: - Ensure each pod has a unique persistent volume. Using `ReadWriteOnce` prevents multiple pods from mounting the same volume. - Verify replication status via application metrics or logs. For databases, check `SHOW SLAVE STATUS` or equivalent. - Run a write test on one replica and read from another, if the application supports it. For example, insert a row in `myapp-0` and query it from `myapp-1` to confirm replication. - Use `kubectl exec` to run application-specific health commands. For a PostgreSQL StatefulSet, you might run: ```bash kubectl exec myapp-0 -- psql -c "SELECT 1" ``` If any check fails, pause further rollout and investigate. ## Failure Modes and Recovery Despite precautions, failures occur. Common failure modes include: - **Pod stuck in `Pending`** due to insufficient resources, storage class issues, or node affinity constraints. - **Pod crash loop** caused by misconfiguration, missing dependencies, or incompatible image version. - **Data inconsistency** after an unclean shutdown, such as a node failure without proper fsync. - **Split-brain** when network partitions isolate replicas and both sides attempt to become primary. Recovery steps: **1. Diagnose the failure.** If a rollout fails, check events and pod details: ```bash kubectl describe pod myapp-1 ``` Look for messages like `Failed to attach volume`, `Insufficient cpu`, or `CrashLoopBackOff`. Also check logs: ```bash kubectl logs myapp-1 --previous ``` **2. Rollback to previous version.** If you have the previous manifest in Git or used Helm, execute: ```bash kubectl rollout undo statefulset/my-statefulset ``` This reverts to the previous revision. Verify the rollback completes: ```bash kubectl rollout status statefulset/my-statefulset ``` If using Helm, you can rollback a release: ```bash helm rollback my-release 2 ``` **3. Restore from backup for data issues.** Ensure backups are tested regularly. For cloud volumes, use snapshots; for databases, use logical dumps (e.g., `pg_dump`). **4. Replace a failed pod if necessary.** If a pod is stuck or corrupted, you can delete it; the StatefulSet controller will recreate it with the same identity and PVC. Use this cautiously: ```bash kubectl delete pod myapp-1 ``` The pod will be recreated, but its PVC remains. Note: this does not roll back configuration; it only reschedules the pod. Always document the incident and the recovery steps for future reference. Regularly test rollback procedures in a non-production environment to ensure they work when needed. ## Operations Checklist Use this checklist before and after making configuration changes to replicated stateful applications. It enforces consistency and reduces mistakes. | Step | Description | Command / Action | |------|-------------|------------------| | 1 | Record baseline versions | `kubectl version --short` | | 2 | List StatefulSet details | `kubectl get statefulset -o yaml` | | 3 | Backup persistent volumes | Use storage provider snapshot or tool (e.g., Velero) | | 4 | Apply change with partition for canary | `kubectl apply -f updated-statefulset.yaml` with `partition: 2` | | 5 | Monitor rollout status | `kubectl rollout status statefulset/my-statefulset` | | 6 | Verify pod readiness | `kubectl get pods -l app=myapp` | | 7 | Check application logs for replication health | `kubectl logs myapp-0` | | 8 | Run application-specific tests | For database, run query test (e.g., write to primary, read from replica) | | 9 | If failure, rollback | `kubectl rollout undo statefulset/my-statefulset` | | 10 | Document outcome and update runbook | Update internal wiki or Git repo | ## Conclusion Running replicated stateful applications on Kubernetes demands careful configuration and disciplined operations. By understanding common mistakes, using scoped updates with partitions, verifying with diagnostics, and having a rollback plan, you can maintain high availability and data integrity. Start with a narrow pilot, measure outcomes, and iterate. The commands and checklist provided give you a solid foundation for operating stateful workloads reliably. Remember that stateful applications require extra care: every change should be treated as potentially disruptive, and every failure should be analyzed to prevent recurrence.