Kubernetes Run Replicated Stateful Application: Production Operations Checklist with Practical Examples

Intro

Running a replicated stateful application on Kubernetes in production requires more than deploying a StatefulSet and hoping for the best. Stateful workloads—databases, message brokers, distributed caches, and similar systems—have strict requirements around identity, storage, ordering, and graceful lifecycle management. When things go wrong, operators need a clear, repeatable process to diagnose and recover without causing further disruption.

This article provides a production operations checklist tailored for replicated stateful applications on Kubernetes. It is written for developers, DevOps consultants, and technical startup teams who are responsible for keeping these systems healthy. The checklist connects Kubernetes operations, best practices, and maintenance to concrete commands, expected outputs, failure signals, and recovery decisions.

The core principle is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached. Each section below addresses a specific operational area—version and environment inventory, safe configuration, verification and diagnostics, failure modes and recovery, and a consolidated operations checklist.

Version and Environment Inventory

Before touching a running stateful application, you must know exactly what you are dealing with. A version and environment inventory establishes the baseline: the installed version, deployment topology, prerequisites, and the exact components being inspected. This prevents mismatched assumptions and reduces the risk of applying a fix intended for a different version or configuration.

For a replicated stateful application running as a Kubernetes StatefulSet, the inventory should capture:

• Kubernetes cluster version and API server compatibility
• StatefulSet manifest version and current replica count
• Container images and their tags
• PersistentVolumeClaim (PVC) status, storage class, and capacity
• Network policies, pod disruption budgets, and any custom resources that affect the workload

Start with read-only observations. Capture current state and timestamps first, protect credentials and private material, then change one scoped item only when its blast radius and recovery path are understood.

Practical Kubernetes commands for the inventory:

# Cluster version
kubectl version --short

# StatefulSet overview
kubectl get statefulset <statefulset-name> -o wide

# Pods with IPs and nodes
kubectl get pods -l app=<app-label> -o wide

# Persistent volume claims
kubectl get pvc -l app=<app-label>

# Events for the namespace (recent)
kubectl get events --sort-by=.metadata.creationTimestamp | tail -20

Example output for a healthy three-replica StatefulSet named postgres:

NAME       READY   AGE   CONTAINERS   IMAGES
postgres   3/3     42d   postgres     postgres:14.5

NAME        READY   STATUS    RESTARTS   AGE   IP            NODE
postgres-0  1/1     Running   0          42d   10.244.1.10   worker-1
postgres-1  1/1     Running   0          42d   10.244.2.20   worker-2
postgres-2  1/1     Running   0          42d   10.244.3.30   worker-3

NAME                            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-postgres-0                 Bound    pvc-8a2b3c4d-...                           20Gi       RWO            standard       42d
data-postgres-1                 Bound    pvc-9b3c4d5e-...                           20Gi       RWO            standard       42d
data-postgres-2                 Bound    pvc-0c4d5e6f-...                           20Gi       RWO            standard       42d

If any of these commands fail or show unexpected versions, investigate before proceeding. For example, a pod in CrashLoopBackOff status requires log inspection:

# Current logs
kubectl logs postgres-0

# Previous container logs (if restarted)
kubectl logs postgres-0 --previous

Keep the local test small. Apply one manifest, inspect the generated resources, and verify traffic with kubectl port-forward or a local service type before moving to a cloud load balancer or ingress controller. For example, after a version upgrade test:

# Port forward to local to test connectivity
kubectl port-forward svc/postgres 5432:5432

# In another terminal
psql -h localhost -U postgres -c 'SELECT version();'

If the output matches the expected new version, the upgrade path is likely sound. If not, roll back before applying to production.

Safe Configuration Path

Configuration changes are a common source of incidents in stateful applications. A safe configuration path means you know what you are changing, why, what the expected outcome is, and how to revert if needed. For StatefulSets, certain fields are immutable after creation (for example, spec.volumeClaimTemplates and spec.serviceName). Attempting to change them will fail; sometimes you must delete and recreate the StatefulSet while preserving data, which requires careful planning.

The safe configuration path consists of:

1. Read-only observation: Check current configuration with kubectl get statefulset <name> -o yaml and compare against a known good baseline stored in version control.
2. Smallest justified change: Modify one field at a time, preferably in a staging environment first. Use kubectl diff to preview changes.
3. Blast radius assessment: Determine which pods will be affected. For a StatefulSet, changes to the pod template will trigger a rolling update only if the update strategy is RollingUpdate. With OnDelete, no pods are updated until manually deleted.
4. Verification: After applying, check pod status, logs, and application health.

Commands for safe configuration:

# Preview changes before applying
kubectl diff -f updated-statefulset.yaml

# Apply the change
kubectl apply -f updated-statefulset.yaml

# Watch the rollout status
kubectl rollout status statefulset/<name>

# If using OnDelete strategy, trigger update pod by pod
kubectl delete pod <name>-0

Example of a controlled resource update. Suppose you need to increase CPU requests for a database container. The diff shows:

--- a/statefulset.yaml
+++ b/statefulset.yaml
@@ -42,7 +42,7 @@ spec:
         resources:
           requests:
-            cpu: "500m"
+            cpu: "750m"
             memory: "1Gi"

Apply and then watch the pods restart in order. For a RollingUpdate StatefulSet, pods are updated in reverse ordinal order (highest index first). You can verify with:

kubectl get pods -w

The pod names postgres-2, then postgres-1, then postgres-0 will transition from Running to Terminating to ContainerCreating to Running again. If a pod gets stuck in Pending or CrashLoopBackOff, roll back the change immediately.

Critical safety points for stateful configuration:

• Never change volumeClaimTemplates on a live StatefulSet; it is immutable. Instead, create a new StatefulSet and migrate data, or expand the volume using your storage provider's expansion feature if supported.
• Always set updateStrategy explicitly. The default is RollingUpdate with partition: 0, but for databases you may want OnDelete to control when each replica restarts.
• Protect secrets: do not put passwords or keys in plain text manifests. Use Kubernetes Secrets, external secrets operators, or cloud provider secret stores. Reference them as environment variables or mounted volumes.

Example of a secure secret usage:

apiVersion: v1
kind: Secret
metadata:
  name: postgres-secret
type: Opaque
stringData:
  POSTGRES_PASSWORD: "s3cr3t-example"
---
apiVersion: apps/v1
kind: StatefulSet
spec:
  template:
    spec:
      containers:
      - name: postgres
        env:
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: postgres-secret
              key: POSTGRES_PASSWORD

Never log or echo the secret value. Use kubectl get secret postgres-secret -o yaml only in a controlled environment, and redact output in documentation.

Verification and Diagnostics

Verification means confirming that the system is behaving as expected after a change or during normal operations. Diagnostics involves identifying the root cause when it is not. For stateful applications, health is more than pod status; you must check application-level readiness, data consistency, and replication status.

A layered verification approach:

Layer 1: Kubernetes object status

kubectl get statefulset <name>
kubectl get pods -l app=<label>
kubectl get pvc -l app=<label>

All pods should be Running with READY 1/1 and zero restarts (or a low, stable restart count). The StatefulSet should show READY equal to the desired replicas.

Layer 2: Container logs

kubectl logs <pod-name> --tail=50
kubectl logs <pod-name> --previous

Look for error patterns: connection refused, disk full, replication lag, etc. Example problematic log from PostgreSQL:

FATAL:  could not write to file "pg_wal/xlogtemp.123": No space left on device

This indicates the PVC is full. Immediate action: increase disk or clean up old WAL files.

Layer 3: Application health checks

Use kubectl exec to run application-specific commands:

# For PostgreSQL
kubectl exec -it postgres-0 -- psql -U postgres -c "SELECT 1;"

# For a generic HTTP endpoint
kubectl exec -it <pod> -- curl -f http://localhost:8080/health

Layer 4: Network and service verification

kubectl get endpoints <service-name>

The endpoints should list the pod IPs. If empty, the service selector does not match pod labels. Test connectivity from another pod:

kubectl run -it --rm debug --image=busybox -- sh
# Inside the debug pod
wget -qO- http://<service-name>:<port>/health

Example diagnostic scenario: a three-replica PostgreSQL StatefulSet shows postgres-2 in CrashLoopBackOff. Steps:

1. kubectl describe pod postgres-2 shows events:

Warning  BackOff  2m (x12 over 3m)  kubelet  Back-off restarting failed container

1. kubectl logs postgres-2 --previous shows:

FATAL:  could not connect to the primary server: connection refused

1. Check the primary pod postgres-0:

kubectl logs postgres-0

If postgres-0 shows recovery or is not accepting connections, the issue may be a split-brain or failed failover. Investigate the replication configuration.

For every diagnostic, record the command, output, and timestamp. This creates an audit trail for post-incident reviews.

Failure Modes and Recovery

Stateful applications can fail in many ways. Understanding common failure modes and having predefined recovery steps reduces downtime and panic. Here we outline several failure modes relevant to replicated stateful workloads on Kubernetes, along with diagnostic signals and recovery actions.

1. Pod CrashLoopBackOff

Signal: kubectl get pods shows CrashLoopBackOff or Error status; kubectl describe pod shows multiple restarts.

Common causes:

• Misconfiguration (wrong environment variable, missing secret)
• Application bug
• Resource limits too low (OOMKilled)
• Corrupted data on persistent volume

Diagnosis:

kubectl describe pod <pod-name> | grep -A 10 "Last State"
kubectl logs <pod-name> --previous

Recovery:

• Fix the configuration or code and apply the change.
• If OOMKilled, increase memory limit or reduce application memory usage.
• If data corruption, restore from backup or snapshot.

2. Persistent Volume Full or Unavailable

Signal: Pod logs show "No space left on device"; kubectl get pvc shows capacity reached; pod may be stuck in ContainerCreating if volume cannot mount.

Diagnosis:

kubectl get pvc
kubectl describe pvc <pvc-name>
kubectl exec <pod> -- df -h

Recovery:

• Expand the PVC if the storage class supports it: kubectl patch pvc <pvc-name> -p '{"spec":{"resources":{"requests":{"storage":"40Gi"}}}}'.
• Delete unneeded data within the application (e.g., old logs, temporary files).
• If volume is unavailable due to node failure, the pod may be stuck; force delete the pod and let it reschedule on a healthy node (but be careful with StatefulSet identity).

3. Replica Set Split Brain

Signal: Multiple pods claim to be primary; clients get inconsistent data; logs show replication conflicts.

Diagnosis:

• Check application-specific primary election status (e.g., patronictl list for Patroni-managed PostgreSQL).
• kubectl get pods -o wide to see if pods are spread across nodes incorrectly.

Recovery:

• Use the application's failover mechanism to force a primary.
• For StatefulSets, ensure pod management policy and network policies do not allow unintended access.
• Never delete the pod that holds the primary data volume without a backup.

4. StatefulSet Update Stuck

Signal: kubectl rollout status statefulset/<name> hangs; some pods are updated but others remain old.

Diagnosis:

kubectl get pods
kubectl describe pod <stuck-pod>

Check events for image pull errors, scheduling failures, or readiness probe failures.

Recovery:

• If image pull error, fix image name or credentials.
• If readiness probe fails, inspect application logs; the new version may not be compatible with existing data.
• Rollback: kubectl rollout undo statefulset/<name>

5. Node Failure

Signal: Pods on a node go Unknown or Terminating; node marked NotReady.

Diagnosis:

kubectl get nodes
kubectl describe node <node-name>

Recovery:

• Wait for Kubernetes to reschedule pods onto healthy nodes. StatefulSet pods with persistent volumes will be recreated on nodes that can mount the volume.
• If a pod is stuck terminating, force delete it: kubectl delete pod <pod-name> --grace-period=0 --force.
• Ensure the storage class supports multi-attach or that the volume is released from the failed node.

For all recovery actions, document the incident and the steps taken. After recovery, verify data integrity and replication status.

Operations Checklist

This section consolidates the key operational checks into a single checklist that can be used during regular maintenance, deployments, or incident response. The checklist is organized by operational phase.

Pre-Deployment Checklist

• [ ] Verify Kubernetes version compatibility with the application's supported versions.
• [ ] Review StatefulSet manifest for immutability constraints (e.g., volumeClaimTemplates).
• [ ] Ensure persistent volume claims are created and bound.
• [ ] Set resource requests and limits; avoid overly tight limits for stateful apps.
• [ ] Define pod disruption budget (PDB) to protect quorum: kubectl get pdb
• [ ] Configure liveness and readiness probes appropriately (do not overload with heavy checks).
• [ ] Test backup and restore procedures in a non-production environment.

Deployment / Update Checklist

• [ ] Use kubectl diff to preview changes.
• [ ] Apply changes with kubectl apply -f.
• [ ] Monitor rollout with kubectl rollout status statefulset/<name>.
• [ ] Watch pod transitions: kubectl get pods -w
• [ ] If using OnDelete, update pods manually in reverse ordinal order.
• [ ] After each pod update, verify application health (e.g., SELECT 1 for database, API health endpoint for custom apps).
• [ ] Check replication status; ensure replicas are in sync before proceeding.
• [ ] If any pod fails, pause and rollback: kubectl rollout undo statefulset/<name>.

Daily / Weekly Health Checks

• [ ] kubectl get statefulset <name> - ensure ready replicas equal desired.
• [ ] kubectl get pods -l app=<label> - all Running, low restart counts.
• [ ] kubectl get pvc - capacity usage below threshold (e.g., 80%).
• [ ] Check application logs for errors: kubectl logs <pod> --tail=100
• [ ] Monitor metrics: CPU, memory, disk I/O, replication lag.
• [ ] Review events for warnings: kubectl get events --field-selector type=Warning
• [ ] Verify backup jobs completed successfully.

Incident Response Checklist

• [ ] Identify the scope: which pods, nodes, services are affected? Use kubectl get pods -o wide and kubectl get events.
• [ ] Preserve evidence: collect logs, describe output, and current state before making changes.
• [ ] Determine if the issue is application-level or infrastructure-level.
• [ ] Check persistent volumes and storage subsystem.
• [ ] If needed, cordon nodes to isolate: kubectl cordon <node>
• [ ] Apply fix with minimal change.
• [ ] Verify recovery thoroughly before declaring resolved.
• [ ] Document the incident: timeline, root cause, actions taken, preventive measures.

Sample Worked Example: Upgrading a Three-Node CockroachDB Cluster

Assume you have a CockroachDB StatefulSet with three replicas. You want to upgrade from v22.1 to v22.2.

1. Pre-checks:

• Verify cluster health: kubectl exec cockroachdb-0 -- cockroach node status shows all nodes live.
• Check disk space: kubectl exec cockroachdb-0 -- df -h /cockroach/cockroach-data shows 35% used.
• Review upgrade notes: CockroachDB requires sequential minor versions.

1. Update image: Change container image tag from cockroachdb/cockroach:v22.1.10 to cockroachdb/cockroach:v22.2.0 in the StatefulSet manifest. Use kubectl edit statefulset cockroachdb or patch:

kubectl set image statefulset/cockroachdb cockroachdb=cockroachdb/cockroach:v22.2.0

1. Monitor rollout: The StatefulSet uses RollingUpdate with partition: 0. Pods will update from cockroachdb-2 to cockroachdb-0. Watch:

kubectl rollout status statefulset/cockroachdb

1. Verify each pod: After each pod restarts, check that it rejoins the cluster:

kubectl exec cockroachdb-0 -- cockroach node status

All three nodes should show is_live: true.

1. Post-upgrade: Run the recommended migration commands if any (see CockroachDB docs). Then verify cluster version:

kubectl exec cockroachdb-0 -- cockroach sql --execute="SHOW CLUSTER SETTING version;"

Output should show 22.2.

If any pod fails to start, roll back the image and consult logs:

kubectl set image statefulset/cockroachdb cockroachdb=cockroachdb/cockroach:v22.1.10
kubectl logs cockroachdb-2 --previous

This checklist and example demonstrate the practical, step-by-step approach necessary for safe operations of replicated stateful applications on Kubernetes.

Conclusion

A production operations checklist for replicated stateful applications on Kubernetes is only valuable when each recommendation is version-scoped, observable, and reversible where the technology permits. Copying a command without checking prerequisites and expected output is not an operations procedure; it is a gamble.

The key takeaways from this article:

• Inventory first: Always know your versions, topology, and resource state before making changes.
• Safe configuration: Use kubectl diff, understand immutability, and protect secrets.
• Layered verification: Check Kubernetes status, logs, application health, and network connectivity.
• Prepare for failure: Know the common failure modes and have recovery runbooks ready.
• Follow a checklist: Use the consolidated checklist for deployments, routine checks, and incident response.

As a next step, choose one low-risk verification for your stateful application, record the current state, run the documented check, compare the result with the expected signal, and review dependencies such as StatefulSet, PersistentVolume, and PersistentVolumeClaim. For example, run kubectl get statefulset <name> -o yaml and confirm that updateStrategy and podManagementPolicy match your operational intent.

A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. By following the practices in this article, you can run your replicated stateful applications on Kubernetes with greater confidence and resilience.