Learn how to safely upgrade and migrate Kubernetes ReplicaSets with step-by-step commands, validation checks, rollback strategies, and an operational checklist to minimize risk. ## Intro ReplicaSets are a fundamental building block in Kubernetes. They ensure that a specified number of pod replicas are running at any given time, providing basic availability and self-healing for stateless applications. However, ReplicaSets do not support rolling updates natively. To change the pod template (for example, to update the container image), you must create a new ReplicaSet and manually migrate traffic. This process requires careful planning to avoid downtime, data loss, or service disruption. This guide provides a practical, hands-on approach to upgrading and migrating ReplicaSets. It covers environment inventory, configuration, verification, failure recovery, and includes a complete operational checklist. Whether you are a developer, DevOps consultant, or part of a technical startup, you will learn how to execute ReplicaSet changes with confidence and minimal risk. While Deployments are the recommended way to manage ReplicaSets in production because they add rolling update and rollback capabilities, many teams still run bare ReplicaSets for legacy or specialized workloads. The techniques in this guide apply to both bare ReplicaSets and those managed by Deployments, but we focus on the manual process required when you operate a ReplicaSet directly. ## Version and Environment Inventory Before any upgrade, you must establish a clear picture of your current environment. This inventory helps identify dependencies, compatibility issues, and potential risks. Skipping this step is a common cause of failed migrations. ### Kubernetes Version Check Verify the control plane and node versions. The target ReplicaSet API version must be compatible with your cluster. The apps/v1 API is stable since Kubernetes 1.9, but it is good practice to confirm. Run: kubectl version --short Example output: Client Version: v1.28.2 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.27.3 Ensure the server version is compatible with the features you plan to use. For example, if you plan to use ephemeral containers for debugging, you need Kubernetes 1.23 or later. ### ReplicaSet Details List all ReplicaSets across all namespaces to get an overview: kubectl get replicasets --all-namespaces For a specific ReplicaSet, inspect its full YAML definition: kubectl get replicaset my-replicaset -n my-namespace -o yaml Key fields to note: - spec.replicas : the desired number of pods. - spec.selector : the label selector used to identify pods owned by this ReplicaSet. - spec.template : the pod template, including container images, resources, and other settings. Save the output to a file as a backup. You will need it for rollback or reference. kubectl get replicaset my-replicaset -n my-namespace -o yaml > replicaset-backup.yaml ### Pod Health Check the health of existing pods managed by the ReplicaSet: kubectl get pods -l app=my-app -n my-namespace All pods should be in Running state and ready. If any pods are in CrashLoopBackOff , Pending , or Error , resolve those issues before attempting an upgrade. Use describe and logs to diagnose: kubectl describe pod -n my-namespace kubectl logs -n my-namespace ### Topology and Dependencies Identify which Services route traffic to these pods and whether any Horizontal Pod Autoscalers (HPAs) reference the ReplicaSet. Services select pods using labels, so you need to know the current selector. Run: kubectl get svc, hpa -n my-namespace -o wide Inspect the Service YAML to see the selector: kubectl get svc my-service -n my-namespace -o yaml Example Service selector: spec: selector: app: my-app version: v1 This selector means the Service routes traffic only to pods with labels app=my-app and version=v1 . When you create a new ReplicaSet, you must ensure its pods have a label that matches the desired selector, or you will need to update the Service. Also check for any Ingress resources, NetworkPolicies, or other controllers that might rely on pod labels. ### Prerequisites Checklist Before you start, ensure you have: - kubectl configured with appropriate permissions to create, update, and delete ReplicaSets and Services in the target namespace. - Access to the manifests (e.g., in a Git repository) or the ability to generate them. - A backup of the current ReplicaSet definition (as shown above). - A staging or test environment that mirrors production as closely as possible. If you don't have a staging environment, consider testing in an isolated namespace. - A rollback plan documented and communicated to the team.

Quick check 1 of 2 What is the recommended way to manage ReplicaSets for production workloads according to the article? - Using bare ReplicaSets directly - Using Deployments instead of directly using ReplicaSets - Using StatefulSets for all workloads - Using ReplicationControllers

The article states that Deployments are the recommended way to manage ReplicaSets in production because they add rolling update and rollback capabilities. It also notes that using Deployments is recommended unless you require custom update orchestration or don't require updates at all.

## Safe Configuration Path Adopt a gradual approach: modify the ReplicaSet in a controlled manner and validate before full rollout. The key principle is to never modify the existing ReplicaSet in place for a major change. Instead, create a new ReplicaSet with the desired changes, validate it, and then shift traffic. ### Step 1: Extract Current Manifest Export the current ReplicaSet definition as a baseline. You already did this in the inventory step, but ensure you have the latest version: kubectl get replicaset my-replicaset -n my-namespace -o yaml > replicaset-current.yaml ### Step 2: Create a New ReplicaSet with Desired Changes Create a new ReplicaSet YAML file based on the current one, but with a new name and updated pod template. For example, if you are upgrading the container image from myapp:1.0 to myapp:2.0 , and you want to add a new label version: v2 to distinguish pods, the new manifest might look like this: apiVersion: apps/v1 kind: ReplicaSet metadata: name: my-replicaset-v2 namespace: my-namespace labels: app: my-app version: v2 spec: replicas: 3 selector: matchLabels: app: my-app version: v2 template: metadata: labels: app: my-app version: v2 spec: containers: - name: app-container image: myapp:2.0 ports: - containerPort: 80 resources: requests: cpu: "100m" memory: "128Mi" limits: cpu: "200m" memory: "256Mi" Important details: - The ReplicaSet name is unique: my-replicaset-v2 . - The selector matchLabels must match the pod template labels. In this example, it is app: my-app and version: v2 . This is different from the old ReplicaSet's selector, which might only have app: my-app or include version: v1 . - The container image is updated to myapp:2.0 . - Resource requests and limits are set explicitly. This is a good practice to ensure predictable scheduling. Apply the new ReplicaSet: kubectl apply -f replicaset-v2.yaml Verify it is created: kubectl get replicaset my-replicaset-v2 -n my-namespace ### Step 3: Validate New ReplicaSet in Isolation The new ReplicaSet's pods have the label version=v2 , while the old pods have version=v1 (if you had that label before). Since the existing Service selector likely still points to version=v1 , traffic continues to flow to old pods. This is intentional: the new pods are running but not receiving production traffic. Wait for the new ReplicaSet's pods to become ready: kubectl get pods -l version=v2 -n my-namespace All pods should show READY 1/1 and STATUS Running . To test the new pods without affecting production traffic, you can use port-forwarding directly to the ReplicaSet or to a specific pod: kubectl port-forward rs/my-replicaset-v2 8080:80 Then open http://localhost:8080 in your browser or use curl from another terminal: curl http://localhost:8080 Additionally, you can create a temporary Service with a selector that only matches version=v2 to test connectivity internally. For example, save the following as test-service-v2.yaml : apiVersion: v1 kind: Service metadata: name: my-service-v2-test namespace: my-namespace spec: selector: app: my-app version: v2 ports: - port: 80 targetPort: 80 Apply it: kubectl apply -f test-service-v2.yaml Then run a temporary pod to test the service: kubectl run test-pod --rm -it --image=busybox --restart=Never -- /bin/sh Inside the pod, use wget or curl (if available) to access the service: wget -qO- http://my-service-v2-test You should see the response from the new version. After testing, delete the temporary service and pod: kubectl delete svc my-service-v2-test -n my-namespace ### Step 4: Shift Traffic Gradually Once you are confident the new version works correctly, you need to update the production Service to route traffic to the new pods. There are two common approaches: - Direct switch : Update the Service selector to match only version=v2 . This immediately moves all traffic to the new pods. - Canary release : Use a more sophisticated mechanism like an Ingress controller with traffic splitting (e.g., NGINX Ingress with canary annotations) or a service mesh like Istio or Linkerd. This allows you to send a small percentage of traffic to the new version and gradually increase it. For simplicity and because this guide focuses on ReplicaSets, we will demonstrate a direct switch. However, if you have the tooling, a canary approach reduces risk further. To perform a direct switch, edit the Service: kubectl edit svc my-service -n my-namespace Change the selector in the editor from: selector: app: my-app version: v1 to: selector: app: my-app version: v2 Save and exit. The Service will now route traffic to pods with version=v2 . You can also use kubectl patch for a non-interactive update: kubectl patch svc my-service -n my-namespace -p '{"spec":{"selector":{"version":"v2"}}}' Verify the Service endpoints: kubectl get endpoints my-service -n my-namespace You should see the IP addresses of the new pods. ### Step 5: Scale Down Old ReplicaSet After verifying that the new version is handling traffic correctly, scale down the old ReplicaSet to zero: kubectl scale replicaset my-replicaset --replicas=0 -n my-namespace Do not delete the old ReplicaSet immediately. Keep it for a rollback window (for example, 24 hours). If something goes wrong with the new version, you can quickly scale the old ReplicaSet back up and switch the Service selector back to version=v1 . After the rollback window passes and you are confident, delete the old ReplicaSet: kubectl delete replicaset my-replicaset -n my-namespace ### Step 6: Update References If you use an HPA that targets the ReplicaSet, update it to reference the new ReplicaSet. Note that HPAs typically target Deployments, but they can be configured to target ReplicaSets using the scaleTargetRef with kind: ReplicaSet . For example: apiVersion: autoscaling/v1 kind: HorizontalPodAutoscaler metadata: name: my-app-hpa namespace: my-namespace spec: scaleTargetRef: apiVersion: apps/v1 kind: ReplicaSet name: my-replicaset-v2 minReplicas: 2 maxReplicas: 10 targetCPUUtilizationPercentage: 80 Apply the updated HPA: kubectl apply -f hpa-v2.yaml ### Scoped Implementation Choices - Use namespaces to isolate changes, especially if you are testing. - Consider using Deployments instead of bare ReplicaSets for future workloads. Deployments manage ReplicaSets and provide rolling updates and rollbacks automatically. - Always keep the old ReplicaSet definition and its YAML backup for rollback. - Use a version label (like version: v1 and version: v2 ) to distinguish pods. This is essential for safe traffic shifting. ## Verification and Diagnostics After applying changes, you must verify that the upgrade or migration is successful and the application behaves correctly. Perform these checks systematically. ### Check ReplicaSet Status kubectl get rs my-replicaset-v2 -n my-namespace Expected output: NAME DESIRED CURRENT READY AGE my-replicaset-v2 3 3 3 5m The DESIRED , CURRENT , and READY columns should all match the number of replicas you specified. If they don't, investigate the pods. ### Check Pod Readiness kubectl get pods -l version=v2 -n my-namespace All pods should show READY 1/1 and STATUS Running . If any pod is not ready, check its events and logs. ### Inspect Logs View logs from all new pods: kubectl logs -l version=v2 -n my-namespace --tail=50 Look for error messages, stack traces, or connection issues. If your application logs startup information, verify it started with the correct version. ### Test Service Connectivity Use a temporary pod to test the Service from inside the cluster: kubectl run test-pod --rm -it --image=busybox --restart=Never -- /bin/sh Inside the pod, use wget or curl (if installed) to access the Service: wget -qO- http://my-service Or, if curl is available: curl http://my-service Expect the application response. If you get a connection refused or timeout, the Service may not be routing correctly. Check the Service endpoints and pod readiness. ### Check Events for Errors kubectl describe rs my-replicaset-v2 -n my-namespace Look for events such as Created pod , Scaled up replica set , or any warnings. Warnings like FailedScheduling or Failed to pull image indicate problems. ### Monitor Resource Usage If you have the metrics server installed, you can view CPU and memory usage of the new pods: kubectl top pods -l version=v2 -n my-namespace Compare resource usage with requests and limits. High CPU or memory usage might indicate a performance issue with the new version. ### Automated Checks Consideration For production environments, consider scripting these checks to run automatically after deployment. A simple shell script or a CI/CD pipeline step can execute the kubectl commands and fail the pipeline if any check fails. This reduces human error and speeds up the process. ## Failure Modes and Recovery Despite careful planning, failures can occur. Here are common failure modes, their causes, diagnostics, and recovery steps. ### Failure: New Pods CrashLoopBackOff Cause : Misconfiguration, incompatible image, missing environment variables, or application bug. Diagnostics : kubectl get pods -l version=v2 -n my-namespace kubectl logs -n my-namespace kubectl describe pod -n my-namespace Example: You see pods in CrashLoopBackOff state. The logs show Error: Cannot find module 'express' indicating a missing dependency in the image. Recovery : - Revert traffic to the old ReplicaSet by updating the Service selector back to version=v1 (if the old ReplicaSet still exists and its replicas are not zero). If you scaled the old ReplicaSet to zero, scale it back up first: kubectl scale replicaset my-replicaset --replicas=3 -n my-namespace kubectl patch svc my-service -n my-namespace -p '{"spec":{"selector":{"version":"v1"}}}' - Delete or scale down the new ReplicaSet to prevent resource usage: kubectl scale replicaset my-replicaset-v2 --replicas=0 -n my-namespace - Fix the configuration or image and plan the next attempt. ### Failure: New Pods Not Scheduled Cause : Insufficient cluster resources, node selector constraints, taints/tolerations, or persistent volume claims not available. Diagnostics : kubectl describe rs my-replicaset-v2 -n my-namespace Look for events like FailedScheduling with messages such as 0/3 nodes are available: 3 Insufficient cpu. Recovery : - Adjust resource requests and limits to fit available capacity. - Modify node selectors or add tolerations if needed. - Scale down the old ReplicaSet to free resources (after traffic is shifted or if you are still testing): kubectl scale replicaset my-replicaset --replicas=1 -n my-namespace Then recreate or scale the new ReplicaSet. ### Failure: Service Traffic Interrupted Cause : Service selector change too abrupt, misconfigured selector, or a bug in the new version that causes poor performance or crashes. Recovery : Immediately revert the Service selector to the previous value: kubectl patch svc my-service -n my-namespace -p '{"spec":{"selector":{"version":"v1"}}}' If the old ReplicaSet was scaled to zero, scale it back up first. Then investigate the issue with the new version. ### Failure: Data Loss or Inconsistency Cause : Application not designed for zero-downtime migration, database schema changes not backward compatible, or version incompatibility. Recovery : - Restore from backups if data was lost. - Consider running old and new versions in parallel with a canary approach to detect inconsistencies early. - If possible, use a blue-green deployment strategy with a database migration tool that supports backward-compatible changes. ### General Rollback Steps - Identify the issue (from logs, metrics, user reports). - Restore the Service selector to the old version. - Scale the old ReplicaSet back to desired replicas if it was scaled down. - Delete or scale down the new ReplicaSet. - Fix the configuration and plan the next attempt. ### Preventive Measures - Always keep the old ReplicaSet until the new one is proven stable. - Use a version label to distinguish pods. - Test in staging first, ideally with production-like traffic. - Automate deployment and rollback to reduce human error. - Monitor application metrics and logs continuously during the migration. ## Operations Checklist Use this checklist for every ReplicaSet upgrade or migration. Fill in the specific names and commands for your environment.

Step	Action	Command / Evidence
1	Verify cluster access and version	`kubectl cluster-info`
2	Inventory existing ReplicaSets	`kubectl get rs --all-namespaces`
3	Backup current ReplicaSet manifest	`kubectl get rs <name> -n <ns> -o yaml > backup.yaml`
4	Identify service selectors and dependencies	`kubectl get svc -n <ns> -o yaml`
5	Create new ReplicaSet with updated template	`kubectl apply -f new-rs.yaml`
6	Validate new pods are running and ready	`kubectl get pods -l version=v2 -n <ns>`
7	Test new pods via port-forward or temp service	`kubectl port-forward rs/new-rs 8080:80`
8	Switch service traffic to new version	`kubectl edit svc <svc> -n <ns>`
9	Monitor application logs and metrics	`kubectl logs -l version=v2 -n <ns>`
10	Scale down old ReplicaSet	`kubectl scale rs <old-rs> --replicas=0 -n <ns>`
11	Keep old ReplicaSet for rollback window (e.g., 24h)	-
12	Delete old ReplicaSet after stable period	`kubectl delete rs <old-rs> -n <ns>`
13	Update documentation and runbooks	-

### Review Points - Confirm the rollback plan before starting. - Communicate with team members about the change window. - Perform the upgrade during a maintenance window if possible, especially for high-traffic services. - Have on-call engineer available during and immediately after the migration. ## Conclusion Upgrading and migrating Kubernetes ReplicaSets can be done safely with careful planning and incremental steps. By inventorying your environment, creating new ReplicaSets alongside old ones, validating in isolation, and having a solid rollback plan, you minimize risk. Use the provided operational checklist to standardize the process. Remember to keep old ReplicaSets until you are certain the new version is stable, and consider using Deployments for future workloads to take advantage of built-in rolling updates. With these practices, you can confidently manage ReplicaSet lifecycle changes in production.