Kubernetes ReplicaSet Upgrade and Migration: A Practical Implementation Guide

Learn how to safely upgrade and migrate Kubernetes ReplicaSets with step-by-step commands, validation checks, rollback strategies, and an operational checklist to minimize risk.

Intro

ReplicaSets are a fundamental building block in Kubernetes. They ensure that a specified number of pod replicas are running at any given time, providing basic availability and self-healing for stateless applications. However, ReplicaSets do not support rolling updates natively. To change the pod template (for example, to update the container image), you must create a new ReplicaSet and manually migrate traffic. This process requires careful planning to avoid downtime, data loss, or service disruption.

This guide provides a practical, hands-on approach to upgrading and migrating ReplicaSets. It covers environment inventory, configuration, verification, failure recovery, and includes a complete operational checklist. Whether you are a developer, DevOps consultant, or part of a technical startup, you will learn how to execute ReplicaSet changes with confidence and minimal risk.

While Deployments are the recommended way to manage ReplicaSets in production because they add rolling update and rollback capabilities, many teams still run bare ReplicaSets for legacy or specialized workloads. The techniques in this guide apply to both bare ReplicaSets and those managed by Deployments, but we focus on the manual process required when you operate a ReplicaSet directly.

Version and Environment Inventory

Before any upgrade, you must establish a clear picture of your current environment. This inventory helps identify dependencies, compatibility issues, and potential risks. Skipping this step is a common cause of failed migrations.

Kubernetes Version Check

Verify the control plane and node versions. The target ReplicaSet API version must be compatible with your cluster. The apps/v1 API is stable since Kubernetes 1.9, but it is good practice to confirm.

Run:

kubectl version --short

Example output:

Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.27.3

Ensure the server version is compatible with the features you plan to use. For example, if you plan to use ephemeral containers for debugging, you need Kubernetes 1.23 or later.

ReplicaSet Details

List all ReplicaSets across all namespaces to get an overview:

kubectl get replicasets --all-namespaces

For a specific ReplicaSet, inspect its full YAML definition:

kubectl get replicaset my-replicaset -n my-namespace -o yaml

Key fields to note:

• spec.replicas: the desired number of pods.
• spec.selector: the label selector used to identify pods owned by this ReplicaSet.
• spec.template: the pod template, including container images, resources, and other settings.

Save the output to a file as a backup. You will need it for rollback or reference.

kubectl get replicaset my-replicaset -n my-namespace -o yaml > replicaset-backup.yaml

Pod Health

Check the health of existing pods managed by the ReplicaSet:

kubectl get pods -l app=my-app -n my-namespace

All pods should be in Running state and ready. If any pods are in CrashLoopBackOff, Pending, or Error, resolve those issues before attempting an upgrade. Use describe and logs to diagnose:

kubectl describe pod <pod-name> -n my-namespace
kubectl logs <pod-name> -n my-namespace

Topology and Dependencies

Identify which Services route traffic to these pods and whether any Horizontal Pod Autoscalers (HPAs) reference the ReplicaSet. Services select pods using labels, so you need to know the current selector. Run:

kubectl get svc, hpa -n my-namespace -o wide

Inspect the Service YAML to see the selector:

kubectl get svc my-service -n my-namespace -o yaml

Example Service selector:

spec:
  selector:
    app: my-app
    version: v1

This selector means the Service routes traffic only to pods with labels app=my-app and version=v1. When you create a new ReplicaSet, you must ensure its pods have a label that matches the desired selector, or you will need to update the Service.

Also check for any Ingress resources, NetworkPolicies, or other controllers that might rely on pod labels.

Prerequisites Checklist

Before you start, ensure you have:

• kubectl configured with appropriate permissions to create, update, and delete ReplicaSets and Services in the target namespace.
• Access to the manifests (e.g., in a Git repository) or the ability to generate them.
• A backup of the current ReplicaSet definition (as shown above).
• A staging or test environment that mirrors production as closely as possible. If you don't have a staging environment, consider testing in an isolated namespace.
• A rollback plan documented and communicated to the team.

Safe Configuration Path

Adopt a gradual approach: modify the ReplicaSet in a controlled manner and validate before full rollout. The key principle is to never modify the existing ReplicaSet in place for a major change. Instead, create a new ReplicaSet with the desired changes, validate it, and then shift traffic.

Step 1: Extract Current Manifest

Export the current ReplicaSet definition as a baseline. You already did this in the inventory step, but ensure you have the latest version:

kubectl get replicaset my-replicaset -n my-namespace -o yaml > replicaset-current.yaml

Step 2: Create a New ReplicaSet with Desired Changes

Create a new ReplicaSet YAML file based on the current one, but with a new name and updated pod template. For example, if you are upgrading the container image from myapp:1.0 to myapp:2.0, and you want to add a new label version: v2 to distinguish pods, the new manifest might look like this:

apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: my-replicaset-v2
  namespace: my-namespace
  labels:
    app: my-app
    version: v2
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
      version: v2
  template:
    metadata:
      labels:
        app: my-app
        version: v2
    spec:
      containers:
      - name: app-container
        image: myapp:2.0
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: "100m"
            memory: "128Mi"
          limits:
            cpu: "200m"
            memory: "256Mi"

Important details:

• The ReplicaSet name is unique: my-replicaset-v2.
• The selector matchLabels must match the pod template labels. In this example, it is app: my-app and version: v2. This is different from the old ReplicaSet's selector, which might only have app: my-app or include version: v1.
• The container image is updated to myapp:2.0.
• Resource requests and limits are set explicitly. This is a good practice to ensure predictable scheduling.

Apply the new ReplicaSet:

kubectl apply -f replicaset-v2.yaml

Verify it is created:

kubectl get replicaset my-replicaset-v2 -n my-namespace

Step 3: Validate New ReplicaSet in Isolation

The new ReplicaSet's pods have the label version=v2, while the old pods have version=v1 (if you had that label before). Since the existing Service selector likely still points to version=v1, traffic continues to flow to old pods. This is intentional: the new pods are running but not receiving production traffic.

Wait for the new ReplicaSet's pods to become ready:

kubectl get pods -l version=v2 -n my-namespace

All pods should show READY 1/1 and STATUS Running.

To test the new pods without affecting production traffic, you can use port-forwarding directly to the ReplicaSet or to a specific pod:

kubectl port-forward rs/my-replicaset-v2 8080:80

Then open http://localhost:8080 in your browser or use curl from another terminal:

curl http://localhost:8080

Additionally, you can create a temporary Service with a selector that only matches version=v2 to test connectivity internally. For example, save the following as test-service-v2.yaml:

apiVersion: v1
kind: Service
metadata:
  name: my-service-v2-test
  namespace: my-namespace
spec:
  selector:
    app: my-app
    version: v2
  ports:
  - port: 80
    targetPort: 80

Apply it:

kubectl apply -f test-service-v2.yaml

Then run a temporary pod to test the service:

kubectl run test-pod --rm -it --image=busybox --restart=Never -- /bin/sh

Inside the pod, use wget or curl (if available) to access the service:

wget -qO- http://my-service-v2-test

You should see the response from the new version. After testing, delete the temporary service and pod:

kubectl delete svc my-service-v2-test -n my-namespace

Step 4: Shift Traffic Gradually

Once you are confident the new version works correctly, you need to update the production Service to route traffic to the new pods. There are two common approaches:

1. Direct switch: Update the Service selector to match only version=v2. This immediately moves all traffic to the new pods.
2. Canary release: Use a more sophisticated mechanism like an Ingress controller with traffic splitting (e.g., NGINX Ingress with canary annotations) or a service mesh like Istio or Linkerd. This allows you to send a small percentage of traffic to the new version and gradually increase it.

For simplicity and because this guide focuses on ReplicaSets, we will demonstrate a direct switch. However, if you have the tooling, a canary approach reduces risk further.

To perform a direct switch, edit the Service:

kubectl edit svc my-service -n my-namespace

Change the selector in the editor from:

selector:
  app: my-app
  version: v1

to:

selector:
  app: my-app
  version: v2

Save and exit. The Service will now route traffic to pods with version=v2. You can also use kubectl patch for a non-interactive update:

kubectl patch svc my-service -n my-namespace -p '{"spec":{"selector":{"version":"v2"}}}'

Verify the Service endpoints:

kubectl get endpoints my-service -n my-namespace

You should see the IP addresses of the new pods.

Step 5: Scale Down Old ReplicaSet

After verifying that the new version is handling traffic correctly, scale down the old ReplicaSet to zero:

kubectl scale replicaset my-replicaset --replicas=0 -n my-namespace

Do not delete the old ReplicaSet immediately. Keep it for a rollback window (for example, 24 hours). If something goes wrong with the new version, you can quickly scale the old ReplicaSet back up and switch the Service selector back to version=v1.

After the rollback window passes and you are confident, delete the old ReplicaSet:

kubectl delete replicaset my-replicaset -n my-namespace

Step 6: Update References

If you use an HPA that targets the ReplicaSet, update it to reference the new ReplicaSet. Note that HPAs typically target Deployments, but they can be configured to target ReplicaSets using the scaleTargetRef with kind: ReplicaSet. For example:

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: my-app-hpa
  namespace: my-namespace
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: ReplicaSet
    name: my-replicaset-v2
  minReplicas: 2
  maxReplicas: 10
  targetCPUUtilizationPercentage: 80

Apply the updated HPA:

kubectl apply -f hpa-v2.yaml

Scoped Implementation Choices

• Use namespaces to isolate changes, especially if you are testing.
• Consider using Deployments instead of bare ReplicaSets for future workloads. Deployments manage ReplicaSets and provide rolling updates and rollbacks automatically.
• Always keep the old ReplicaSet definition and its YAML backup for rollback.
• Use a version label (like version: v1 and version: v2) to distinguish pods. This is essential for safe traffic shifting.

Verification and Diagnostics

After applying changes, you must verify that the upgrade or migration is successful and the application behaves correctly. Perform these checks systematically.

Check ReplicaSet Status

kubectl get rs my-replicaset-v2 -n my-namespace

Expected output:

NAME               DESIRED   CURRENT   READY   AGE
my-replicaset-v2   3         3         3       5m

The DESIRED, CURRENT, and READY columns should all match the number of replicas you specified. If they don't, investigate the pods.

Check Pod Readiness

kubectl get pods -l version=v2 -n my-namespace

All pods should show READY 1/1 and STATUS Running. If any pod is not ready, check its events and logs.

Inspect Logs

View logs from all new pods:

kubectl logs -l version=v2 -n my-namespace --tail=50

Look for error messages, stack traces, or connection issues. If your application logs startup information, verify it started with the correct version.

Test Service Connectivity

Use a temporary pod to test the Service from inside the cluster:

kubectl run test-pod --rm -it --image=busybox --restart=Never -- /bin/sh

Inside the pod, use wget or curl (if installed) to access the Service:

wget -qO- http://my-service

Or, if curl is available:

curl http://my-service

Expect the application response. If you get a connection refused or timeout, the Service may not be routing correctly. Check the Service endpoints and pod readiness.

Check Events for Errors

kubectl describe rs my-replicaset-v2 -n my-namespace

Look for events such as Created pod, Scaled up replica set, or any warnings. Warnings like FailedScheduling or Failed to pull image indicate problems.

Monitor Resource Usage

If you have the metrics server installed, you can view CPU and memory usage of the new pods:

kubectl top pods -l version=v2 -n my-namespace

Compare resource usage with requests and limits. High CPU or memory usage might indicate a performance issue with the new version.

Automated Checks Consideration

For production environments, consider scripting these checks to run automatically after deployment. A simple shell script or a CI/CD pipeline step can execute the kubectl commands and fail the pipeline if any check fails. This reduces human error and speeds up the process.

Failure Modes and Recovery

Despite careful planning, failures can occur. Here are common failure modes, their causes, diagnostics, and recovery steps.

Failure: New Pods CrashLoopBackOff

Cause: Misconfiguration, incompatible image, missing environment variables, or application bug.

Diagnostics:

kubectl get pods -l version=v2 -n my-namespace
kubectl logs <pod-name> -n my-namespace
kubectl describe pod <pod-name> -n my-namespace

Example: You see pods in CrashLoopBackOff state. The logs show Error: Cannot find module 'express' indicating a missing dependency in the image.

Recovery:

1. Revert traffic to the old ReplicaSet by updating the Service selector back to version=v1 (if the old ReplicaSet still exists and its replicas are not zero). If you scaled the old ReplicaSet to zero, scale it back up first:

kubectl scale replicaset my-replicaset --replicas=3 -n my-namespace
kubectl patch svc my-service -n my-namespace -p '{"spec":{"selector":{"version":"v1"}}}'

1. Delete or scale down the new ReplicaSet to prevent resource usage:

kubectl scale replicaset my-replicaset-v2 --replicas=0 -n my-namespace

1. Fix the configuration or image and plan the next attempt.

Failure: New Pods Not Scheduled

Cause: Insufficient cluster resources, node selector constraints, taints/tolerations, or persistent volume claims not available.

Diagnostics:

kubectl describe rs my-replicaset-v2 -n my-namespace

Look for events like FailedScheduling with messages such as 0/3 nodes are available: 3 Insufficient cpu.

Recovery:

• Adjust resource requests and limits to fit available capacity.
• Modify node selectors or add tolerations if needed.
• Scale down the old ReplicaSet to free resources (after traffic is shifted or if you are still testing):

kubectl scale replicaset my-replicaset --replicas=1 -n my-namespace

Then recreate or scale the new ReplicaSet.

Failure: Service Traffic Interrupted

Cause: Service selector change too abrupt, misconfigured selector, or a bug in the new version that causes poor performance or crashes.

Recovery:

Immediately revert the Service selector to the previous value:

kubectl patch svc my-service -n my-namespace -p '{"spec":{"selector":{"version":"v1"}}}'

If the old ReplicaSet was scaled to zero, scale it back up first.

Then investigate the issue with the new version.

Failure: Data Loss or Inconsistency

Cause: Application not designed for zero-downtime migration, database schema changes not backward compatible, or version incompatibility.

Recovery:

• Restore from backups if data was lost.
• Consider running old and new versions in parallel with a canary approach to detect inconsistencies early.
• If possible, use a blue-green deployment strategy with a database migration tool that supports backward-compatible changes.

General Rollback Steps

1. Identify the issue (from logs, metrics, user reports).
2. Restore the Service selector to the old version.
3. Scale the old ReplicaSet back to desired replicas if it was scaled down.
4. Delete or scale down the new ReplicaSet.
5. Fix the configuration and plan the next attempt.

Preventive Measures

• Always keep the old ReplicaSet until the new one is proven stable.
• Use a version label to distinguish pods.
• Test in staging first, ideally with production-like traffic.
• Automate deployment and rollback to reduce human error.
• Monitor application metrics and logs continuously during the migration.

Operations Checklist

Use this checklist for every ReplicaSet upgrade or migration. Fill in the specific names and commands for your environment.

Review Points

• Confirm the rollback plan before starting.
• Communicate with team members about the change window.
• Perform the upgrade during a maintenance window if possible, especially for high-traffic services.
• Have on-call engineer available during and immediately after the migration.

Conclusion

Upgrading and migrating Kubernetes ReplicaSets can be done safely with careful planning and incremental steps. By inventorying your environment, creating new ReplicaSets alongside old ones, validating in isolation, and having a solid rollback plan, you minimize risk. Use the provided operational checklist to standardize the process. Remember to keep old ReplicaSets until you are certain the new version is stable, and consider using Deployments for future workloads to take advantage of built-in rolling updates. With these practices, you can confidently manage ReplicaSet lifecycle changes in production.