Kubernetes Service Upgrade and Migration: A Practical Implementation Guide

Intro

Upgrading or migrating a Kubernetes Service is rarely a single kubectl command. It is a sequence of observations, scoped changes, verifications, and recovery decisions that must remain safe in a live cluster. This guide provides a practical, command-driven path for developers, DevOps consultants, and technical startup teams who need to move a Service from one version or shape to another without breaking traffic.

The focus is operational safety: capture the current state before changing anything, limit the blast radius, avoid exposing secrets in manifests or commands, verify each step with an observable signal, and document recovery before you need it. Whether you are moving from a legacy iptables-based kube-proxy mode to IPVS, migrating a Service from the in-tree cloud provider to an external controller, or simply changing selectors and ports, the principles below apply.

Throughout this article, you will find concrete commands, expected outputs, and decision thresholds. No step relies on prose alone. For each major operation, there is a concrete check you can run in a test namespace first. By the end, you will have a reusable workflow for Kubernetes Service upgrade, migration, rollback, and validation.

Version and Environment Inventory

Before upgrading or migrating a Service, you must know exactly what runs in your cluster and what the target version supports. A controlled change starts with a complete, read-only inventory. This means capturing the Kubernetes version, the kube-proxy mode, the CNI plugin, the service controller, and the current Service definition.

Commands for Environment Inventory

Run these commands and save their output with timestamps. They are safe and read-only.

# Cluster and node versions
kubectl version --short
kubectl get nodes -o wide

# kube-proxy mode (iptables, ipvs, userspace) and proxy image
kubectl get pods -n kube-system -l k8s-app=kube-proxy -o jsonpath='{.items[0].spec.containers[0].args}' | tr ' ' '\n' | grep -E "proxy-mode|cluster-cidr"
kubectl get pods -n kube-system -l k8s-app=kube-proxy -o jsonpath='{.items[0].spec.containers[0].image}'

# CNI plugin and version (example: Calico)
kubectl get pods -n kube-system | grep -E "calico|cni"
kubectl get daemonset -n kube-system -o wide | grep -E "calico|cni"

# Service controller (cloud provider or external)
kubectl get pods -n kube-system | grep -E "cloud-controller|service-controller"

Expected output includes the Kubernetes server version (for example, v1.28.5), node OS images, kernel versions, and the exact kube-proxy argument list. If you see --proxy-mode=iptables and plan to move to IPVS, that is a migration trigger. If you see the in-tree cloud provider is still active, you may need to migrate to the external cloud controller manager before upgrading.

Capture the Current Service Manifest

Export the live Service object exactly as it exists, not as you think it exists. This manifest is your baseline for diffing and rollback.

kubectl get service <service-name> -n <namespace> -o yaml > service-current.yaml

Open the file and inspect these fields:

• spec.type: ClusterIP, NodePort, LoadBalancer, or ExternalName.
• spec.selector: Labels that select the backend pods.
• spec.ports: Port mapping and targetPort.
• spec.clusterIP: The allocated IP, if any.
• metadata.annotations: Cloud provider annotations, external-dns, etc.
• status.loadBalancer: The external ingress, if assigned.

For a LoadBalancer Service, also check the associated EndpointSlice objects:

kubectl get endpointslice -n <namespace> -l kubernetes.io/service-name=<service-name> -o yaml

This becomes critical when migrating to a new service controller, because the controller must adopt existing endpoints or you will lose traffic.

Scope the Smallest Justified Change

Example: if your goal is only to upgrade a Service to use a new annotation for the cloud load balancer, do not simultaneously change the selector or ports. Sequence the changes. For a version upgrade of the Service controller, first verify that the new controller can reconcile the existing Service without modification. Then apply the upgrade, then change annotations or other fields.

Document the supported version range for each component. For instance:

• Kubernetes control plane: v1.26 to v1.29 (your target).
• kube-proxy: must match control plane within one minor version skew.
• Cloud controller manager: v1.28+ for AWS, v1.27+ for GCP.
• CNI: Calico v3.25+ for Kubernetes v1.28.

If your current cluster is v1.25 and you want to jump to v1.29, you must perform sequential upgrades: 1.25 to 1.26, then 1.26 to 1.27, etc. Skipping minor versions is unsupported and can break Services.

Local Test Strategy

Do not test on a production LoadBalancer first. Create a local test by applying a minimal Service of type ClusterIP in a dedicated namespace, or use kubectl port-forward to test connectivity to a single pod. Only after local verification should you proceed to cloud load balancer or ingress controller changes.

Safe Configuration Path

The Safe Configuration Path is about making changes that are reversible and observable. You should apply one manifest at a time, verify, and only then move to the next change.

Before Any Change: Protect Credentials and Private Material

Never put secrets directly in a Service manifest. If you need to configure a cloud provider or external-dns annotation that requires credentials, use a referenced Secret. For example:

apiVersion: v1
kind: Service
metadata:
  name: my-service
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:us-east-1:123456789012:certificate/uuid
    external-dns.alpha.kubernetes.io/hostname: app.example.com
spec:
  type: LoadBalancer
  ports:
  - port: 443
    targetPort: 8443
  selector:
    app: my-app

The certificate ARN is not secret, but any private key or token must be in a Secret and referenced via annotation like service.beta.kubernetes.io/aws-load-balancer-ssl-cert pointing to a certificate stored in AWS Certificate Manager. For on-prem controllers, use Kubernetes Secrets mounted in the controller pod, not in the Service object.

Use Version Control and Diffing

Store all manifests in Git. Before applying, always diff against the live object:

kubectl diff -f service-new.yaml

This shows exactly what would change. If you see unintended modifications, stop. Use a tool like kubectl apply --server-side with a manager name to track ownership and avoid conflicts.

Smallest Justified Change in Practice

Suppose you need to migrate a Service from the in-tree AWS cloud provider to the external AWS Cloud Controller Manager. The in-tree provider may be deprecated in your Kubernetes version. The safe path is:

1. Ensure the external cloud controller manager is running and healthy.
2. Test that it can reconcile a new LoadBalancer Service in a test namespace.
3. For each existing Service, add the annotation service.kubernetes.io/load-balancer-cleanup with value true to the current Service before removing the in-tree provider. This tells the new controller to adopt the existing load balancer.
4. Change the service controller. For many managed Kubernetes services, this happens automatically during the upgrade. For self-managed clusters, you must disable the in-tree provider and start the external controller.
5. Verify that the existing external IP remains unchanged and the load balancer is still routing.

Example annotation adoption command:

kubectl annotate service my-service -n production \
  service.kubernetes.io/load-balancer-cleanup=true --overwrite

Then upgrade the cloud controller manager. The new controller sees the annotation and takes ownership without recreating the load balancer.

Rollback Before You Need It

For every manifest change, save the previous version. Use kubectl get service my-service -o yaml > service-backup-$(date +%s).yaml. To rollback, apply the backup:

kubectl apply -f service-backup-1717000000.yaml

Verify that the Service spec returns to its prior state and the endpoints are unchanged. If you are migrating to a new controller, rollback may require re-enabling the old controller. Document this process in your operations runbook.

Verification and Diagnostics

Verification is not just checking that kubectl apply succeeded. You must verify that the Service behaves as expected from the perspective of an actual client. This section gives you concrete checks for connectivity, endpoints, DNS, and controller status.

Verify Service Endpoints

A Service is only useful if it has healthy endpoints. Use:

kubectl get endpoints my-service -n production

Expected output shows a list of pod IPs and ports:

NAME         ENDPOINTS                                           AGE
my-service   10.244.1.5:8080,10.244.2.3:8080,10.244.3.7:8080   5m

If endpoints are empty, the selector does not match any pods, or the pods are not ready. Immediately check the pods:

kubectl get pods -n production -l app=my-app -o wide
kubectl describe pod <pod-name> -n production

Look for events like FailedScheduling, CrashLoopBackOff, or Readiness probe failed.

For a more modern view, check EndpointSlices:

kubectl get endpointslice -n production -l kubernetes.io/service-name=my-service -o yaml

Each slice should list the same endpoint addresses. If you recently migrated the Service controller, confirm that the controller updated the EndpointSlices and not just the legacy Endpoints object.

Test Connectivity from a Client Pod

Create a temporary pod in the same namespace:

kubectl run test-client --rm -it --image=busybox --restart=Never -- /bin/sh

Inside the pod, run:

nslookup my-service.production.svc.cluster.local
wget -qO- http://my-service.production.svc.cluster.local:8080/health

Expected output includes the ClusterIP returned by DNS and the HTTP 200 response from the health endpoint. If DNS fails, check CoreDNS pods. If connection times out, check network policy and kube-proxy.

For a LoadBalancer Service, test from outside the cluster:

curl -v https://app.example.com

Confirm the TLS certificate is valid and the response comes from your service. Check the load balancer's health check target and healthy threshold. In AWS, use:

aws elbv2 describe-target-health --target-group-arn <arn>

Check kube-proxy and Service Rules

If you are upgrading or migrating kube-proxy modes (iptables to IPVS), you must verify that the new mode installed correctly and that iptables rules are consistent.

Check the mode on each node:

kubectl get pods -n kube-system -l k8s-app=kube-proxy -o jsonpath='{.items[*].spec.containers[0].args}' | tr ' ' '\n' | grep proxy-mode

On a node, inspect the rules. For iptables mode:

sudo iptables-save | grep -A5 "KUBE-SERVICES"

You should see a chain for each Service. For IPVS mode:

sudo ipvsadm -Ln

The output should list the Service ClusterIP, scheduler (rr, wrr, etc.), and backend pod IPs.

Controller Status and Events

For a service controller upgrade or migration, watch its logs and events:

kubectl logs -n kube-system -l app=aws-cloud-controller-manager --tail=50
kubectl get events -n production --field-selector involvedObject.name=my-service

Look for messages about failing to reconcile load balancer, security group conflicts, or annotation errors. If the controller cannot find the load balancer because of missing permissions, the Service will stay in Pending state for the LoadBalancer IP. In that case, check IAM roles or provider credentials.

Failure Modes and Recovery

Despite careful planning, failures happen. This section describes common failure modes during Service upgrade and migration, their symptoms, and recovery steps.

Failure Mode 1: Empty Endpoints After Selector Change

Symptom: You change the selector to point to a new set of pods, but kubectl get endpoints shows no endpoints.

Recovery:

1. Revert the selector change by applying the backup manifest.
2. Check that the new pods exist and have matching labels:

kubectl get pods -n production --show-labels | grep <new-label>

1. If labels do not match, correct the labels before re-applying the selector change. Use kubectl label pods <pod-name> app=new-app --overwrite if needed.

1. Verify endpoints return.

Failure Mode 2: LoadBalancer IP Changes After Migration

Symptom: After migrating to the external cloud controller manager, the Service gets a new external IP, causing DNS to point to a new load balancer and dropping all traffic.

Recovery:

1. Immediately delete the new Service or revert the migration by re-enabling the old controller.
2. Check if the old load balancer still exists and its DNS name is valid.
3. If possible, reassign the old DNS name to the old load balancer by updating DNS records manually.
4. Investigate why the new controller did not adopt the existing load balancer. The annotation service.kubernetes.io/load-balancer-cleanup must be present on the old Service before the migration. If absent, the new controller created a new load balancer.

Example of correct annotation before migration:

kubectl annotate service my-service -n production \
  service.kubernetes.io/load-balancer-cleanup=true --overwrite

Then confirm the annotation is in place:

kubectl get service my-service -n production -o jsonpath='{.metadata.annotations}'

Failure Mode 3: kube-proxy Upgrade Breaks Service Routing

Symptom: After upgrading kube-proxy to a new version or changing proxy mode, some Services become unreachable or intermittently time out.

Recovery:

1. Roll back kube-proxy to the previous version using your deployment method (e.g., kubectl rollout undo daemonset kube-proxy -n kube-system).
2. If changing mode from iptables to IPVS, ensure the kernel modules are loaded on all nodes. On each node, run:

sudo modprobe ip_vs
sudo modprobe ip_vs_rr
sudo modprobe ip_vs_wrr
sudo modprobe ip_vs_sh

1. Also check that --proxy-mode=ipvs is set in the kube-proxy config. If not, update the configmap and restart kube-proxy.

1. After rollback or fix, verify connectivity using the test client pod described earlier.

Failure Mode 4: Service Controller Loses Permissions

Symptom: The Service remains in Pending state for the LoadBalancer IP, and controller logs show AccessDenied or Forbidden.

Recovery:

1. Check the cloud provider IAM role attached to the controller. Ensure it has permissions for load balancer operations (e.g., elasticloadbalancing:* for AWS).
2. If using a ServiceAccount with IRSA, verify the annotation on the ServiceAccount:

kubectl get serviceaccount aws-load-balancer-controller -n kube-system -o yaml

1. Correct the IAM policy or role binding, then restart the controller pod. Monitor the Service status:

kubectl describe service my-service -n production

Expect the LoadBalancer Ingress field to populate within a few minutes.

General Recovery Principles

• Always have a backup of the Service manifest before any change.
• Use kubectl apply --dry-run=server to preview changes without applying.
• For complex migrations, use a canary approach: migrate a non-critical Service first, verify, then proceed.
• Document the rollback steps for each change in your runbook before executing the change.
• If a change causes an outage, restore service first, then investigate root cause. Do not leave a broken Service while debugging.

Operations Checklist

The following checklist consolidates the safe workflow for any Kubernetes Service upgrade or migration. Use it every time. Replace the example values with your own environment.

• [ ] Inventory captured: Run kubectl version --short, kubectl get nodes -o wide, and capture kube-proxy mode and controller images. Store output with timestamp.
• [ ] Current Service exported: kubectl get service my-service -n production -o yaml > service-current.yaml and saved in Git.
• [ ] Prerequisites verified: Confirm that cluster version is within supported range for the target controller or proxy. Example: Kubernetes v1.27.3 and AWS Cloud Controller Manager v1.27.1.
• [ ] Test in local namespace: Applied a modified Service in namespace test-ns and verified endpoint connectivity using a busybox test client.
• [ ] Backup manifest created: kubectl get service my-service -n production -o yaml > service-backup-1717000000.yaml and committed.
• [ ] Smallest change applied: Used kubectl diff -f service-new.yaml to confirm only intended changes. Applied with kubectl apply -f service-new.yaml.
• [ ] Endpoints verified: kubectl get endpoints my-service -n production shows expected pod IPs and ports. Example: 10.244.1.5:8080,10.244.2.3:8080.
• [ ] Connectivity tested: From test client pod, wget -qO- http://my-service.production.svc.cluster.local:8080/health returns HTTP 200.
• [ ] Controller status checked: kubectl logs -n kube-system -l app=aws-cloud-controller-manager --tail=20 shows no errors. kubectl get events -n production --field-selector involvedObject.name=my-service shows successful reconciliation.
• [ ] Rollback procedure documented: For this change, rollback command is kubectl apply -f service-backup-1717000000.yaml. If controller migration, also document how to revert controller.
• [ ] Post-change monitoring: Set up a watch on Service and endpoints for 10 minutes: kubectl get service my-service -n production -w and kubectl get endpoints my-service -n production -w.
• [ ] Secrets not exposed: Confirm no secrets in Service manifest or command history. Any credentials are in Kubernetes Secrets referenced by annotations or controller pods.

Conclusion

Upgrading and migrating Kubernetes Services is a process, not a one-off command. This guide has walked through the essential phases: environment inventory, safe configuration, verification, failure modes, and an operations checklist. The core principle is to observe before changing, change one thing at a time, verify with real traffic, and always have a rollback path.

Start with a low-risk Service in a test namespace. Run the inventory commands, export the current manifest, make a single change (such as adding a controller annotation), and verify connectivity from a test client. Only after that succeeds should you move to production Services and larger migrations like controller upgrades or proxy mode changes.

Remember that components like Ingress, EndpointSlice, and Network Policy are related but separate; include them in your planning only when they directly affect the Service upgrade or migration. A successful upgrade is not just a version bump; it is a controlled, verifiable transition that keeps your applications reachable.

Your next action: pick one Service in your cluster, run the Version and Environment Inventory commands, and save the output. Then identify one small, reversible improvement you can test locally. This builds the muscle memory for larger, more complex migrations. Reliable operations are built on visible failures, protected secrets, scoped changes, and documented recovery.