Kubernetes EndpointSlice Backup and Restore: A Practical Field Guide

Introduction

When a Kubernetes Service stops routing traffic correctly, the root cause often hides in its EndpointSlice objects. These small API resources track ready pod IPs and ports so kube-proxy and ingress controllers can forward requests. If an EndpointSlice is deleted accidentally, corrupted by a bad manifest, or lost during a cluster migration, the Service becomes a black hole: no errors appear on the Service itself, but every request stalls or fails.

This guide walks through practical backup, restore, and validation procedures for EndpointSlice objects. It covers version detection, environment inventory, safe configuration changes, diagnostics, failure modes, and an operations checklist. Each section includes concrete commands, expected output, and recovery decisions. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached.

This article is written for developers, DevOps consultants, and technical startup teams running Kubernetes in production or staging. It assumes basic familiarity with kubectl and core networking concepts, but no prior EndpointSlice experience is required.

Version and Environment Inventory

Before touching EndpointSlices, confirm what is running in the cluster. EndpointSlice graduated to stable in Kubernetes 1.21, and older clusters may still rely on Endpoints objects. The backup and restore procedure differs depending on which object is present.

Start with a read-only inventory. Run the following commands to capture the current state:

kubectl version --short
kubectl get nodes -o wide
kubectl get services --all-namespaces
kubectl get endpointslices --all-namespaces

Expected output for kubectl get endpointslices --all-namespaces on a healthy cluster looks like this:

NAMESPACE   NAME                   ADDRESSTYPE   PORTS   ENDPOINTS   AGE
default     nginx-service-abcde   IPv4          80      3           2d
kube-system kube-dns-xyz12        IPv4          53      2           30d

If no EndpointSlices appear but Services exist, the cluster may be using legacy Endpoints. Check with kubectl get endpoints --all-namespaces. In Kubernetes 1.21 and later, both Endpoints and EndpointSlices exist, but kube-proxy uses EndpointSlices by default.

Record the exact Kubernetes version because restore behavior changes between versions. For example, in Kubernetes 1.22, the EndpointSlice addressType field defaulted to IPv4, while in 1.25 the field supports IPv4, IPv6, and FQDN. Backing up on 1.22 and restoring on 1.25 may require adjusting the manifest.

Also capture the current EndpointSlice labels and owner references. The owner reference links an EndpointSlice to its Service. Without a correct owner reference, the EndpointSlice may not be garbage-collected or may be ignored by controllers. Use:

kubectl get endpointslices -n default -o yaml

Inspect the output for ownerReferences and labels.kubernetes.io/service-name. If these are missing, the EndpointSlice is orphaned and should be treated as a manual backup candidate.

Prerequisites for safe backup:

• Read access to all namespaces where Services are deployed.
• kubectl version matching the cluster minor version (within one minor version skew).
• A writable directory for backup files, such as /var/backups/k8s-endpointslices.
• No active changes to Services or Pods during backup (or use a maintenance window).

A small verification step before any change: run kubectl get endpointslices -n default -o jsonpath='{.items[*].metadata.name}' and compare the output with the Service names. Every Service should have at least one associated EndpointSlice. If a Service has zero EndpointSlices, traffic to it will fail; this is the first failure signal.

Safe Configuration Path

Backing up EndpointSlices is a read operation, but restoring them is a write operation. The safest path is to restore into a temporary namespace first, verify the objects are created correctly, then move them to the target namespace only after confirmation.

Backup Procedure

Use kubectl get with -o yaml to export all EndpointSlices in a namespace. For example:

mkdir -p /var/backups/k8s-endpointslices/default
kubectl get endpointslices -n default -o yaml > /var/backups/k8s-endpointslices/default/all-endpointslices.yaml

This file contains the full live state, including metadata.resourceVersion and metadata.uid. Those fields are cluster-specific and must be removed before a restore into a different cluster. A safer backup strips dynamic fields:

kubectl get endpointslices -n default -o yaml | grep -v 'resourceVersion\|uid\|creationTimestamp\|generation' > /var/backups/k8s-endpointslices/default/all-endpointslices-clean.yaml

However, this simple grep may remove lines inside containers spec if any field names match. A more reliable method uses kubectl get --export, but that flag is deprecated. The recommended approach is to use a tool like kubectl neat (from the itaysk/kubectl-neat project) or write a small jq script:

kubectl get endpointslices -n default -o json | jq 'del(.items[].metadata.resourceVersion, .items[].metadata.uid, .items[].metadata.creationTimestamp, .items[].metadata.generation)' > /var/backups/k8s-endpointslices/default/all-endpointslices-clean.json

Store the backup in version control or an encrypted object store. For disaster recovery, keep backups of both the EndpointSlice objects and the associated Service objects, because EndpointSlices are typically owned by Services and are recreated by the EndpointSlice controller.

Restore Procedure (Dry Run First)

Before applying a backup, test the restore in a scratch namespace. Create a namespace restore-test and apply the clean backup with --dry-run=server:

kubectl create namespace restore-test
kubectl apply -f /var/backups/k8s-endpointslices/default/all-endpointslices-clean.yaml -n restore-test --dry-run=server

Expected output for a valid restore:

endpointslice.discovery.k8s.io/nginx-service-abcde created (server dry run)

If the output shows errors about missing fields or invalid addresses, fix the backup file before proceeding.

Then apply for real in the scratch namespace:

kubectl apply -f /var/backups/k8s-endpointslices/default/all-endpointslices-clean.yaml -n restore-test

Verify the endpoints are populated:

kubectl get endpointslices -n restore-test -o wide

The ENDPOINTS column should show the same number of ready pod IPs as the original. If the original had 3 endpoints, the restored slice should also show 3.

After verification, delete the scratch namespace:

kubectl delete namespace restore-test

Finally, restore into the target namespace only if the Service does not already have an EndpointSlice. If the Service exists and its EndpointSlice is missing, applying the backup will recreate it; the EndpointSlice controller may also recreate it automatically within minutes. Check with:

kubectl get endpointslices -n default -l kubernetes.io/service-name=nginx-service

If the backup was successful, output should show the slice with the correct labels and endpoints. If the controller recreates it first, the applied backup may conflict. Always observe the current state before applying a restore.

Rollback Strategy

If a restore causes problems (e.g., wrong endpoint IPs because pods have changed), delete the restored EndpointSlice and let the controller recreate it from current pod state:

kubectl delete endpointslice -n default nginx-service-abcde

Within seconds, the EndpointSlice controller will create a new slice based on the current pod readiness. This is the fastest rollback for a corrupted slice, but it relies on the controller running correctly. If the controller is broken, restore from a known-good backup instead.

Verification and Diagnostics

Verification of EndpointSlice health goes beyond checking that the object exists. The slice must have ready endpoints, correct ports, and matching labels. Use the following diagnostic checks.

Check EndpointSlice Connectivity

The simplest end-to-end test is to curl a Service that uses the slice. For a ClusterIP Service:

kubectl run -it --rm debug --image=curlimages/curl --restart=Never -- curl -v http://nginx-service.default.svc.cluster.local

Expected output includes HTTP/1.1 200 OK if the endpoints are healthy. If the connection hangs or returns 502 Bad Gateway, the EndpointSlice may be empty or have stale IPs.

Inspect the slice contents directly:

kubectl get endpointslices -n default nginx-service-abcde -o yaml

Look for the endpoints array. Each endpoint should have an addresses field with a pod IP and a conditions.ready field set to true. If ready is false or missing, the pod is not ready and traffic will not be routed. Example healthy endpoint entry:

endpoints:
- addresses:
  - 10.244.1.5
  conditions:
    ready: true
    serving: true
    terminating: false
  targetRef:
    kind: Pod
    name: nginx-deployment-6d4cf56db6-abcde
    namespace: default

If ready is false, check the pod status with kubectl get pods -n default -o wide. The pod may be in CrashLoopBackOff or Pending. Troubleshoot with kubectl describe pod <name> and kubectl logs <name> --previous.

Validate Port Mappings

EndpointSlices store port numbers and protocols. Confirm they match the Service definition. Compare:

kubectl get service nginx-service -n default -o yaml
kubectl get endpointslice nginx-service-abcde -n default -o yaml

The Service ports should map to the EndpointSlice ports exactly. For example, a Service with port: 80, targetPort: 8080 should have an EndpointSlice with ports: [{name: http, port: 8080, protocol: TCP}]. If the targetPort is absent or wrong, traffic will be sent to the wrong container port.

Diagnose Controller Issues

If EndpointSlices are not being created or updated, check the EndpointSlice controller logs in kube-controller-manager. On a managed cluster (EKS, GKE, AKS), you cannot access controller logs directly, but you can check events:

kubectl get events -n default --sort-by='.lastTimestamp'

Look for FailedCreateEndpointSlice or FailedUpdateEndpointSlice events. If present, the controller may lack permissions or the API server may be rejecting the slice due to invalid fields.

On a self-managed cluster, view controller logs:

kubectl logs -n kube-system kube-controller-manager-<node> | grep endpointslice

Errors such as endpointslice controller: error syncing service indicate a broken controller loop. Restarting the kube-controller-manager pod may resolve it.

Failure Modes and Recovery

Understanding common failure modes helps plan recovery. Here are the most frequent scenarios and concrete recovery steps.

Accidental Deletion of an EndpointSlice

An operator mistakenly runs kubectl delete endpointslice -n default nginx-service-abcde. The Service still exists, but traffic fails because no endpoints are available.

Recovery:

• Immediately restore from backup: kubectl apply -f /var/backups/k8s-endpointslices/default/nginx-service-abcde.yaml
• If no backup, delete and recreate the Service to force the controller to generate a new slice: kubectl delete service nginx-service -n default && kubectl apply -f nginx-service.yaml
• Or wait: in most clusters, the EndpointSlice controller recreates a deleted slice within 1-2 minutes, as long as the Service still exists and pods are ready.

Stale IPs After Pod Restart

A pod restarts and gets a new IP, but the EndpointSlice still shows the old IP. This can happen if the EndpointSlice controller is delayed or if a manual update set the wrong address. Traffic may be sent to a dead pod and time out.

Recovery:

• Force the controller to resync by deleting the EndpointSlice: kubectl delete endpointslice -n default nginx-service-abcde
• The controller will create a new slice with the current pod IPs.
• If the controller is broken, manually edit the slice with kubectl edit endpointslice -n default nginx-service-abcde and update the addresses field.

EndpointSlice Not Created for a New Service

A new Service is deployed, but kubectl get endpointslices shows nothing. This usually means the Service selector does not match any pods.

Recovery:

• Check the Service selector: kubectl get service new-service -n default -o yaml | grep selector -A3
• Check pods with matching labels: kubectl get pods -n default -l app=new-app
• If no pods match, update the Service selector or pod labels.
• After fixing, wait a few seconds and check again. The controller should create the slice automatically.

Invalid EndpointSlice After Manual Edit

A manual edit corrupts the slice, for example, setting port: 9999 when the container listens on 8080. Traffic fails with connection refused.

Recovery:

• Restore from backup: kubectl apply -f /var/backups/k8s-endpointslices/default/nginx-service-abcde.yaml
• Or delete the slice and let the controller recreate it.

Disaster Recovery After Cluster Loss

If the entire cluster is lost and must be rebuilt from backups, EndpointSlices alone are not enough. The backup must include Service objects, Deployments, and other resources.

Recovery steps:

1. Rebuild the cluster and configure networking.
2. Apply all Deployment and StatefulSet manifests.
3. Apply Service manifests.
4. Wait for pods to become ready.
5. The EndpointSlice controller will create slices automatically. Do not restore old EndpointSlices, as their pod IPs are stale from the old cluster.
6. Only restore EndpointSlices manually if the controller is disabled or the cluster runs a custom service proxy that does not use EndpointSlices.

Operations Checklist

Use this checklist before and after any EndpointSlice backup or restore operation. Replace the example values with your own.

Keep this checklist in your team's runbook. Every operator should know where backups are stored, how to restore, and how to verify without causing further disruption.

Conclusion

Kubernetes EndpointSlice backup and restore is not a complex procedure, but it requires discipline. The objects are small and often auto-managed, which leads many teams to ignore them until a Service outage exposes the gap. By following the steps in this guide, you can create reliable backups, test restores safely, diagnose common failures, and recover quickly.

Start with a low-risk verification: choose one Service in a development namespace, back up its EndpointSlice, delete it, and restore from backup. Time how long the process takes and document any errors. Then extend the procedure to production namespaces with appropriate change controls.

Remember that EndpointSlices are only one part of the Service routing chain. Dependencies such as kube-proxy, CoreDNS, and the EndpointSlice controller all interact. When troubleshooting, consider the whole stack, but keep your backup and restore procedures focused and versioned.

A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. With this guide, you have the commands and checklists to implement that workflow for EndpointSlices today.