Kubernetes Server-Side Apply: A Practical Guide to Network Troubleshooting

Intro

When a Kubernetes Deployment or Service configuration drifts unexpectedly, Server-Side Apply (SSA) changes how you track and fix the problem. Instead of guessing which client last patched a field, SSA records field ownership and conflict metadata, making network misconfigurations easier to isolate and repair. This guide shows how to troubleshoot Kubernetes networking in an SSA-managed cluster using concrete commands, expected outputs, and recovery steps.

The article is written for developers, DevOps engineers, and technical startup teams who operate Kubernetes clusters with SSA enabled. It covers the most common networking trouble spots: DNS resolution, port mismatches, inter-pod connectivity, and network policy behavior. Each section starts with a read-only observation of the current state, moves to a minimal change, and ends with a verification command that proves the fix worked.

The operational principles throughout are simple: observe before you change, limit the blast radius, never put secrets in manifests, verify the result, and document a rollback path before you need it.

Version and Environment Inventory

Before touching any network configuration, establish the exact versions and components in play. Start by confirming the Kubernetes server version and API capabilities.

Verify SSA support

SSA became stable in Kubernetes 1.22, but earlier versions may enable it with a feature gate. Check the server version:

kubectl version --short

Expected output looks like:

Client Version: v1.27.3
Kustomize Version: v5.0.1
Server Version: v1.27.3

If the server version is 1.22 or higher, SSA is available by default. To confirm SSA is active for the apply command:

kubectl apply --server-side --dry-run=server -f deployment.yaml

The absence of an error indicates support. For older clusters, check the feature gate:

kubectl get --raw /metrics | grep server_apply

If the metric is absent or zero, SSA may not be enabled.

Check core networking components

Networking in Kubernetes depends on several components. Identify the CNI plugin in use:

kubectl get pods -n kube-system | grep -E 'calico|flannel|weave|cilium|aws-node|azure-cni'

For example, a Calico-based cluster shows:

calico-node-4x7jz                          1/1     Running   0          12d
calico-kube-controllers-7c8c9c4d56-8p5vz   1/1     Running   0          12d

If you are using a managed service, the CNI may not run as pods. Check the node status for the CNI annotation:

kubectl get nodes -o jsonpath='{.items[0].metadata.annotations}'

Look for keys like cni.projectcalico.org/podIP or similar.

Inspect SSA-managed network objects

List network-related resources that SSA manages:

kubectl get services,endpoints,ingresses -o wide

Note the MANAGED FIELDS column if present; with kubectl get you can request it explicitly:

kubectl get service my-service -o yaml | grep -A 5 managedFields

A typical entry shows the manager and operation:

managedFields:
- manager: kubectl
  operation: Apply
  apiVersion: v1
  time: "2024-03-26T10:11:12Z"
  fieldsType: FieldsV1
  fieldsV1:
    f:spec:
      f:ports: {}

This confirms SSA is tracking the field. If a field is missing ownership, conflicts will surface later.

Safe Configuration Path

Changing network settings with SSA requires a controlled sequence: record current state, apply a minimal change, verify, and revert if needed.

Capture the current state

Before any change, dump the live object to a backup file using kubectl get with output in YAML:

kubectl get service my-service -o yaml > service-backup-$(date +%Y%m%d-%H%M%S).yaml

Also run kubectl diff to preview the exact changes SSA would make without applying:

kubectl diff -f updated-service.yaml

If the diff is empty, your local file matches the live object. If you see lines starting with + or -, those are the changes.

Apply a minimal SSA change

Suppose you need to change a service port from 80 to 8080. The original YAML snippet:

spec:
  ports:
  - name: http
    port: 80
    targetPort: 80

Modify only the targetPort field:

spec:
  ports:
  - name: http
    port: 80
    targetPort: 8080

Apply with server-side apply and specify the field manager explicitly:

kubectl apply --server-side --field-manager=network-fix -f updated-service.yaml

Expected output:

service/my-service serverside-applied

If another manager owns the targetPort field, you will see a conflict:

error: Apply failed with 1 conflict: conflict with "other-manager": .spec.ports[0].targetPort

To resolve, you can force ownership with --force-conflicts, but only after verifying it is safe:

kubectl apply --server-side --field-manager=network-fix --force-conflicts -f updated-service.yaml

Use --force-conflicts sparingly; it transfers field ownership to you and may overwrite intentional changes by another team.

Verify the change

After applying, check the live object:

kubectl get service my-service -o yaml | grep -A 3 ports

You should see targetPort: 8080. Also confirm the endpoints updated:

kubectl get endpoints my-service

Expected output shows the pod IP and new port:

NAME         ENDPOINTS           AGE
my-service   10.244.1.5:8080     2m

Verification and Diagnostics

When a network issue appears, start with diagnostics that do not change state: inspect pod IPs, DNS resolution, and connectivity.

Check pod networking basics

List pods with their IPs and nodes:

kubectl get pods -o wide

Example output:

NAME                     READY   STATUS    RESTARTS   AGE   IP            NODE          NOMINATED NODE   READINESS GATES
web-748c8d9f5-8x7zr     1/1     Running   0          14m   10.244.1.5    worker-1      <none>           <none>
api-6f7b8c9d5-2p4tq     1/1     Running   0          14m   10.244.2.9    worker-2      <none>           <none>

If a pod is stuck in ContainerCreating, check events:

kubectl describe pod <pod-name> | tail -20

Look for messages like Failed to create pod sandbox or network plugin is not ready. That often points to CNI issues.

Test DNS resolution

DNS problems are common. Check if the cluster DNS service is up:

kubectl get service -n kube-system kube-dns

Or for CoreDNS:

kubectl get pods -n kube-system -l k8s-app=kube-dns

Expected output:

NAME                      READY   STATUS    RESTARTS   AGE
coredns-5d78c9869d-7m6n2   1/1     Running   0          3d
coredns-5d78c9869d-9x4k8   1/1     Running   0          3d

If pods are running, test resolution from inside a pod. Create a temporary debug pod:

kubectl run dns-test --image=busybox:1.28 --restart=Never -- sleep 3600

Then exec into it:

kubectl exec -it dns-test -- nslookup kubernetes.default

Expected output includes the cluster IP for the Kubernetes service:

Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes.default
Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local

If nslookup fails, check the CoreDNS logs:

kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50

Diagnose port and service issues

If a Service is not routing traffic, check that the selector matches pod labels. Suppose the Service has:

selector:
  app: web
  tier: frontend

List pods with those labels:

kubectl get pods -l app=web,tier=frontend

If no pods appear, the selector is wrong. Compare with actual pod labels:

kubectl get pods --show-labels

Also verify the Service's targetPort matches the container port. For instance, if the container listens on 8080 but the Service targetPort is 80, connections will be refused. Check the container port in the pod spec:

kubectl get pod <pod-name> -o jsonpath='{.spec.containers[*].ports}'

Expected output:

[{"containerPort":8080,"protocol":"TCP"}]

Failure Modes and Recovery

SSA introduces specific failure modes related to field ownership and conflicts. Knowing how to recover is essential.

Conflict during apply

Symptom: kubectl apply --server-side returns a conflict error indicating which fields are contested.

Example:

error: Apply failed with 2 conflicts: conflict with "ci-bot": .spec.ports, .spec.selector

This means another SSA manager, ci-bot, owns those fields. To proceed safely, first inspect who owns what:

kubectl get service my-service -o yaml --show-managed-fields

Look for the manager names and operations. If the conflict is from a CI system that no longer exists or should not own the field, you can force ownership:

kubectl apply --server-side --field-manager=network-fix --force-conflicts -f service.yaml

After forcing, verify ownership changed:

kubectl get service my-service -o yaml --show-managed-fields | grep -B 2 manager

You should see manager: network-fix for the fields you applied.

Partial apply due to failed validation

SSA validates the whole object before applying. If one field is invalid, the entire apply fails, leaving the object unchanged. To recover, fix the invalid field in your local file and reapply. You can preview validation errors with dry-run:

kubectl apply --server-side --dry-run=server -f broken-service.yaml

An error message appears:

The Service "my-service" is invalid: spec.ports[0].port: Invalid value: 0: must be between 1 and 65535, inclusive

Correct the port and apply again.

Ownership loss after force-conflicts

If you force ownership of a field, you may overwrite a change made by another team. Recovery requires identifying the previous value. If you have a backup (as recommended earlier), you can restore it:

kubectl apply --server-side --field-manager=network-fix -f service-backup-20240326-101112.yaml

If no backup exists, check the audit logs or the object's history in a Git repository if GitOps is used. Then reapply the correct configuration.

Common Pitfalls and How to Avoid Them

1. Not specifying a field manager

Why it happens: Users run kubectl apply --server-side without --field-manager, so Kubernetes uses a default manager name derived from the user and tool, such as kubectl. If multiple people apply the same object, they all become the same manager, leading to silent field ownership merges and harder conflict resolution.

How to avoid: Always set an explicit --field-manager value per team or workflow. For example, use --field-manager=platform-team for platform changes and --field-manager=security-team for network policy updates.

Recovery: If you already applied without a custom manager, you can reapply with a new manager and --force-conflicts to take ownership of the fields you intend to manage.

2. Using client-side apply and server-side apply interchangeably on the same object

Why it happens: Teams migrate gradually; some manifests are applied with kubectl apply (client-side) and others with --server-side. This creates mixed ownership and can lead to unexpected field deletions because client-side apply wipes server-side managed fields not present in the local file.

How to avoid: Standardize on SSA for all production objects. If you must mix, be aware that client-side apply uses the last-applied-configuration annotation, which does not know about server-side field ownership. Test in a staging cluster.

Recovery: If a field was deleted, restore it by re-applying with SSA and the correct field manager.

3. Ignoring network policy when diagnosing connectivity

Why it happens: When pods cannot connect, developers often assume DNS or service misconfiguration, overlooking a NetworkPolicy that denies traffic.

How to avoid: Always check for NetworkPolicies in the namespace:

kubectl get networkpolicy -n <namespace>

If any exist, describe them to see which pods they select and what traffic they allow:

kubectl describe networkpolicy <policy-name> -n <namespace>

Recovery: If a NetworkPolicy is blocking legitimate traffic, edit it to allow the required egress or ingress. Apply the change with SSA and verify connectivity.

4. Modifying resources without checking SSA field ownership

Why it happens: An operator edits a live object with kubectl edit or patches it without realizing SSA tracks field ownership. Later, a server-side apply fails with conflicts.

How to avoid: Before editing, check managed fields:

kubectl get <resource> <name> -o yaml --show-managed-fields

If a field is owned by another manager, coordinate or use --force-conflicts only when necessary.

Recovery: If a conflict occurs, inspect the ownership, then decide whether to force or abandon the change. If you force, document the change and notify the other manager's team.

Operations Checklist

Use this checklist before and after making network changes in an SSA-managed cluster.

Each owner is a single accountable person; they may delegate execution but remain responsible for the step.

Conclusion

Server-Side Apply gives you precise control over Kubernetes configuration, but it also demands a disciplined troubleshooting approach. By inventorying your environment, applying changes safely, diagnosing with read-only commands, and planning for common failure modes, you can resolve network issues without adding new ones.

The next step is to pick one low-risk verification from this guide, such as backing up a Service and checking its managed fields. Run the commands in a development namespace, observe the output, and compare with the expected results. Then explore how SSA ownership affects your team's existing workflows.

A reliable SSA workflow always makes failures visible, protects sensitive data, limits changes to the intended resource, and defines recovery verification before an incident forces a hurried decision.