## Intro When a Kubernetes Deployment or Service configuration drifts unexpectedly, Server-Side Apply (SSA) changes how you track and fix the problem. Instead of guessing which client last patched a field, SSA records field ownership and conflict metadata, making network misconfigurations easier to isolate and repair. This guide shows how to troubleshoot Kubernetes networking in an SSA-managed cluster using concrete commands, expected outputs, and recovery steps. The article is written for developers, DevOps engineers, and technical startup teams who operate Kubernetes clusters with SSA enabled. It covers the most common networking trouble spots: DNS resolution, port mismatches, inter-pod connectivity, and network policy behavior. Each section starts with a read-only observation of the current state, moves to a minimal change, and ends with a verification command that proves the fix worked. The operational principles throughout are simple: observe before you change, limit the blast radius, never put secrets in manifests, verify the result, and document a rollback path before you need it. ## Version and Environment Inventory Before touching any network configuration, establish the exact versions and components in play. Start by confirming the Kubernetes server version and API capabilities. ### Verify SSA support SSA became stable in Kubernetes 1.22, but earlier versions may enable it with a feature gate. Check the server version: ```bash kubectl version --short ``` Expected output looks like: ```text Client Version: v1.27.3 Kustomize Version: v5.0.1 Server Version: v1.27.3 ``` If the server version is 1.22 or higher, SSA is available by default. To confirm SSA is active for the `apply` command: ```bash kubectl apply --server-side --dry-run=server -f deployment.yaml ``` The absence of an error indicates support. For older clusters, check the feature gate: ```bash kubectl get --raw /metrics | grep server_apply ``` If the metric is absent or zero, SSA may not be enabled. ### Check core networking components Networking in Kubernetes depends on several components. Identify the CNI plugin in use: ```bash kubectl get pods -n kube-system | grep -E 'calico|flannel|weave|cilium|aws-node|azure-cni' ``` For example, a Calico-based cluster shows: ```text calico-node-4x7jz 1/1 Running 0 12d calico-kube-controllers-7c8c9c4d56-8p5vz 1/1 Running 0 12d ``` If you are using a managed service, the CNI may not run as pods. Check the node status for the CNI annotation: ```bash kubectl get nodes -o jsonpath='{.items[0].metadata.annotations}' ``` Look for keys like `cni.projectcalico.org/podIP` or similar. ### Inspect SSA-managed network objects List network-related resources that SSA manages: ```bash kubectl get services,endpoints,ingresses -o wide ``` Note the `MANAGED FIELDS` column if present; with `kubectl get` you can request it explicitly: ```bash kubectl get service my-service -o yaml | grep -A 5 managedFields ``` A typical entry shows the manager and operation: ```yaml managedFields: - manager: kubectl operation: Apply apiVersion: v1 time: "2024-03-26T10:11:12Z" fieldsType: FieldsV1 fieldsV1: f:spec: f:ports: {} ``` This confirms SSA is tracking the field. If a field is missing ownership, conflicts will surface later. ## Safe Configuration Path Changing network settings with SSA requires a controlled sequence: record current state, apply a minimal change, verify, and revert if needed. ### Capture the current state Before any change, dump the live object to a backup file using `kubectl get` with output in YAML: ```bash kubectl get service my-service -o yaml > service-backup-$(date +%Y%m%d-%H%M%S).yaml ``` Also run `kubectl diff` to preview the exact changes SSA would make without applying: ```bash kubectl diff -f updated-service.yaml ``` If the diff is empty, your local file matches the live object. If you see lines starting with `+` or `-`, those are the changes. ### Apply a minimal SSA change Suppose you need to change a service port from 80 to 8080. The original YAML snippet: ```yaml spec: ports: - name: http port: 80 targetPort: 80 ``` Modify only the `targetPort` field: ```yaml spec: ports: - name: http port: 80 targetPort: 8080 ``` Apply with server-side apply and specify the field manager explicitly: ```bash kubectl apply --server-side --field-manager=network-fix -f updated-service.yaml ``` Expected output: ```text service/my-service serverside-applied ``` If another manager owns the `targetPort` field, you will see a conflict: ```text error: Apply failed with 1 conflict: conflict with "other-manager": .spec.ports[0].targetPort ``` To resolve, you can force ownership with `--force-conflicts`, but only after verifying it is safe: ```bash kubectl apply --server-side --field-manager=network-fix --force-conflicts -f updated-service.yaml ``` Use `--force-conflicts` sparingly; it transfers field ownership to you and may overwrite intentional changes by another team. ### Verify the change After applying, check the live object: ```bash kubectl get service my-service -o yaml | grep -A 3 ports ``` You should see `targetPort: 8080`. Also confirm the endpoints updated: ```bash kubectl get endpoints my-service ``` Expected output shows the pod IP and new port: ```text NAME ENDPOINTS AGE my-service 10.244.1.5:8080 2m ``` ## Verification and Diagnostics When a network issue appears, start with diagnostics that do not change state: inspect pod IPs, DNS resolution, and connectivity. ### Check pod networking basics List pods with their IPs and nodes: ```bash kubectl get pods -o wide ``` Example output: ```text NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES web-748c8d9f5-8x7zr 1/1 Running 0 14m 10.244.1.5 worker-1 api-6f7b8c9d5-2p4tq 1/1 Running 0 14m 10.244.2.9 worker-2 ``` If a pod is stuck in `ContainerCreating`, check events: ```bash kubectl describe pod | tail -20 ``` Look for messages like `Failed to create pod sandbox` or `network plugin is not ready`. That often points to CNI issues. ### Test DNS resolution DNS problems are common. Check if the cluster DNS service is up: ```bash kubectl get service -n kube-system kube-dns ``` Or for CoreDNS: ```bash kubectl get pods -n kube-system -l k8s-app=kube-dns ``` Expected output: ```text NAME READY STATUS RESTARTS AGE coredns-5d78c9869d-7m6n2 1/1 Running 0 3d coredns-5d78c9869d-9x4k8 1/1 Running 0 3d ``` If pods are running, test resolution from inside a pod. Create a temporary debug pod: ```bash kubectl run dns-test --image=busybox:1.28 --restart=Never -- sleep 3600 ``` Then exec into it: ```bash kubectl exec -it dns-test -- nslookup kubernetes.default ``` Expected output includes the cluster IP for the Kubernetes service: ```text Server: 10.96.0.10 Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local Name: kubernetes.default Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local ``` If `nslookup` fails, check the CoreDNS logs: ```bash kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50 ``` ### Diagnose port and service issues If a Service is not routing traffic, check that the selector matches pod labels. Suppose the Service has: ```yaml selector: app: web tier: frontend ``` List pods with those labels: ```bash kubectl get pods -l app=web,tier=frontend ``` If no pods appear, the selector is wrong. Compare with actual pod labels: ```bash kubectl get pods --show-labels ``` Also verify the Service's targetPort matches the container port. For instance, if the container listens on 8080 but the Service targetPort is 80, connections will be refused. Check the container port in the pod spec: ```bash kubectl get pod -o jsonpath='{.spec.containers[*].ports}' ``` Expected output: ```text [{"containerPort":8080,"protocol":"TCP"}] ``` ## Failure Modes and Recovery SSA introduces specific failure modes related to field ownership and conflicts. Knowing how to recover is essential. ### Conflict during apply Symptom: `kubectl apply --server-side` returns a conflict error indicating which fields are contested. Example: ```text error: Apply failed with 2 conflicts: conflict with "ci-bot": .spec.ports, .spec.selector ``` This means another SSA manager, `ci-bot`, owns those fields. To proceed safely, first inspect who owns what: ```bash kubectl get service my-service -o yaml --show-managed-fields ``` Look for the `manager` names and operations. If the conflict is from a CI system that no longer exists or should not own the field, you can force ownership: ```bash kubectl apply --server-side --field-manager=network-fix --force-conflicts -f service.yaml ``` After forcing, verify ownership changed: ```bash kubectl get service my-service -o yaml --show-managed-fields | grep -B 2 manager ``` You should see `manager: network-fix` for the fields you applied. ### Partial apply due to failed validation SSA validates the whole object before applying. If one field is invalid, the entire apply fails, leaving the object unchanged. To recover, fix the invalid field in your local file and reapply. You can preview validation errors with dry-run: ```bash kubectl apply --server-side --dry-run=server -f broken-service.yaml ``` An error message appears: ```text The Service "my-service" is invalid: spec.ports[0].port: Invalid value: 0: must be between 1 and 65535, inclusive ``` Correct the port and apply again. ### Ownership loss after force-conflicts If you force ownership of a field, you may overwrite a change made by another team. Recovery requires identifying the previous value. If you have a backup (as recommended earlier), you can restore it: ```bash kubectl apply --server-side --field-manager=network-fix -f service-backup-20240326-101112.yaml ``` If no backup exists, check the audit logs or the object's history in a Git repository if GitOps is used. Then reapply the correct configuration. ## Common Pitfalls and How to Avoid Them ### 1. Not specifying a field manager Why it happens: Users run `kubectl apply --server-side` without `--field-manager`, so Kubernetes uses a default manager name derived from the user and tool, such as `kubectl`. If multiple people apply the same object, they all become the same manager, leading to silent field ownership merges and harder conflict resolution. How to avoid: Always set an explicit `--field-manager` value per team or workflow. For example, use `--field-manager=platform-team` for platform changes and `--field-manager=security-team` for network policy updates. Recovery: If you already applied without a custom manager, you can reapply with a new manager and `--force-conflicts` to take ownership of the fields you intend to manage. ### 2. Using client-side apply and server-side apply interchangeably on the same object Why it happens: Teams migrate gradually; some manifests are applied with `kubectl apply` (client-side) and others with `--server-side`. This creates mixed ownership and can lead to unexpected field deletions because client-side apply wipes server-side managed fields not present in the local file. How to avoid: Standardize on SSA for all production objects. If you must mix, be aware that client-side apply uses the `last-applied-configuration` annotation, which does not know about server-side field ownership. Test in a staging cluster. Recovery: If a field was deleted, restore it by re-applying with SSA and the correct field manager. ### 3. Ignoring network policy when diagnosing connectivity Why it happens: When pods cannot connect, developers often assume DNS or service misconfiguration, overlooking a NetworkPolicy that denies traffic. How to avoid: Always check for NetworkPolicies in the namespace: ```bash kubectl get networkpolicy -n ``` If any exist, describe them to see which pods they select and what traffic they allow: ```bash kubectl describe networkpolicy -n ``` Recovery: If a NetworkPolicy is blocking legitimate traffic, edit it to allow the required egress or ingress. Apply the change with SSA and verify connectivity. ### 4. Modifying resources without checking SSA field ownership Why it happens: An operator edits a live object with `kubectl edit` or patches it without realizing SSA tracks field ownership. Later, a server-side apply fails with conflicts. How to avoid: Before editing, check managed fields: ```bash kubectl get -o yaml --show-managed-fields ``` If a field is owned by another manager, coordinate or use `--force-conflicts` only when necessary. Recovery: If a conflict occurs, inspect the ownership, then decide whether to force or abandon the change. If you force, document the change and notify the other manager's team. ## Operations Checklist Use this checklist before and after making network changes in an SSA-managed cluster. | Step | Action | Command / Detail | Owner | Review Frequency | |------|--------|------------------|-------|------------------| | 1 | Confirm Kubernetes version and SSA support | `kubectl version --short` and `kubectl apply --server-side --dry-run=server -f test.yaml` | Platform Lead (e.g., Priya Shah) | Quarterly or after cluster upgrade | | 2 | Back up all affected network objects | `kubectl get svc,ep,ingress -o yaml > backup-$(date +%Y%m%d).yaml` | On-call engineer | Before each change | | 3 | Preview changes with diff | `kubectl diff -f updated.yaml` | Change requester | Every change | | 4 | Apply with explicit field manager | `kubectl apply --server-side --field-manager=team-name -f updated.yaml` | Change requester | Every change | | 5 | Verify the change and endpoints | `kubectl get service -o yaml` and `kubectl get endpoints ` | On-call engineer | Immediately after apply | | 6 | Test DNS resolution if relevant | `kubectl exec -it -- nslookup ` | Network SME (e.g., Marcus Chen) | After service creation or DNS changes | | 7 | Check NetworkPolicies for unintended blocks | `kubectl get networkpolicy -A` | Security Engineer (e.g., Aisha Williams) | Weekly or on connectivity incidents | | 8 | Document rollback procedure | Keep backup file and note the command: `kubectl apply --server-side --field-manager= -f backup.yaml` | Change requester | Each change | Each owner is a single accountable person; they may delegate execution but remain responsible for the step. ## Conclusion Server-Side Apply gives you precise control over Kubernetes configuration, but it also demands a disciplined troubleshooting approach. By inventorying your environment, applying changes safely, diagnosing with read-only commands, and planning for common failure modes, you can resolve network issues without adding new ones. The next step is to pick one low-risk verification from this guide, such as backing up a Service and checking its managed fields. Run the commands in a development namespace, observe the output, and compare with the expected results. Then explore how SSA ownership affects your team's existing workflows. A reliable SSA workflow always makes failures visible, protects sensitive data, limits changes to the intended resource, and defines recovery verification before an incident forces a hurried decision.