## Intro

Server-side apply (SSA) changes how Kubernetes manages object configuration. Instead of relying on `kubectl` to compute a three-way merge on the client, the Kubernetes API server performs the merge and records field ownership. This shift matters for CI/CD automation because it removes ambiguity about who changed a field and prevents accidental overwrites between controllers, GitOps tools, and manual edits.

This guide focuses on operational use: enabling SSA, building a pipeline that applies manifests safely, detecting conflicts before they break a deployment, and rolling back when a change goes wrong. It includes the exact commands, expected outputs, and failure signals you will see in a real cluster.

You will need a running Kubernetes cluster (version 1.16+ for basic SSA, 1.18+ for full field management), `kubectl` version 1.22+ recommended, and a namespace where you can safely test.

## Version and Environment Inventory

Before using SSA in a pipeline, confirm that your cluster, client, and API resources support the feature. Run these read-only checks first:

```bash
kubectl version --short
```

Expected output shows both client and server versions. For SSA to work reliably, the server version should be at least 1.18. The client version matters less because the server does the merge, but using a recent `kubectl` ensures the `--server-side` flag is available.

Next, verify that the API server has the `ServerSideApply` feature gate enabled (it is enabled by default in all supported versions, but a custom cluster might disable it):

```bash
kubectl get --raw /metrics | grep server_side_apply
```

If you see metrics like `apiserver_request_total` with `verb="PATCH"` and `dry_run=""`, the feature is active. You can also inspect the API server's enabled admission plugins:

```bash
kubectl get --raw /healthz?verbose | grep -i server-side-apply
```

A healthy response includes `server-side-apply` in the list of checks.

### Prerequisites for CI/CD

- **Kubernetes cluster**: v1.18+ preferred. Some advanced features like `managedFields` are stable in v1.22+.
- **kubectl client**: v1.22+ to avoid client-side merge attempts. Older clients may try to send a client-side apply patch unless you force `--server-side`.
- **RBAC permissions**: The CI service account must have `patch` and `update` verbs on the target resources, plus `get` and `list` for conflict detection.
- **Namespace isolation**: Use a dedicated namespace for testing SSA behavior before enabling it for production workloads.

Run this small test to confirm SSA works end-to-end:

Create a ConfigMap with SSA:

```bash
kubectl apply --server-side -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: ssa-test
  namespace: default
data:
  key1: value1
EOF
```

Expected output:

```
configmap/ssa-test serverside-applied
```

Inspect the `managedFields` to see ownership:

```bash
kubectl get configmap ssa-test -o yaml --show-managed-fields | grep -A 5 managedFields
```

You should see an entry with `manager: kubectl` and `operation: Apply`. That confirms SSA recorded the field manager.

Clean up the test object with:

```bash
kubectl delete configmap ssa-test
```

## Safe Configuration Path

SSA works best when every actor in the system uses it consistently. If some tools still use client-side apply (`kubectl apply` without `--server-side`), field ownership can become fragmented, leading to conflicts.

### Standardizing on SSA for Deployments

For a Deployment managed by your CI pipeline, always apply with `--server-side`. Here is a minimal strategy:

1. Store manifests in Git as the source of truth.
2. Use a CI step that applies manifests with a stable manager name. The manager name is how SSA tracks ownership. Use a name that identifies your pipeline, like `ci-deployer`.
3. Include `--force-conflicts` only after reviewing that the conflicts are expected. In normal operation, you should not force conflicts.

Example CI snippet (GitHub Actions, but any CI works):

```yaml
- name: Apply with SSA
  run: |
    kubectl apply --server-side \
      --field-manager=ci-deployer \
      --validate=false \
      -f deploy/
```

Why `--validate=false`? SSA already performs structural schema validation on the server. Skipping client-side validation can speed up large apply operations and avoids version skew issues between client and server schemas. The server still validates.

### Controlling Field Ownership

SSA introduces `managedFields`, which tracks which manager owns which fields. When two managers try to modify the same field, the API server returns a conflict error. This is a safety feature, not a bug. To avoid conflicts in CI, have a single manager for each resource type. For example, your CI pipeline could be the sole manager of `Deployment` objects, while an autoscaler manages `replicas` separately. SSA allows different managers to own different fields.

To inspect ownership in detail:

```bash
kubectl get deployment my-app -o yaml --show-managed-fields | grep -A 20 managedFields
```

Look for entries like:

```yaml
- apiVersion: apps/v1
  fieldsType: FieldsV1
  fieldsV1:
    f:spec:
      f:replicas: {}
      f:template:
        f:spec:
          f:containers:
            k:{"name":"app"}:
              f:image: {}
              f:resources: {}
  manager: ci-deployer
  operation: Apply
  time: "2025-01-10T12:00:00Z"
```

This shows that `ci-deployer` owns the fields for `replicas`, container `image`, and `resources`. If your autoscaler later tries to change `replicas`, it would need to use SSA with its own manager and might conflict if it also claims ownership of that field.

### Rolling Out Changes Incrementally

Use `kubectl apply --server-side --dry-run=server` to see the effect without persisting:

```bash
kubectl apply --server-side --dry-run=server -f deploy/my-app.yaml
```

Expected output includes a note that the request was dry-run and no changes were applied. This is useful in a pull request review step to surface potential conflicts before merge.

## Verification and Diagnostics

After applying a change, verify that the cluster state matches your intent and that no unexpected field ownership changes occurred.

### Checking Applied Resources

For a Deployment:

```bash
kubectl rollout status deployment/my-app --timeout=120s
```

Expected success:

```
deployment "my-app" successfully rolled out
```

If the rollout stalls, dig into pod events:

```bash
kubectl describe deployment my-app
```

Look for conditions like `Progressing` and `Available`. Also check the replicaset:

```bash
kubectl get replicasets -l app=my-app
```

If pods are crash-looping, get logs from the previous container instance:

```bash
kubectl logs deployment/my-app --previous --tail=50
```

### Diagnosing SSA Conflicts

A common error during SSA apply looks like:

```
error: Apply failed with 1 conflict: conflict with "kubectl-client-side-apply" using apps/v1: .spec.template.spec.containers[name="app"].image
```

This means another manager (here, `kubectl-client-side-apply`) already owns the `image` field, and your manager tried to change it. To view the current owners of that field, use:

```bash
kubectl get deployment my-app -o yaml | grep -B 5 -A 10 'f:image'
```

The output shows the field path and the manager that owns it. You can then decide:

- If the conflicting manager is outdated, you can take ownership by applying with `--force-conflicts`. But be cautious: this overwrites the other manager's intent.
- If the conflict is valid, coordinate with the other manager or adjust your pipeline to not modify that field.

### Debugging with Server-Side Dry Run

Use `--dry-run=server` with a specific field manager to see ownership changes:

```bash
kubectl apply --server-side --dry-run=server --field-manager=ci-deployer -f deploy/my-app.yaml -o yaml | grep -A 10 managedFields
```

This shows what the managedFields would look like after the apply without actually applying. It helps you spot if your manager would take over unintended fields.

## Failure Modes and Recovery

SSA introduces new failure modes, especially around field ownership and conflicts. Planning for recovery before a failure happens reduces downtime and confusion.

### Failure: Apply Conflict Prevents Deployment

**Symptom**: CI pipeline fails with a conflict error as above.

**Why it happens**: Two managers claim the same field. This often occurs when you first adopt SSA and existing resources have `kubectl-client-side-apply` as manager.

**Recovery**: Migrate ownership deliberately.

1. Identify the conflicting field(s) from the error.
2. Decide who should own those fields going forward.
3. If your CI manager should own them, run a one-time command to take ownership by forcing the conflict:

```bash
kubectl apply --server-side --force-conflicts --field-manager=ci-deployer -f deploy/my-app.yaml
```

4. Verify the new ownership:

```bash
kubectl get deployment my-app -o yaml | grep -A 10 managedFields
```

5. Ensure all future applies use the same `--field-manager` and avoid `--force-conflicts` unless absolutely necessary.

### Failure: Unexpected Field Reset

**Symptom**: A field you did not intend to change reverts to a default or disappears.

**Why it happens**: With SSA, if a manager applies a manifest that omits a field it previously owned, the server treats that as an instruction to remove the field. For example, if your manifest no longer includes `resources.limits` but your manager previously owned it, SSA will remove the limits.

**Recovery**: Prevent this by always including the full desired spec in your manifests. Use a templating tool like Kustomize or Helm to maintain complete specs. Also review diffs in CI before applying. If a reset already occurred, restore the field by applying a manifest with the correct value using the same field manager.

### Failure: Partial Apply Due to Validation Error

**Symptom**: `kubectl apply --server-side` reports a validation error for one resource, but other resources in the same batch may have already been applied.

**Why it happens**: Unlike client-side apply which validates all before sending, server-side apply processes resources individually. Some may succeed before the failing one is reached.

**Recovery**: Use `kubectl apply --server-side --dry-run=server` on the full directory before the real apply. That catches validation errors without side effects. If a partial apply occurs, check which resources were applied:

```bash
kubectl get all -n my-app -o name
```

Then re-apply the corrected manifests. Since SSA is idempotent, re-applying is safe.

### Rollback Strategies

Rolling back with SSA is similar to client-side apply, but you must ensure the rollback manifest includes all fields your manager owns to avoid unintentional resets.

**Option 1: Re-apply a previous known-good manifest**

```bash
kubectl apply --server-side --field-manager=ci-deployer -f deploy/known-good/my-app.yaml
```

This restores the spec to a previous state. Ensure the manifest is complete.

**Option 2: Use `kubectl rollout undo`**

`kubectl rollout undo` reverts the Deployment to the previous revision. However, it uses the server-side stored configuration, which includes SSA-managed fields. It does not change field ownership; the manager that made the change still owns the fields. So after undo, the fields are in the older state but still managed by the same manager. This is often sufficient for quick rollback.

```bash
kubectl rollout undo deployment/my-app
```

To verify:

```bash
kubectl rollout status deployment/my-app
```

**Option 3: Git revert plus apply**

In a GitOps flow, reverting the commit and letting the pipeline apply with SSA is the cleanest, because it keeps the Git history as the audit trail and uses the same manager. This is the recommended approach for CI/CD.

## Operations Checklist

Use this checklist before promoting SSA to production CI/CD. Each item has an owner and a review cadence.

| # | Item | Owner | Frequency |
|---|------|-------|-----------|
| 1 | Confirm all production resources are managed with `--server-side` and a consistent `--field-manager` name. | DevOps lead (e.g., Priya Shah) | Monthly |
| 2 | Verify no `kubectl-client-side-apply` managers remain on critical resources. | Release engineer (e.g., Marco Diaz) | Weekly |
| 3 | Run `--dry-run=server` on all manifests in CI before merge to catch conflicts early. | CI pipeline (automatic) | Every PR |
| 4 | Test rollback procedure in a staging namespace at least once per sprint. | QA engineer (e.g., Lena Park) | Biweekly |
| 5 | Review RBAC permissions for CI service accounts to ensure they have only necessary `patch` and `update` scopes. | Security officer (e.g., Ahmed Khan) | Quarterly |
| 6 | Audit `managedFields` for any unexpected manager ownership or field drift. | Platform engineer (e.g., Chris O'Brien) | Monthly |

### Command Reference

Run these commands regularly to verify SSA hygiene:

List all managers across resources in a namespace:

```bash
kubectl get configmaps,secrets,deployments -n my-app -o yaml | grep 'manager:' | sort | uniq -c
```

Show fields owned by a specific manager:

```bash
kubectl get deployment my-app -o yaml | yq eval '.metadata.managedFields[] | select(.manager == "ci-deployer") | .fieldsV1' -
```

(If you don't have `yq`, use `grep` with context.)

Detect resources not using SSA by checking for the absence of managed fields:

```bash
kubectl get deployment -o json | jq -r '.items[] | select(.metadata.managedFields | length == 0) | .metadata.name'
```

This returns names of resources that were likely created without SSA.

## Common Pitfalls and How to Avoid Them

### Pitfall 1: Mixing Client-Side and Server-Side Apply

**The mistake**: Using `kubectl apply` (client-side) for some updates and `kubectl apply --server-side` for others on the same resource. The client-side apply uses a different merge algorithm and manager name, causing ownership fragmentation and mysterious conflicts.

**Why it happens**: Teams adopt SSA gradually and forget which commands are used in different scripts or documentation.

**How to avoid**: Enforce a policy: once a resource is managed with SSA, all subsequent modifications must use SSA with a consistent `--field-manager`. Update all scripts and docs. Add a CI check that greps for `kubectl apply` without `--server-side` in pipeline definitions.

**Recovery**: If mixed usage occurs, choose a single manager and re-apply the desired state with `--server-side --force-conflicts` once to consolidate ownership, then stick to SSA thereafter.

### Pitfall 2: Omitting Fields in Manifests Leads to Silent Deletion

**The mistake**: Removing a field (like a resource limit) from a manifest and applying with SSA deletes that field from the live object because the server interprets the omission as intent to remove.

**Why it happens**: With client-side apply, omitted fields often remain unchanged, so developers assume the same with SSA.

**How to avoid**: Always generate manifests from complete templates. Use Kustomize overlays or Helm values to ensure all needed fields are present. In CI, diff the applied manifest against the live object to show removed fields (e.g., `kubectl diff -f deploy/` before apply).

**Recovery**: If a field was unintentionally removed, restore it by applying a manifest that includes that field with the correct value, using the same field manager. The field will reappear and ownership will remain with that manager.

### Pitfall 3: Using `--force-conflicts` Routinely

**The mistake**: Adding `--force-conflicts` to every apply command to avoid dealing with conflict errors. This masks real conflicts and can overwrite changes made by other controllers or users.

**Why it happens**: Developers see conflict errors as nuisances and force their way through.

**How to avoid**: Never use `--force-conflicts` in automated pipelines. Investigate every conflict. If a conflict is legitimate, coordinate field ownership. Only use `--force-conflicts` in a controlled, one-time migration step.

**Recovery**: If you have been forcing conflicts, review recent changes to see if any unintended overwrites occurred. Use audit logs and Git history to restore correct values.

### Pitfall 4: Not Setting a Consistent Field Manager

**The mistake**: Different CI jobs or team members use different manager names (e.g., `kubectl`, `deploy-script`, `jenkins`), leading to multiple owners for different subsets of fields. This makes it hard to know who controls what and can cause conflicts when one manager tries to modify fields owned by another.

**Why it happens**: The `--field-manager` flag is often omitted, and the default manager is `kubectl` client-side or the user's name for server-side. In CI, the user may be the service account, which is not descriptive.

**How to avoid**: Explicitly set `--field-manager` to a stable identifier for your pipeline (e.g., `ci-deployer`). Document it and include it in all apply commands.

**Recovery**: Migrate to a single manager by applying all resources with the desired manager using `--force-conflicts` once, then enforce consistency.

### Pitfall 5: Ignoring `managedFields` in Debugging

**The mistake**: When a resource behaves unexpectedly after an SSA apply, engineers don't inspect `managedFields` to understand who changed what. They may scatter blame or make incorrect fixes.

**Why it happens**: `managedFields` is verbose and often omitted from default `kubectl get` output, so it is easily overlooked.

**How to avoid**: Include `managedFields` inspection in your debugging checklist. Use `kubectl get <resource> -o yaml | grep -A 30 managedFields` to see recent operations. Tools like `kubectl fields` (if available) can simplify this.

**Recovery**: If you find unexpected field ownership, trace the manager to its source and correct the process.

## Conclusion

Server-side apply brings rigorous field-level ownership tracking to Kubernetes configuration, which is a boon for CI/CD automation. It prevents accidental overwrites, provides clear conflict signals, and enables multiple controllers to safely manage different parts of a resource. However, it requires discipline: consistent field manager naming, complete manifests, and a thorough understanding of how omissions and conflicts behave.

Start with low-risk resources like ConfigMaps and Secrets, then expand to Deployments and CRDs. Use dry-run verification in CI, monitor `managedFields` for drift, and have a tested rollback procedure. With these practices, your Kubernetes pipeline will be more predictable and less prone to configuration drift.

As a next step, pick one existing deployment in your cluster, inspect its `managedFields`, and try applying a small change with `--server-side` in a staging environment. Observe how ownership is recorded and how conflicts surface. This hands-on experiment will build the confidence needed to adopt SSA across your pipelines.