Kubernetes Server-Side Apply Architecture Explained with Practical Examples

Introduction

Kubernetes Server-Side Apply (SSA) is a feature that shifts the logic of merging configuration changes from the client (kubectl) to the Kubernetes API server. In traditional client-side apply, kubectl calculates a three-way merge patch locally and sends it to the API server. With server-side apply, the API server itself stores the applied configuration in a dedicated field (metadata.managedFields) and resolves conflicts based on field ownership. This change improves consistency, enables better conflict detection, and allows multiple controllers to safely update different fields of the same object.

This article explains the architecture of Kubernetes Server-Side Apply with practical examples. We focus on operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify results, and document recovery paths. You will learn how to enable and use server-side apply, inspect field management, handle conflicts, and operationalize SSA in your cluster.

Version and Environment Inventory

Before using server-side apply, verify that your Kubernetes cluster and client tools support it. Server-side apply became beta in Kubernetes 1.16 and is generally available and enabled by default from Kubernetes 1.18 onward. For production use, run Kubernetes 1.18 or later. Ensure your kubectl version is at least 1.18 to use the --server-side flag.

Check your cluster version:

kubectl version --short

Expected output includes the server version, for example Server Version: v1.25.4.

Check the API server feature gate for server-side apply (should be enabled by default):

kubectl get --raw /metrics | grep server_side_apply

You should see a metric like apiserver_request_total{...,resource="apply"...} indicating that the apply endpoint is active.

Kubernetes Server-Side Apply uses the apply verb on the API server. The endpoint for patching with apply is PATCH /api/v1/namespaces/{namespace}/configmaps/{name}?fieldManager=my-manager&force=false with content type application/apply-patch+yaml. The API server stores the applied configuration in the object's metadata.managedFields. Each entry in managedFields tracks the manager name, operation (Apply or Update), the API version used, and the fields owned by that manager.

To see how server-side apply tracks ownership, create a Deployment using server-side apply and inspect managedFields. First, create a namespace:

kubectl create namespace ssa-demo

Then create a Deployment with server-side apply:

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: ssa-demo
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:1.21
          ports:
            - containerPort: 80

Apply with server-side:

kubectl apply --server-side -f deployment.yaml --field-manager=deployment-manager

Expected output:

deployment.apps/nginx-deployment serverside-applied

Now inspect the managedFields. Use the following command to view the YAML representation with managedFields:

kubectl get deployment nginx-deployment -n ssa-demo -o yaml

In the output, under metadata.managedFields, you will see an entry with manager: deployment-manager, operation: Apply, and the fieldsV1 structure describing which fields this manager owns. For example:

managedFields:
- apiVersion: apps/v1
  fieldsType: FieldsV1
  fieldsV1:
    f:spec:
      f:replicas: {}
      f:selector: {}
      f:template:
        f:metadata:
          f:labels:
            f:app: {}
        f:spec:
          f:containers:
            k:{"name":"nginx"}:
              f:image: {}
              f:name: {}
              f:ports: {}
  manager: deployment-manager
  operation: Apply
  time: "2023-10-01T12:00:00Z"

This shows that deployment-manager owns the fields it set. If another manager tries to change a field owned by deployment-manager without force, the API server will return a conflict error.

To list all managers and their operations for the deployment, use:

kubectl get deployment nginx-deployment -n ssa-demo -o json | jq '.metadata.managedFields[] | {manager: .manager, operation: .operation}'

Expected output:

{
  "manager": "deployment-manager",
  "operation": "Apply"
}

Safe Configuration Path

Server-side apply introduces a conflict resolution mechanism based on field ownership. When a manager applies a change to a field that is already owned by another manager, the API server rejects the change with a conflict error unless the force flag is set. Using force transfers ownership of the field to the new manager, which can overwrite values unintentionally. Therefore, it is safer to avoid force in production and instead resolve conflicts explicitly.

Consider a scenario where two managers are trying to configure the same Deployment. Manager A (deployment-manager) owns spec.replicas. Manager B (autoscaler) wants to change spec.replicas from 2 to 3. If Manager B applies without force, the API server returns a conflict. Let's simulate this.

Create a second YAML file that attempts to update the replicas:

# deployment-replicas-update.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: ssa-demo
spec:
  replicas: 3

Apply as a different field manager:

kubectl apply --server-side -f deployment-replicas-update.yaml --field-manager=autoscaler

Expected output includes a conflict error similar to:

error: Apply failed with 1 conflict: conflict with "deployment-manager": .spec.replicas

This error indicates that autoscaler cannot change spec.replicas because it is owned by deployment-manager. To resolve the conflict, you have a few options:

1. Use force to take ownership (not recommended for production unless you are certain):

   kubectl apply --server-side -f deployment-replicas-update.yaml --field-manager=autoscaler --force

Expected output: deployment.apps/nginx-deployment serverside-applied. After this, autoscaler owns spec.replicas, and the value is set to 3.

1. Coordinate field ownership: Ensure that only one manager is responsible for a given field. For example, the autoscaler should not try to set replicas directly; instead, it should use a custom controller or annotation.

1. Use client-side apply for this update, which does not enforce field ownership and uses the last-applied-configuration annotation for merging. However, this can lead to silent overwrites.

To verify the current ownership after the conflict, use:

kubectl get deployment nginx-deployment -n ssa-demo -o json | jq '.metadata.managedFields[] | select(.manager=="autoscaler") | .fieldsV1'

If the conflict was not forced, the output will be empty. If forced, you will see f:spec with f:replicas: {}.

Best practices for safe configuration with SSA:

• Always specify --field-manager to identify the manager uniquely.
• Use separate field managers for different components to avoid unintentional conflicts.
• Monitor conflicts by checking the API server audit logs or metrics.
• Test changes in a non-production cluster first.
• Use --dry-run=server to preview changes without applying:

  kubectl apply --server-side -f deployment-replicas-update.yaml --field-manager=autoscaler --dry-run=server

Expected output: deployment.apps/nginx-deployment serverside-applied (dry run) and the conflict will be reported if any.

Verification and Diagnostics

After applying changes with server-side apply, verify that the resource reflects the intended state and that managedFields are updated correctly. Use read-only commands to observe the current state before and after changes.

Check the Deployment status:

kubectl rollout status deployment/nginx-deployment -n ssa-demo

Expected output:

deployment "nginx-deployment" successfully rolled out

Inspect the managedFields to ensure the correct manager owns the relevant fields:

kubectl get deployment nginx-deployment -n ssa-demo -o json | jq '.metadata.managedFields'

Example output showing two managers:

[
  {
    "apiVersion": "apps/v1",
    "fieldsType": "FieldsV1",
    "fieldsV1": {
      "f:spec": {
        "f:replicas": {},
        "f:selector": {},
        "f:template": {}
      }
    },
    "manager": "deployment-manager",
    "operation": "Apply",
    "time": "2023-10-01T12:00:00Z"
  },
  {
    "apiVersion": "apps/v1",
    "fieldsType": "FieldsV1",
    "fieldsV1": {
      "f:spec": {
        "f:replicas": {}
      }
    },
    "manager": "autoscaler",
    "operation": "Apply",
    "time": "2023-10-01T12:05:00Z"
  }
]

Note that after the forced apply, deployment-manager still owns spec.selector and spec.template, while autoscaler owns spec.replicas. If the deployment-manager later applies a change to replicas, it will get a conflict unless it uses force or relinquishes ownership.

To view the applied configuration for a specific manager, use the kubectl apply view-last-applied command (only for server-side apply):

kubectl apply view-last-applied deployment nginx-deployment -n ssa-demo

Expected output shows the YAML that was last applied by the kubectl field manager (or you can specify --field-manager). For example:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: ssa-demo
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:1.21
          ports:
            - containerPort: 80

Diagnose issues by checking events:

kubectl describe deployment nginx-deployment -n ssa-demo

Look for events related to apply operations if conflicts occur. The API server also emits metrics for apply requests. Check the metric apiserver_request_total filtered by verb="PATCH" and code to see apply requests and success/failure counts:

kubectl get --raw /metrics | grep 'apiserver_request_total{.*verb="PATCH".*code="409"'

A non-zero count for 409 (Conflict) indicates that apply conflicts have occurred. This can help in monitoring.

Failure Modes and Recovery

Understanding common failure modes with server-side apply helps in quick recovery. Here are scenarios and how to recover.

Failure: Conflict Error on Apply

Symptom: kubectl apply --server-side returns a conflict error similar to:

error: Apply failed with 1 conflict: conflict with "deployment-manager": .spec.replicas

Cause: Another manager owns the field you are trying to modify.

Recovery:

• Identify the current owner using kubectl get <resource> -o yaml and inspecting managedFields.
• If the change is necessary, coordinate with the owner to relinquish ownership. You can force ownership with --force, but this may overwrite the field value.
• Alternatively, update the field using a regular update (PUT) or patch (strategic merge patch) as the ownership may not be enforced for those operations (though it is not recommended because it bypasses SSA semantics).

Failure: Lost Field Ownership After Force

Symptom: After forcing an apply, the previous manager's updates to the field are ignored or cause conflicts.

Cause: The force operation transferred ownership to the new manager, and the previous manager no longer has rights to that field.

Recovery:

• The previous manager can also use force to regain ownership, but this can cause a tug-of-war. Fix the root cause by assigning clear field ownership.
• To explicitly relinquish ownership of a field, apply a patch with $setElementOrder? Actually, there is no direct relinquish; you can apply an empty value for the field and then let the other manager take over. Or, you can delete the manager's entry from managedFields using a raw API call (not recommended for production as it can cause unexpected behavior).

Failure: Incorrect Field Manager Name

Symptom: Multiple users or controllers use the same field manager name (e.g., kubectl by default), causing unexpected conflicts.

Cause: Default --field-manager is kubectl for kubectl apply --server-side. If multiple automation scripts use kubectl without specifying a manager, they all share ownership.

Recovery:

• Always specify a unique --field-manager for each logical actor. For CI/CD, use the pipeline name or service account name.
• To correct existing ownership, re-apply with the correct manager and force if necessary, but ensure consistency going forward.

Failure: Unsupported Resource for Server-Side Apply

Symptom: Applying to a custom resource or built-in resource returns an error like:

error: Apply failed with 1 conflict: conflicts with "..." using "...": .metadata.managedFields

Cause: Some resources may not fully support SSA or have validation issues.

Recovery:

• Check the resource's API documentation for SSA support. Most built-in resources support SSA.
• If the resource is a CustomResourceDefinition, ensure the CRD has spec.preserveUnknownFields: false (or uses structural schema) since SSA requires structured types.

Operations Checklist

Use this checklist to operationalize server-side apply in your environment.

1. Verify cluster and client versions: Ensure Kubernetes >=1.18 and kubectl >=1.18. Use kubectl version --short.
2. Choose field manager names: Define a naming convention for field managers. For example:

• platform-team for infrastructure deployments
• autoscaler for HorizontalPodAutoscaler controller
• ci-cd-pipeline for application deployments

1. Apply resources with server-side for new resources:

   kubectl apply --server-side -f manifest.yaml --field-manager=platform-team

1. Inspect managedFields after apply: Run kubectl get <resource> -o yaml and verify the manager and operation.
2. Test conflict scenarios in a staging cluster: Simulate two managers changing the same field to understand conflict messages and recovery.
3. Set up monitoring for apply conflicts: Use Prometheus to alert on increase in apiserver_request_total{verb="PATCH", code="409"} or scrape audit logs for conflict events.
4. Document ownership for critical resources: Maintain a table like:

1. Use dry-run in CI/CD: Before applying in production, run:

   kubectl apply --server-side -f manifest.yaml --field-manager=ci-cd --dry-run=server

If the dry run succeeds without conflict, proceed with actual apply.

1. Ensure secrets are not managed via SSA alone: Use external secret management (e.g., SealedSecrets, External Secrets) and reference secrets in pods.
2. Regularly review managedFields: Periodically audit managedFields to detect stale entries. Remove unused manager entries by applying an empty patch with the old manager? Actually, you can delete the manager entry by applying with the same manager an empty object? There is no direct command; you can use the API to remove the managedFields entry. For simplicity, use kubectl apply --server-side --field-manager=<old-manager> -f <manifest with no fields owned> and then delete the manager entry? Not straightforward. Alternative is to use a tool like kubectl patch with a raw application/merge-patch+json to remove the managedFields entry, but that is advanced. It is often safe to leave old entries as they do not affect operation.

Conclusion

Kubernetes Server-Side Apply provides a robust mechanism for managing configuration by tracking field-level ownership on the API server. By understanding its architecture, you can avoid common pitfalls such as unintended conflicts and ownership loss. Operational best practices include using unique field managers, testing in non-production, monitoring conflicts, and having clear recovery procedures.

Start with a low-risk resource like a ConfigMap in a test namespace. Practice applying with server-side, inspect managedFields, and simulate conflicts. Then gradually roll out server-side apply to more critical workloads, ensuring your team is familiar with conflict resolution. With careful management, server-side apply can make your Kubernetes configuration workflows safer and more predictable.