Kubernetes Server-Side Apply Capacity Planning: A Practical Operator's Guide

Intro

Kubernetes Server-Side Apply (SSA) changes how configuration is merged and stored, and that changes how you plan capacity. Instead of a client-side patch that can silently drop fields, SSA sends a fully specified intent to the API server, which then merges it with existing object state using field management. This gives you better conflict detection and predictable ownership, but it also means more work for the API server, more data stored in etcd, and more care required when you roll out changes at scale.

This article is a practical guide to capacity planning for Server-Side Apply. It is written for developers, DevOps consultants, and startup platform teams who are already running Kubernetes and want to adopt or operate SSA safely. You will learn how to assess your current environment, size resources for SSA workloads, set limits, and verify that your cluster remains healthy under load.

We will use concrete commands and example outputs throughout. Where possible, we reference a small reference cluster (three control-plane nodes, etcd colocated, 50 worker nodes, ~2,000 namespaces). All numbers are illustrative and should be adapted to your environment.

1. Version and Environment Inventory

Before making any changes, know exactly what you are running. Server-Side Apply has been beta since Kubernetes 1.16 and stable since 1.22, but its behavior has changed subtly between releases. The kubectl apply --server-side flag uses SSA, and the field management implementation depends on the API server version.

Check Kubernetes Version

Run this on a control-plane node or via kubectl:

kubectl version --short

Expected output (example):

Client Version: v1.28.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.3

If your server is older than 1.22, SSA is still available as beta, but consider upgrading. For production capacity planning, assume SSA is stable only on 1.22+.

Identify SSA Usage

SSA is used implicitly by many tools (kubectl apply --server-side, kustomize with server-side apply, GitOps tools like Argo CD and Flux, and controllers). To see if your cluster has active SSA usage, check the managedFields on any object. Every object created via SSA has managed fields entries with a manager name.

Example:

kubectl get deployment my-app -o yaml | grep -A5 managedFields

If you see entries like manager: kubectl-client-side-apply or manager: argocd-application-controller, those indicate SSA usage. The manager name tells you which tool is applying the configuration.

Prerequisites for Safe SSA Operation

• Kubernetes 1.22+: Stable SSA.
• kubectl 1.22+: Needed for --server-side flag.
• API Priority and Fairness (APF): SSA requests are normal API requests, but if you expect many concurrent applies, check APF configuration.
• etcd capacity: SSA writes more data to etcd because of managed fields. Monitor etcd database size.

Check APF status:

kubectl get flowschemas.flowcontrol.apiserver.k8s.io

If no resources are returned, your cluster may be using default settings; ensure you have at least the default flowschemas.

Read-Only Observation First

Do not change anything yet. Capture baseline metrics:

• API server request rate and latency
• etcd database size
• Number of managed fields entries per object type

Use the following commands to get a feel for the current state:

kubectl get --raw /metrics | grep apiserver_request_total
kubectl get --raw /metrics | grep etcd_db_total_size_in_bytes

Record these before you roll out SSA broadly.

Blast Radius and Recovery Path

SSA changes are not inherently more dangerous than client-side apply, but they can create conflicts if multiple managers try to own the same field. Start with a single test namespace and a single Deployment before enabling SSA for your whole GitOps pipeline.

2. Safe Configuration Path

Adopting SSA should be incremental. The following path minimizes risk and gives you early feedback.

Step 1: Enable SSA for a Single Object

Take an existing Deployment manifest and apply it with the server-side flag:

kubectl apply --server-side -f deployment.yaml

Expected output:

deployment.apps/my-app serverside-applied

If you see deployment.apps/my-app configured or unchanged, that's also fine. But note: with --server-side, kubectl does not print a warning if the object is unchanged.

Step 2: Verify Managed Fields

Check that the manager is recorded correctly:

kubectl get deployment my-app -o yaml | grep -B2 -A10 managedFields

Look for an entry with manager: kubectl-client-side-apply (or whatever you set as manager). Confirm that the operation is Apply.

Step 3: Avoid Conflicts

SSA conflicts occur when another manager owns a field you are trying to change and you do not force the change. Test conflict detection:

Apply a change to a field owned by another manager without --force-conflicts:

kubectl apply --server-side -f modified-deployment.yaml

Expected output (conflict):

error: Apply failed with 1 conflict: conflict with "kubectl-client-side-apply" using apps/v1: .spec.replicas

This is a good thing: it tells you exactly which field is contested. Resolve by coordinating with the other manager or, if you are certain, add --force-conflicts.

Step 4: Use a Dedicated Manager Name

In production, do not rely on the default manager name derived from the tool. Set a meaningful manager name so you can track ownership:

kubectl apply --server-side --field-manager=platform-team -f deployment.yaml

Then verify:

kubectl get deployment my-app -o yaml | grep 'manager: platform-team'

Step 5: Integrate with GitOps

If you use Argo CD, enable server-side apply in the Application spec:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
spec:
  syncPolicy:
    syncOptions:
    - ServerSideApply=true

For Flux, set --server-side-apply in the kustomize-controller or helm-controller flags.

Capacity Considerations During Rollout

SSA generates larger objects because of the managed fields metadata. A Deployment that was 2 KB with client-side apply might become 3-4 KB with SSA. Multiply that by the number of objects you manage, and you can estimate additional etcd storage. For our reference cluster (2,000 namespaces, average 50 objects per namespace = 100,000 objects), a 2 KB increase per object means 200 MB extra etcd data. This is usually manageable, but you should monitor etcd size.

Check etcd size:

kubectl get --raw /metrics | grep etcd_db_total_size_in_bytes

Compare before and after enabling SSA for a subset of objects.

3. Verification and Diagnostics

After applying SSA, verify that your objects are healthy and that the API server is not overwhelmed.

Object-Level Verification

For each critical object, check:

• Status conditions
• Events
• Replica counts

Example for a Deployment:

kubectl rollout status deployment/my-app
kubectl describe deployment my-app
kubectl get events --field-selector involvedObject.name=my-app

API Server Performance

SSA puts more load on the API server because it must compute field ownership and merge. Watch these metrics:

• apiserver_request_duration_seconds (p99)
• apiserver_request_total
• apiserver_current_inflight_requests

Get a quick snapshot:

kubectl get --raw /metrics | grep apiserver_current_inflight_requests

Expected output example:

# HELP apiserver_current_inflight_requests Maximal number of currently used inflight request limit of this apiserver per request kind in last second.
# TYPE apiserver_current_inflight_requests gauge
apiserver_current_inflight_requests{requestKind="mutating"} 12
apiserver_current_inflight_requests{requestKind="readOnly"} 5

If mutating inflight requests are consistently high (e.g., >80% of your APF limit), you may need to adjust APF or scale the API server.

Diagnose SSA Conflicts

Conflicts can cause sync failures in GitOps. When a sync fails, inspect the Application status and events:

kubectl get application my-app -o yaml | grep -A20 status.conditions
kubectl get events --field-selector involvedObject.name=my-app

Look for messages like Apply failed with N conflicts. Identify the conflicting fields and managers, then decide whether to force or to adjust ownership.

Capacity Stress Testing

To understand how SSA affects your cluster, run a controlled load test. Create a script that applies a hundred ConfigMaps with SSA in a loop and measure API server latency and etcd growth.

Simplified loop (run in a test namespace):

for i in {1..100}; do
  kubectl apply --server-side -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: cm-$i
  namespace: test-ssa
data:
  key: value$i
EOF
done

While running, watch API server metrics and etcd size. This gives you empirical data for capacity planning.

Diagnostic Checklist

• [ ] Object applied successfully with --server-side?
• [ ] Managed fields show correct manager?
• [ ] No unexpected conflicts?
• [ ] API server p99 latency within SLI?
• [ ] etcd size increased roughly as predicted?
• [ ] GitOps sync completes without issues?

4. Failure Modes and Recovery

Even with careful planning, failures happen. Here are common SSA failure modes and how to recover.

Failure Mode 1: Conflict on Apply

Symptom: kubectl apply --server-side returns an error listing conflicting fields.

Cause: Another manager already owns those fields. This could be a controller, a previous client-side apply, or another team's GitOps tool.

Recovery:

• Identify the current owner: kubectl get <resource> -o yaml | grep -A10 managedFields
• If the owner should not be managing the field, coordinate with them or use --force-conflicts to take ownership.
• If it is a system field (e.g., status), do not force; instead, structure your manifest to avoid setting that field.

Failure Mode 2: Object Not Applied Due to Dry-Run Error

Symptom: SSA dry-run (default for kubectl apply) fails with validation error.

Cause: The manifest has an invalid field or type.

Recovery:

• Run kubectl apply --server-side --dry-run=server -f manifest.yaml -o yaml to see the server's response.
• Fix the manifest based on the error message.

Failure Mode 3: API Server Overload

Symptom: Increased API server latency, timeouts, or 503 responses during mass applies.

Cause: Too many concurrent SSA requests, especially with large objects or many managed fields, exhausting APF queues or CPU.

Recovery:

• Reduce concurrency in your GitOps tool (e.g., Argo CD parallelism settings).
• Increase API server resources or add more replicas.
• Review APF priority levels and adjust if necessary.
• Consider batching applies or using server-side apply only for objects that benefit from it.

Failure Mode 4: etcd Database Growth Too Fast

Symptom: etcd database size increases rapidly, possibly hitting the default 8 GB quota.

Cause: Managed fields metadata added to every object, especially if many objects have multiple managers.

Recovery:

• Reduce the number of managers per object by standardizing on a single GitOps tool.
• Use kubectl apply-set (if available) to prune old managers?
• Manually remove obsolete managed fields entries using kubectl patch with a strategic merge patch on metadata.managedFields.
• If quota exceeded, perform etcd compaction and defragmentation, and increase quota if necessary.

Recovery Verification

After any recovery action, verify the system is stable:

kubectl get --raw /metrics | grep etcd_db_total_size_in_bytes
kubectl get --raw /metrics | grep apiserver_request_duration_seconds_bucket | tail -5
kubectl get pods -n kube-system

Ensure API server pods are healthy, etcd size is below 80% of quota, and no pending conflicts remain.

5. Operations Checklist

Use this checklist to run SSA in production with confidence. Each item has an owner and a review interval.

Pre-Change Readiness

Before any new SSA rollout, run this quick terminal session:

# 1. Check cluster health
kubectl get nodes
kubectl get pods -n kube-system

# 2. Check APF status
kubectl get flowschemas

# 3. Check etcd size
kubectl get --raw /metrics | grep etcd_db_total_size_in_bytes

# 4. Apply a test object with SSA in a dedicated namespace
kubectl create ns ssa-test
kubectl apply --server-side -f test-deployment.yaml -n ssa-test

# 5. Verify managed fields
kubectl get deployment test-app -n ssa-test -o yaml | grep -A5 managedFields

If all checks pass, you are ready for a gradual rollout.

6. Common Pitfalls and How to Avoid Them

Pitfall 1: Ignoring Managed Fields Bloat

Why it happens: Teams apply with multiple managers (kubectl, Argo CD, CI scripts) and never prune old managers.

Impact: Objects grow large, etcd fills up, API server serialization cost increases.

Avoidance: Standardize on one manager per object class. Use --field-manager explicitly. Periodically review managedFields on high-churn objects.

Pitfall 2: Overusing --force-conflicts

Why it happens: Developers hit a conflict and immediately force, thinking it is a shortcut.

Impact: Silently steals ownership from controllers or other teams, causing later unexpected reverts.

Avoidance: Treat conflicts as signals. Investigate the other manager. Only force after confirmation that you should own the field.

Pitfall 3: Not Monitoring API Priority and Fairness

Why it happens: APF is invisible until it starts queuing or rejecting requests.

Impact: During a large apply wave, low-priority requests (including SSA applies) may be throttled, causing timeouts.

Avoidance: Set up alerts on apiserver_flowcontrol_rejected_requests_total and apiserver_flowcontrol_current_inqueue_requests. Tune flowschemas if needed.

Pitfall 4: Applying SSA to Huge Custom Resources

Why it happens: Teams enable SSA globally without considering object size.

Impact: Large CRDs (e.g., 1 MB+) with SSA cause significant CPU and memory spikes on the API server and etcd.

Avoidance: Test SSA on large objects in staging first. Watch API server memory and request duration. Exclude very large CRs from SSA if necessary.

Pitfall 5: Lack of Rollback Strategy

Why it happens: SSA is seen as just another apply method, and rollback is not planned.

Impact: If a bad configuration is applied with SSA and conflicts arise, reverting is more complex than just reapplying an old file.

Avoidance: Keep versioned manifests in Git. Use GitOps to enable easy rollback. Document how to remove unwanted manager entries if needed.

7. Capacity Planning Numbers You Can Use

Here are some starting numbers based on our reference cluster. Adjust based on your measurements.

• Object size increase: Expect 1-3 KB extra per object due to managed fields. For 100,000 objects, that is 100-300 MB in etcd.
• API server CPU: SSA merge logic adds about 5-10% CPU overhead per apply request compared to client-side apply. Plan for an additional small margin.
• API request latency: SSA apply requests can be 10-20% slower than client-side applies on a loaded server. Monitor p99.
• Memory on API server: Managed fields caching may increase memory usage slightly; allocate an extra 5% headroom.
• Concurrency limits: With default APF, a single user can queue up to 100 mutating requests. For bulk applies, tune the exempt or global-default flowschema.

Example APF flowschema adjustment (illustrative, not a full manifest):

apiVersion: flowcontrol.apiserver.k8s.io/v1beta3
kind: FlowSchema
metadata:
  name: ssa-apply-burst
spec:
  priorityLevelConfiguration:
    name: global-default
  matchingPrecedence: 200
  distinguisherMethod:
    type: ByUser
  rules:
  - subjects:
    - kind: ServiceAccount
      serviceAccount:
        namespace: argocd
        name: argocd-application-controller
    resourceRules:
    - verbs: ["apply"]
      apiGroups: ["*"]
      resources: ["*"]

This gives the Argo CD controller a dedicated flow with more room to burst. Always test in staging.

8. Conclusion

Kubernetes Server-Side Apply is a powerful tool for configuration management, but it requires deliberate capacity planning. By following the steps in this guide—inventorying your environment, adopting SSA incrementally, verifying performance, and preparing for failure modes—you can avoid common pitfalls and keep your cluster stable.

Start with one low-risk application. Apply it with --server-side, inspect the managed fields, and measure the impact on API server and etcd. Then expand gradually while monitoring key metrics. Assign clear owners for SSA health checks and review them regularly.

Remember: SSA's field management gives you clarity and safety, but only if you treat it as a first-class operational concern. Measure, verify, and iterate.