## Intro Kubernetes Server-Side Apply (SSA) changes how configuration is merged and stored, and that changes how you plan capacity. Instead of a client-side patch that can silently drop fields, SSA sends a fully specified intent to the API server, which then merges it with existing object state using field management. This gives you better conflict detection and predictable ownership, but it also means more work for the API server, more data stored in etcd, and more care required when you roll out changes at scale. This article is a practical guide to capacity planning for Server-Side Apply. It is written for developers, DevOps consultants, and startup platform teams who are already running Kubernetes and want to adopt or operate SSA safely. You will learn how to assess your current environment, size resources for SSA workloads, set limits, and verify that your cluster remains healthy under load. We will use concrete commands and example outputs throughout. Where possible, we reference a small reference cluster (three control-plane nodes, etcd colocated, 50 worker nodes, ~2,000 namespaces). All numbers are illustrative and should be adapted to your environment. ## 1. Version and Environment Inventory Before making any changes, know exactly what you are running. Server-Side Apply has been beta since Kubernetes 1.16 and stable since 1.22, but its behavior has changed subtly between releases. The `kubectl apply --server-side` flag uses SSA, and the field management implementation depends on the API server version. ### Check Kubernetes Version Run this on a control-plane node or via kubectl: ```bash kubectl version --short ``` Expected output (example): ``` Client Version: v1.28.3 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.28.3 ``` If your server is older than 1.22, SSA is still available as beta, but consider upgrading. For production capacity planning, assume SSA is stable only on 1.22+. ### Identify SSA Usage SSA is used implicitly by many tools (kubectl apply --server-side, kustomize with server-side apply, GitOps tools like Argo CD and Flux, and controllers). To see if your cluster has active SSA usage, check the `managedFields` on any object. Every object created via SSA has managed fields entries with a manager name. Example: ```bash kubectl get deployment my-app -o yaml | grep -A5 managedFields ``` If you see entries like `manager: kubectl-client-side-apply` or `manager: argocd-application-controller`, those indicate SSA usage. The manager name tells you which tool is applying the configuration. ### Prerequisites for Safe SSA Operation - **Kubernetes 1.22+**: Stable SSA. - **kubectl 1.22+**: Needed for `--server-side` flag. - **API Priority and Fairness (APF)**: SSA requests are normal API requests, but if you expect many concurrent applies, check APF configuration. - **etcd capacity**: SSA writes more data to etcd because of managed fields. Monitor etcd database size. Check APF status: ```bash kubectl get flowschemas.flowcontrol.apiserver.k8s.io ``` If no resources are returned, your cluster may be using default settings; ensure you have at least the default flowschemas. ### Read-Only Observation First Do not change anything yet. Capture baseline metrics: - API server request rate and latency - etcd database size - Number of managed fields entries per object type Use the following commands to get a feel for the current state: ```bash kubectl get --raw /metrics | grep apiserver_request_total kubectl get --raw /metrics | grep etcd_db_total_size_in_bytes ``` Record these before you roll out SSA broadly. ### Blast Radius and Recovery Path SSA changes are not inherently more dangerous than client-side apply, but they can create conflicts if multiple managers try to own the same field. Start with a single test namespace and a single Deployment before enabling SSA for your whole GitOps pipeline. ## 2. Safe Configuration Path Adopting SSA should be incremental. The following path minimizes risk and gives you early feedback. ### Step 1: Enable SSA for a Single Object Take an existing Deployment manifest and apply it with the server-side flag: ```bash kubectl apply --server-side -f deployment.yaml ``` Expected output: ``` deployment.apps/my-app serverside-applied ``` If you see `deployment.apps/my-app configured` or `unchanged`, that's also fine. But note: with `--server-side`, kubectl does not print a warning if the object is unchanged. ### Step 2: Verify Managed Fields Check that the manager is recorded correctly: ```bash kubectl get deployment my-app -o yaml | grep -B2 -A10 managedFields ``` Look for an entry with `manager: kubectl-client-side-apply` (or whatever you set as manager). Confirm that the operation is `Apply`. ### Step 3: Avoid Conflicts SSA conflicts occur when another manager owns a field you are trying to change and you do not force the change. Test conflict detection: Apply a change to a field owned by another manager without `--force-conflicts`: ```bash kubectl apply --server-side -f modified-deployment.yaml ``` Expected output (conflict): ``` error: Apply failed with 1 conflict: conflict with "kubectl-client-side-apply" using apps/v1: .spec.replicas ``` This is a good thing: it tells you exactly which field is contested. Resolve by coordinating with the other manager or, if you are certain, add `--force-conflicts`. ### Step 4: Use a Dedicated Manager Name In production, do not rely on the default manager name derived from the tool. Set a meaningful manager name so you can track ownership: ```bash kubectl apply --server-side --field-manager=platform-team -f deployment.yaml ``` Then verify: ```bash kubectl get deployment my-app -o yaml | grep 'manager: platform-team' ``` ### Step 5: Integrate with GitOps If you use Argo CD, enable server-side apply in the Application spec: ```yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: my-app spec: syncPolicy: syncOptions: - ServerSideApply=true ``` For Flux, set `--server-side-apply` in the kustomize-controller or helm-controller flags. ### Capacity Considerations During Rollout SSA generates larger objects because of the managed fields metadata. A Deployment that was 2 KB with client-side apply might become 3-4 KB with SSA. Multiply that by the number of objects you manage, and you can estimate additional etcd storage. For our reference cluster (2,000 namespaces, average 50 objects per namespace = 100,000 objects), a 2 KB increase per object means 200 MB extra etcd data. This is usually manageable, but you should monitor etcd size. Check etcd size: ```bash kubectl get --raw /metrics | grep etcd_db_total_size_in_bytes ``` Compare before and after enabling SSA for a subset of objects. ## 3. Verification and Diagnostics After applying SSA, verify that your objects are healthy and that the API server is not overwhelmed. ### Object-Level Verification For each critical object, check: - Status conditions - Events - Replica counts Example for a Deployment: ```bash kubectl rollout status deployment/my-app kubectl describe deployment my-app kubectl get events --field-selector involvedObject.name=my-app ``` ### API Server Performance SSA puts more load on the API server because it must compute field ownership and merge. Watch these metrics: - `apiserver_request_duration_seconds` (p99) - `apiserver_request_total` - `apiserver_current_inflight_requests` Get a quick snapshot: ```bash kubectl get --raw /metrics | grep apiserver_current_inflight_requests ``` Expected output example: ``` # HELP apiserver_current_inflight_requests Maximal number of currently used inflight request limit of this apiserver per request kind in last second. # TYPE apiserver_current_inflight_requests gauge apiserver_current_inflight_requests{requestKind="mutating"} 12 apiserver_current_inflight_requests{requestKind="readOnly"} 5 ``` If mutating inflight requests are consistently high (e.g., >80% of your APF limit), you may need to adjust APF or scale the API server. ### Diagnose SSA Conflicts Conflicts can cause sync failures in GitOps. When a sync fails, inspect the Application status and events: ```bash kubectl get application my-app -o yaml | grep -A20 status.conditions kubectl get events --field-selector involvedObject.name=my-app ``` Look for messages like `Apply failed with N conflicts`. Identify the conflicting fields and managers, then decide whether to force or to adjust ownership. ### Capacity Stress Testing To understand how SSA affects your cluster, run a controlled load test. Create a script that applies a hundred ConfigMaps with SSA in a loop and measure API server latency and etcd growth. Simplified loop (run in a test namespace): ```bash for i in {1..100}; do kubectl apply --server-side -f - < -o yaml | grep -A10 managedFields` - If the owner should not be managing the field, coordinate with them or use `--force-conflicts` to take ownership. - If it is a system field (e.g., status), do not force; instead, structure your manifest to avoid setting that field. ### Failure Mode 2: Object Not Applied Due to Dry-Run Error **Symptom**: SSA dry-run (default for kubectl apply) fails with validation error. **Cause**: The manifest has an invalid field or type. **Recovery**: - Run `kubectl apply --server-side --dry-run=server -f manifest.yaml -o yaml` to see the server's response. - Fix the manifest based on the error message. ### Failure Mode 3: API Server Overload **Symptom**: Increased API server latency, timeouts, or 503 responses during mass applies. **Cause**: Too many concurrent SSA requests, especially with large objects or many managed fields, exhausting APF queues or CPU. **Recovery**: - Reduce concurrency in your GitOps tool (e.g., Argo CD parallelism settings). - Increase API server resources or add more replicas. - Review APF priority levels and adjust if necessary. - Consider batching applies or using server-side apply only for objects that benefit from it. ### Failure Mode 4: etcd Database Growth Too Fast **Symptom**: etcd database size increases rapidly, possibly hitting the default 8 GB quota. **Cause**: Managed fields metadata added to every object, especially if many objects have multiple managers. **Recovery**: - Reduce the number of managers per object by standardizing on a single GitOps tool. - Use `kubectl apply-set` (if available) to prune old managers? - Manually remove obsolete managed fields entries using `kubectl patch` with a strategic merge patch on `metadata.managedFields`. - If quota exceeded, perform etcd compaction and defragmentation, and increase quota if necessary. ### Recovery Verification After any recovery action, verify the system is stable: ```bash kubectl get --raw /metrics | grep etcd_db_total_size_in_bytes kubectl get --raw /metrics | grep apiserver_request_duration_seconds_bucket | tail -5 kubectl get pods -n kube-system ``` Ensure API server pods are healthy, etcd size is below 80% of quota, and no pending conflicts remain. ## 5. Operations Checklist Use this checklist to run SSA in production with confidence. Each item has an owner and a review interval. | # | Checklist Item | Verification Command | Owner | Review Cadence | |---|---|---|---|---| | 1 | Kubernetes version >= 1.22 | `kubectl version --short` | Platform Team Lead (Priya Shah) | Monthly (or after upgrades) | | 2 | APF flowschemas present | `kubectl get flowschemas` | Platform Engineer (Alex Chen) | Weekly | | 3 | etcd size below 70% quota | `kubectl get --raw /metrics \| grep etcd_db_total_size_in_bytes` | Platform Engineer (Alex Chen) | Weekly | | 4 | No unresolved SSA conflicts on critical workloads | `kubectl get events --all-namespaces \| grep "Apply failed"` | DevOps Lead (Sam Rivera) | Daily (automated alert) | | 5 | API server p99 latency < 200 ms | Query metrics from Prometheus | Platform Team Lead (Priya Shah) | Weekly review of dashboards | | 6 | GitOps sync health for all apps | Argo CD UI or `kubectl get applications` | DevOps Engineer (Jamie Lee) | Continuous monitoring, weekly report | | 7 | Rollback plan tested | Run a simulated conflict and recovery in staging | Platform Engineer (Alex Chen) | Quarterly | ### Pre-Change Readiness Before any new SSA rollout, run this quick terminal session: ```bash # 1. Check cluster health kubectl get nodes kubectl get pods -n kube-system # 2. Check APF status kubectl get flowschemas # 3. Check etcd size kubectl get --raw /metrics | grep etcd_db_total_size_in_bytes # 4. Apply a test object with SSA in a dedicated namespace kubectl create ns ssa-test kubectl apply --server-side -f test-deployment.yaml -n ssa-test # 5. Verify managed fields kubectl get deployment test-app -n ssa-test -o yaml | grep -A5 managedFields ``` If all checks pass, you are ready for a gradual rollout. ## 6. Common Pitfalls and How to Avoid Them ### Pitfall 1: Ignoring Managed Fields Bloat **Why it happens**: Teams apply with multiple managers (kubectl, Argo CD, CI scripts) and never prune old managers. **Impact**: Objects grow large, etcd fills up, API server serialization cost increases. **Avoidance**: Standardize on one manager per object class. Use `--field-manager` explicitly. Periodically review `managedFields` on high-churn objects. ### Pitfall 2: Overusing `--force-conflicts` **Why it happens**: Developers hit a conflict and immediately force, thinking it is a shortcut. **Impact**: Silently steals ownership from controllers or other teams, causing later unexpected reverts. **Avoidance**: Treat conflicts as signals. Investigate the other manager. Only force after confirmation that you should own the field. ### Pitfall 3: Not Monitoring API Priority and Fairness **Why it happens**: APF is invisible until it starts queuing or rejecting requests. **Impact**: During a large apply wave, low-priority requests (including SSA applies) may be throttled, causing timeouts. **Avoidance**: Set up alerts on `apiserver_flowcontrol_rejected_requests_total` and `apiserver_flowcontrol_current_inqueue_requests`. Tune flowschemas if needed. ### Pitfall 4: Applying SSA to Huge Custom Resources **Why it happens**: Teams enable SSA globally without considering object size. **Impact**: Large CRDs (e.g., 1 MB+) with SSA cause significant CPU and memory spikes on the API server and etcd. **Avoidance**: Test SSA on large objects in staging first. Watch API server memory and request duration. Exclude very large CRs from SSA if necessary. ### Pitfall 5: Lack of Rollback Strategy **Why it happens**: SSA is seen as just another apply method, and rollback is not planned. **Impact**: If a bad configuration is applied with SSA and conflicts arise, reverting is more complex than just reapplying an old file. **Avoidance**: Keep versioned manifests in Git. Use GitOps to enable easy rollback. Document how to remove unwanted manager entries if needed. ## 7. Capacity Planning Numbers You Can Use Here are some starting numbers based on our reference cluster. Adjust based on your measurements. - **Object size increase**: Expect 1-3 KB extra per object due to managed fields. For 100,000 objects, that is 100-300 MB in etcd. - **API server CPU**: SSA merge logic adds about 5-10% CPU overhead per apply request compared to client-side apply. Plan for an additional small margin. - **API request latency**: SSA apply requests can be 10-20% slower than client-side applies on a loaded server. Monitor p99. - **Memory on API server**: Managed fields caching may increase memory usage slightly; allocate an extra 5% headroom. - **Concurrency limits**: With default APF, a single user can queue up to 100 mutating requests. For bulk applies, tune the `exempt` or `global-default` flowschema. Example APF flowschema adjustment (illustrative, not a full manifest): ```yaml apiVersion: flowcontrol.apiserver.k8s.io/v1beta3 kind: FlowSchema metadata: name: ssa-apply-burst spec: priorityLevelConfiguration: name: global-default matchingPrecedence: 200 distinguisherMethod: type: ByUser rules: - subjects: - kind: ServiceAccount serviceAccount: namespace: argocd name: argocd-application-controller resourceRules: - verbs: ["apply"] apiGroups: ["*"] resources: ["*"] ``` This gives the Argo CD controller a dedicated flow with more room to burst. Always test in staging. ## 8. Conclusion Kubernetes Server-Side Apply is a powerful tool for configuration management, but it requires deliberate capacity planning. By following the steps in this guide—inventorying your environment, adopting SSA incrementally, verifying performance, and preparing for failure modes—you can avoid common pitfalls and keep your cluster stable. Start with one low-risk application. Apply it with `--server-side`, inspect the managed fields, and measure the impact on API server and etcd. Then expand gradually while monitoring key metrics. Assign clear owners for SSA health checks and review them regularly. Remember: SSA's field management gives you clarity and safety, but only if you treat it as a first-class operational concern. Measure, verify, and iterate.