Kubernetes Custom Resource Definition Versioning Capacity Planning: A Practical Guide

Intro

CustomResourceDefinitions (CRDs) extend the Kubernetes API with your own resource types. As your application evolves, you will introduce new versions of these resources. Each version adds storage, conversion, and API server overhead. Without careful capacity planning, versioned CRDs can silently degrade cluster performance, exhaust etcd storage, or cause API request failures.

This guide explains how to plan capacity for CRD versioning. You will learn how to inventory your current CRD usage, estimate the cost of each version, set safe resource limits, monitor scaling signals, and implement recovery procedures. We use concrete examples based on Kubernetes 1.27+ and assume familiarity with basic CRD concepts.

By the end, you will be able to answer:

• How much additional etcd storage will a new CRD version consume?
• What is the acceptable request latency for conversion webhooks?
• How do I detect that the API server is under pressure from CRD conversions?
• What rollback steps are safe when a conversion fails?

Version and Environment Inventory

Before planning capacity, you must know your current state. Gather the following information for every CRD that will have multiple versions.

1. List CRDs and Their Versions

Run:

kubectl get crd -o custom-columns=NAME:.metadata.name,STORAGE:.spec.versions[*].name,SERVED:.spec.versions[*].served,STORAGE_VERSION:.spec.versions[?(@.storage==true)].name

Example output:

NAME                 STORAGE   SERVED      STORAGE_VERSION
widgets.example.com  v1, v2    true, true  [v1]

This shows that widgets.example.com has two versions: v1 and v2. Both are served, but v1 is the current storage version. Any new v2 object is converted to v1 for storage unless you change the storage version.

2. Count Objects per Version

You need to know how many objects exist for each version to estimate storage growth. Use a label selector if your CRD versions are represented by different labels, but typically you must query each version separately.

# Get count of v1 objects
kubectl get widgets.v1.example.com --all-namespaces -o json | jq '.items | length'

# Get count of v2 objects
kubectl get widgets.v2.example.com --all-namespaces -o json | jq '.items | length'

Suppose you have 10,000 v1 objects and 2,000 v2 objects. The v2 objects are already stored as v1 after conversion, so they do not add extra storage beyond their v1 representation. But keep the numbers for planning future storage version changes.

3. Estimate Object Size

Determine the average serialized size of a CR object. Use kubectl get with -o json and measure the size of a sample.

kubectl get widget.v1.example.com my-widget -n default -o json | wc -c

If the average object is 2 KB, then 10,000 objects consume about 20 MB in etcd (before replication and compression). When you introduce a new version, the additional storage depends on whether you store both versions or convert on write.

4. Identify Conversion Strategy

Check whether the CRD uses a conversion webhook or None.

kubectl get crd widgets.example.com -o jsonpath='{.spec.conversion}'

Possible outputs:

• {"strategy":"None"}: No conversion between versions. Each version is stored separately.
• {"strategy":"Webhook","webhook":{...}}: Conversion is handled by an external webhook. This adds API server request overhead and webhook latency.

If strategy is None, the API server only serves the exact version stored; no conversion happens. This is simpler but limits your ability to read older versions after a storage version change.

5. Cluster and etcd Capacity

Know your etcd storage limits and current usage. If you run managed Kubernetes, check the provider's quotas. For self-managed, query etcd metrics.

# Example: etcd database size in bytes
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key endpoint status --write-out=table

Look for DB SIZE. If you are near the default 2 GB quota (or your configured quota), plan for compaction and possibly adding more storage.

Environment Prerequisites

• Kubernetes 1.16+ for CRD versioning features.
• etcd v3.4+ recommended for better storage efficiency.
• Access to metrics-server or Prometheus for API server and etcd metrics.
• A test cluster that mirrors production sizes for load testing conversion webhooks.

Safe Configuration Path

Now that you have an inventory, you can design a capacity plan. The goal is to introduce versioning with minimal risk and predictable resource usage.

1. Define Resource Limits for Conversion Webhooks

If you use a conversion webhook, it runs as a service in your cluster. It must handle conversion requests for every object version mismatch. Set appropriate resource requests and limits.

Example deployment for a conversion webhook:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: crd-conversion-webhook
  namespace: conversion
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: webhook
        image: example/conversion-webhook:1.2.3
        resources:
          requests:
            cpu: "500m"
            memory: "512Mi"
          limits:
            cpu: "1"
            memory: "1Gi"
        ports:
        - containerPort: 443

Estimate CPU and memory based on expected conversion rate and object size. For example, if each conversion takes 5 ms CPU and you expect 100 conversions per second, you need at least 500m CPU for one replica. Add headroom (e.g., 2x) for spikes.

2. Configure Webhook Timeouts and Failure Policy

Kubernetes API server waits for a conversion webhook response. The default timeout is 10 seconds, configurable in the CRD's webhook client config.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: widgets.example.com
spec:
  conversion:
    strategy: Webhook
    webhook:
      clientConfig:
        url: "https://conversion.example.com/v1/convert"
        caBundle: <base64-encoded-ca>
        timeoutSeconds: 2
      conversionReviewVersions: ["v1"]
  versions:
  - name: v1
    served: true
    storage: true
  - name: v2
    served: true
    storage: false

Set timeoutSeconds under clientConfig if needed. A shorter timeout (e.g., 2 seconds) prevents API server thread exhaustion but may cause more conversion failures under load. Balance based on your webhook's P99 latency.

Set failurePolicy to Fail (default) for consistency, but be aware that if the webhook is unavailable, all requests for the new version will fail. For initial rollout, consider Ignore while you test, but switch to Fail for production to avoid silent data inconsistencies.

3. Plan Storage Version Migration

When you introduce a new version and set it as storage, existing objects are not automatically converted. Kubernetes performs lazy migration: objects are converted to the new storage version when they are updated or accessed. To force migration, you can trigger updates or use a migration tool.

To estimate capacity impact, calculate the storage size difference between versions. If v2 objects are smaller (e.g., due to field pruning), migrating reduces storage. If larger, you need to ensure etcd can handle the growth.

Use a dry-run in a test cluster to see conversion time per object. For example:

# Simulate converting a v1 object to v2
kubectl get widget.v1.example.com my-widget -n default -o json | jq '.apiVersion = "example.com/v2"' | kubectl convert -f -

4. Set Rate Limits and Quotas

Kubernetes API server has built-in rate limiting via --max-requests-inflight and --max-mutating-requests-inflight. Defaults are 400 and 200 respectively. If many conversions occur, mutating requests can spike. Monitor and adjust these flags if needed, but understand the impact on other workloads.

Also consider ResourceQuota in namespaces to limit the number of CR objects, preventing runaway growth.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: crd-quota
  namespace: default
spec:
  hard:
    count/widgets.example.com: "10000"

5. Gradual Rollout

Do not serve a new version to all users at once. Use a staged approach:

• Add the new version with served: false and storage: false to the CRD. This defines the schema but does not serve it.
• Deploy the conversion webhook and ensure it can handle conversions between versions.
• Set served: true for the new version while keeping old storage. Now clients can create objects with the new version, but they are stored as old version after conversion.
• After monitoring, change storage: true to the new version and set old version's storage: false. Trigger migration gradually.
• Optionally set served: false on the old version after all clients have migrated.

This approach lets you validate conversion performance at each step.

Verification and Diagnostics

Capacity planning is not a one-time activity. You must continuously verify that your system is operating within limits.

1. Monitor API Server Metrics

Expose API server metrics (default at :6443/metrics). Key metrics to watch:

• apiserver_request_duration_seconds_bucket with resource=customresourcedefinitions or your resource group. High latency indicates conversion overhead or etcd pressure.
• apiserver_current_inflight_requests - should stay below limit.
• apiserver_request_total with verb=POST/PUT and your resource.
• etcd_db_total_size_in_bytes and etcd_object_counts for storage growth.

Example Prometheus query for conversion webhook latency:

histogram_quantile(0.99, sum(rate(apiserver_request_duration_seconds_bucket{resource="widgets", verb="POST"}[5m])) by (le))

If P99 exceeds 1 second, investigate webhook performance.

2. Test Conversion Webhook Under Load

Use a load generator to simulate conversions. For instance, create many objects with the new version and measure the API server throughput and webhook resource usage. A simple shell loop:

for i in $(seq 1 1000); do
  kubectl apply -f - <<EOF
apiVersion: example.com/v2
kind: Widget
metadata:
  name: widget-$i
  namespace: loadtest
spec:
  size: 10
EOF
done

Watch the webhook pod CPU/memory with kubectl top pods -n conversion. If CPU is near limits, increase replicas or resources.

3. Validate Storage Version Migration Progress

Confirm the current storage version after you change it:

kubectl get crd widgets.example.com -o jsonpath='{.spec.versions[?(@.storage==true)].name}'

If you changed storage from v1 to v2, this command should output v2. The status.storedVersions field should eventually show only ["v2"] after all objects are migrated. You can force migration by updating objects or using kubectl get --raw to trigger conversion.

4. Check etcd Size and Fragmentation

After migration, monitor etcd database size. If it grows unexpectedly, check for duplicate storage of both versions (possible if conversion is None and you have both versions stored). Use compaction to reclaim space:

ETCDCTL_API=3 etcdctl --endpoints=... compact <latest-revision>
ETCDCTL_API=3 etcdctl --endpoints=... defrag

5. Logs and Alerts

Set up alerts for:

• API server mutating requests exceeding 80% of limit.
• etcd database size above 80% quota.
• Conversion webhook error rate above 1%.
• Stored versions containing unexpected version names.

Common Pitfalls and How to Avoid Them

Even with careful planning, teams often hit the same issues. Here are the most frequent pitfalls and how to avoid or recover from them.

1. Underestimating Conversion Overhead

Pitfall: Assuming conversion is free or trivial, so you don't allocate enough CPU to the webhook. Why it happens: Normal CRUD operations don't show the cost of conversion; it only appears when the webhook is invoked. How to avoid: Load test conversions with representative object sizes and rates. Set resource requests based on measured P99 latency, not guesses. Recovery: Scale up the webhook replicas and increase CPU limits, then re-test.

2. Ignoring etcd Growth from New Fields

Pitfall: Adding a new version with additional fields increases object size, but you assume storage won't grow much. Why it happens: You compare object counts but not average object size. How to avoid: Measure the serialized size of sample objects in both versions and project total etcd usage before rolling out. Recovery: Set a ResourceQuota to cap object count or compact etcd more frequently.

3. Leaving failurePolicy: Ignore in Production

Pitfall: You set failurePolicy: Ignore during testing and forget to change it to Fail. Why it happens: It's easy to overlook a CRD specification after testing. How to avoid: Use GitOps or a checklist to ensure all CRDs have failurePolicy: Fail before production. Set up an audit policy to detect Ignore. Recovery: Change the policy to Fail and validate that the webhook is healthy.

4. Not Monitoring storedVersions After Migration

Pitfall: You change the storage version but never check that all objects migrated. Why it happens: Lazy migration means objects may remain in old storage for months. How to avoid: After changing storage, set up a job to periodically check storedVersions and alert if more than one version is present longer than expected. Recovery: Force migration by updating all objects or using a tool like kubectl get --raw to trigger conversion, then monitor until storedVersions shows only the new version.

5. Not Testing Rollback Procedures

Pitfall: You only test the forward path, not the rollback. Why it happens: Rollback is considered an edge case. How to avoid: Practice rollback in staging: simulate a failed storage migration and revert to the old version, measuring time and impact. Recovery: If you find yourself in a failed state, follow the rollback steps in the failure modes section below, then improve your rollback runbook.

Failure Modes and Recovery

Even with careful planning, failures occur. Know how to recover.

1. Conversion Webhook Unavailable

If the webhook service is down and failurePolicy: Fail, all requests for the new version will be rejected with errors like:

Error from server (InternalError): error when creating "STDIN": conversion webhook for example.com/v2, Kind=Widget failed: Post "https://conversion.example.com/v1/convert": dial tcp 10.0.0.5:443: connect: connection refused

Recovery:

• Restore the webhook deployment.
• If recovery is slow, temporarily set failurePolicy: Ignore on the CRD to allow requests without conversion. This may cause data loss if fields are not compatible, so use with caution.
• Alternatively, remove the new version from served list to stop serving it.

2. High Conversion Latency Causing API Server Overload

If conversions are slow, API server threads may block, increasing inflight requests and causing new requests to be throttled or timeout.

Recovery:

• Scale up the conversion webhook replicas.
• Optimize the webhook code (e.g., caching, parallel conversion).
• Increase API server --max-mutating-requests-inflight temporarily, but watch memory.
• Reduce load by disabling serving of the new version until latency improves.

3. Storage Migration Incomplete or Stuck

Sometimes a CRD remains with multiple stored versions because some objects are never updated. This can cause read amplification and increased etcd size.

Recovery:

• Identify objects still in old storage version using kubectl get --raw and checking the stored version.
• Force update those objects (e.g., by adding an annotation) to trigger conversion.
• If many objects, use a script to list and update them in batches.

4. etcd Quota Exceeded

If etcd runs out of space, the API server may become read-only, causing widespread failures.

Recovery:

• Immediately compact and defrag etcd.
• If quota is hard, increase etcd quota or add more storage.
• Reduce object count by deleting obsolete CRs or namespaces.
• Review your capacity plan for future growth.

5. Rollback of Versioning Changes

If a new version causes problems, you can roll back partially or fully.

• Set served: false for the new version to stop serving it. Existing stored objects remain, but no new objects can be created with that version.
• If you changed storage version and migration is incomplete, you can revert storage to the original version. However, objects already migrated to the new storage version will be converted back when accessed. This may cause temporary load.
• Always have a backup of etcd before major changes. Restore from backup if needed.

Operations Checklist and Decision Ownership

Use this checklist for ongoing operations and capacity review. Each item has a clear owner and review cadence.

Decision ownership: The final decision to change storage version or serve a new version rests with the Platform Lead. The decision should be reviewed at least every six months or when a new version is proposed.

Review Steps

• Inventory: Update your CRD version list and object counts.
• Measure: Collect etcd size and API server metrics.
• Analyze: Compare current usage to capacity limits. Look for trends.
• Plan: If approaching limits (e.g., etcd 70% full, API server inflight > 70% of max), plan scaling actions such as:
• Increase etcd storage or compact more frequently.
• Scale conversion webhook.
• Increase API server resources or adjust rate limits.
• Prune old CR objects.
• Test: Before applying changes, test in a staging cluster with realistic load.
• Document: Update runbooks with new thresholds and procedures.

Conclusion

Capacity planning for Kubernetes CRD versioning is essential to maintain cluster performance and stability. By inventorying your current state, estimating resource needs, configuring safe limits, and monitoring key metrics, you can introduce new versions without surprises.

Remember the key points:

• Versioned CRDs add storage and CPU overhead, especially with conversion webhooks.
• Use a staged rollout to gradually expose new versions and monitor impact.
• Set resource limits and alerts based on measured load.
• Have recovery procedures for common failures like webhook unavailability and etcd pressure.
• Regularly review capacity metrics and adjust as your application grows.

Start with a narrow pilot: pick one CRD with moderate usage, implement versioning following this guide, and observe the metrics before rolling out to all CRDs. This practical approach reduces risk and builds confidence.

For further reading, consult the Kubernetes documentation on CustomResourceDefinitions and API server metrics. Experiment with the provided commands in a test environment to adapt them to your setup.