## Introduction Custom Resource Definitions (CRDs) extend Kubernetes with domain-specific APIs, enabling teams to manage applications and infrastructure through declarative configurations. However, CRDs are powerful and complex. A single misconfigured field can break existing resources, cause silent API failures, or make upgrades impossible. This guide examines frequent CRD configuration mistakes through practical examples, showing how to detect, prevent, and recover from them in production clusters. You will learn to validate schemas, avoid common pitfalls, and establish reliable operational workflows. ## Version and Environment Inventory Before changing any CRD, know exactly what you are running. Capture the Kubernetes version, CRD API version, and the controller version consuming the CRD. Some behaviors differ between Kubernetes releases, especially around structural schemas and pruning. Run these read-only commands to record the environment: ```bash kubectl version --short kubectl get crd -o yaml kubectl get deployment -n -o yaml ``` Expected output includes the API version (apiextensions.k8s.io/v1 or v1beta1), the CRD's stored versions, and the controller image tag. For example: ``` API Version: apiextensions.k8s.io/v1 Kind: CustomResourceDefinition Spec: Group: example.com Versions: - Name: v1 Served: true Storage: true ``` If the CRD uses `apiextensions.k8s.io/v1beta1`, plan a migration to v1 because v1beta1 is removed in Kubernetes 1.22 and later. Check that all fields required by the v1 schema are present, especially `schema.openAPIV3Schema` with structural schema requirements. **Practical environment test**: apply a minimal custom resource and verify it appears in `kubectl get ` without errors. For example: ```yaml apiVersion: example.com/v1 kind: MyResource metadata: name: test-instance spec: size: 1 ``` Then: ```bash kubectl apply -f test-instance.yaml kubectl get myresources ``` If the resource is not listed, check controller logs and CRD conditions. ## Safe Configuration Path A safe CRD rollout minimizes the risk of breaking existing custom resources and controllers. Follow these steps: 1. **Review current CRD**: `kubectl get crd -o yaml > crd-backup.yaml` 2. **Test schema changes locally**: Use `kubectl apply --dry-run=client` and `--dry-run=server` to validate without persisting. 3. **Stage the new CRD**: If adding a new version, ensure `served: false` initially, then gradually enable it. 4. **Update the controller first**: The controller should understand the new schema before you apply it to the cluster. 5. **Apply the CRD change**: `kubectl apply -f crd-updated.yaml` Example: Adding a new version `v2` to an existing CRD. Start with `served: true, storage: false`: ```yaml versions: - name: v1 served: true storage: true - name: v2 served: true storage: false ``` After migrating data, set `storage: true` on v2 and `storage: false` on v1 gradually. **Common mistake**: Changing the storage version without a migration path. This can cause data loss or API errors. Kubernetes does not automatically convert stored objects; you must provide a conversion webhook or handle it manually. **Verification**: After applying, check the CRD status conditions: ```bash kubectl get crd -o jsonpath='{.status.conditions}' | jq . ``` Look for `Established: True` and `NamesAccepted: True`. Also try creating a resource with both old and new fields to ensure compatibility. ## Verification and Diagnostics Effective diagnostics start with observing the system's response to your configuration. Use `kubectl describe` to surface events and status details. **Validate schema correctness** by applying an intentionally invalid resource: ```yaml apiVersion: example.com/v1 kind: MyResource metadata: name: bad-instance spec: size: "large" # expected integer ``` Expected output from `kubectl apply`: ``` The request is invalid: spec.size: Invalid value: "string": spec.size in body must be of type integer: "string" ``` If you do not see a validation error, the schema may be missing `type: integer` or the field is not marked as required. Review the OpenAPI v3 schema under `spec.versions[].schema.openAPIV3Schema`. **Diagnose controller issues**: Fetch logs for the controller: ```bash kubectl logs -n deployment/ --tail=50 ``` Look for repeated reconciliation errors, such as `unable to parse resource` or `field not recognized`. These indicate a mismatch between the CRD schema and the controller's expectations. **Check CRD status conditions**: ```bash kubectl get crd -o yaml | grep -A 10 conditions: ``` Conditions to monitor: - `NamesAccepted`: true if the names are valid. - `Established`: true when the REST endpoint is active. - `NonStructuralSchema`: present if the schema is not structural. **Example**: A CRD with a non-structural schema will trigger this warning: ``` message: 'spec.validation.openAPIV3Schema.type: Required value: must not be empty at the root' ``` Fix by specifying `type: object` at the root and adding `properties` for each field. ## Failure Modes and Recovery CRD misconfigurations can cause partial failures, silent data loss, or complete API unavailability. Prepare recovery procedures for the most likely scenarios. ### Scenario 1: Invalid Schema Breaks Existing Resources **Symptom**: After updating a CRD, existing resources fail validation on update or controller operations. **Cause**: The new schema has stricter requirements (e.g., new required fields) that old resources do not satisfy. **Recovery**: 1. Revert to the previous CRD version using your backup: ```bash kubectl apply -f crd-backup.yaml ``` 2. Fix the schema to be backward compatible, e.g., by not adding required fields without defaults. 3. Reapply the updated CRD. ### Scenario 2: Storage Version Changed Without Conversion **Symptom**: `kubectl get ` returns errors like `no kind is registered for the version "v2"` or conversion errors. **Cause**: You changed the `storage` version but did not provide a conversion webhook, and existing objects are stored in the old version. **Recovery**: 1. Set the storage version back to the previous one: ```bash kubectl patch crd --type='json' -p='[{"op": "replace", "path": "/spec/versions/0/storage", "value": true},{"op": "replace", "path": "/spec/versions/1/storage", "value": false}]' ``` (Adjust indices to match your versions.) 2. Implement a conversion webhook or manually migrate objects using `kubectl get --raw` and `kubectl replace`. 3. Then repeat the storage version change. ### Scenario 3: Deleted CRD Orphaned Custom Resources **Symptom**: You delete a CRD and all its custom resources are deleted immediately, causing data loss. **Cause**: Cascading deletion is default. You intended to keep the data for later migration. **Recovery**: Restore from etcd backup or any snapshots. To prevent this in the future, set `spec.preserveUnknownFields: false` is not enough; you need to export resources before deletion: ```bash kubectl get -o yaml > resources-backup.yaml ``` After recreating the CRD, apply the resources back. ### Scenario 4: Controller Not Updated for New Fields **Symptom**: New fields are ignored or cause controller crashes. **Cause**: The controller's Go structs do not include the new fields, or the controller uses unstructured code with strict decoding. **Recovery**: Update the controller to match the new schema, then restart it. To avoid this, use a conversion webhook to keep multiple API versions or coordinate CRD and controller releases. ## Common Mistakes and How to Avoid Them ### 1. Omitting Structural Schema **Why it happens**: Early CRDs often used `preserveUnknownFields: true` or minimal schemas for flexibility. **Risk**: Kubernetes cannot prune unknown fields, leading to API server memory bloat and incompatibility with admission webhooks and CRD validation tools. **Avoidance**: Always define a complete `openAPIV3Schema` with `type: object` and properties for all known fields. Set `preserveUnknownFields: false` (default in v1). **Example minimal structural schema**: ```yaml schema: openAPIV3Schema: type: object properties: spec: type: object properties: size: type: integer minimum: 1 ``` ### 2. Forgetting Default Values for New Fields **Why it happens**: Adding new fields without `default` can break existing objects that do not have that field set. **Risk**: Controllers may become nil pointer to a field, causing crashes. Or admission webhooks reject old objects. **Avoidance**: Define a `default` for every new field, or make the controller handle missing values gracefully. **Example**: ```yaml properties: replicas: type: integer default: 1 ``` ### 3. Changing Field Types Incompatibly **Why it happens**: Developers assume changing a field from string to integer is easy. **Risk**: It breaks API compatibility. Existing clients and stored objects may fail. **Avoidance**: Never change a field's type in the same API version. Introduce a new version (v2) with the changed type and use a conversion webhook. ### 4. Not Testing with `kubectl explain` **Why it happens**: Teams rush to apply CRDs without checking documentation generation. **Risk**: Users cannot discover fields properly; `kubectl explain` gives incomplete information. **Avoidance**: After applying the CRD, run `kubectl explain .` to ensure descriptions appear. Add `description` fields in the schema. **Example**: ```bash kubectl explain myresource.spec.size ``` Expected output includes description and type. ### 5. Ignoring Conversion Webhooks **Why it happens**: Teams serve only one version initially, then later add a new version without a webhook. **Risk**: When multiple versions are served, the API server cannot convert between them, causing errors for clients that use a specific version. **Avoidance**: Implement a conversion webhook as soon as you have more than one version. Use tools like controller-runtime to generate webhooks. **Example webhook configuration** in CRD: ```yaml conversion: strategy: Webhook webhook: conversionReviewVersions: ["v1", "v1beta1"] clientConfig: service: name: conversion-webhook namespace: default path: /convert ``` ### 6. Pruning Custom Fields Accidentally **Why it happens**: You add a field but do not include it in the schema, and the API server prunes it because unknown fields are dropped by default. **Risk**: Data silently disappears when a user submits a resource with that field. **Avoidance**: Ensure the schema includes every field your controller expects. Use `x-kubernetes-preserve-unknown-fields: true` only in specific nested contexts if necessary. **Example** of preserving nested unknown fields: ```yaml properties: spec: type: object x-kubernetes-preserve-unknown-fields: true ``` But use sparingly; better to model all fields. ## Operations Checklist Maintain a repeatable CRD operations checklist. Assign an owner for each major step and review the checklist quarterly or after incidents. | Step | Action | Owner | Frequency | Verification | |------|--------|-------|-----------|--------------| | 1 | Backup current CRD manifest to version control | Platform Lead (e.g., Priya Shah) | Before any change | `kubectl get crd -o yaml` diff with repo | | 2 | Review schema changes for backward compatibility | Backend Engineer (e.g., Tom Nguyen) | Every PR | `kubectl apply --dry-run=server` succeeds | | 3 | Test with sample resources | QA Engineer (e.g., Maria Lopez) | Each release | `kubectl apply -f samples/` and check status | | 4 | Check controller logs for errors | On-call SRE (e.g., Chris Evans) | After deployment | `kubectl logs` shows no CRD-related errors | | 5 | Validate `kubectl explain` output | Developer (e.g., Sam Chen) | Each schema change | `kubectl explain ` shows fields | | 6 | Review conversion webhook metrics | Platform Lead | Monthly | Webhook latency < 100ms, error rate < 0.1% | | 7 | Test rollback procedure | SRE (e.g., Dana White) | Quarterly | Apply previous CRD and confirm resources still work | Use this checklist as a starting point: adapt owners and frequencies to your team size and release cadence. ## Conclusion CRD configuration mistakes can have far-reaching consequences, from silent data loss to cluster-wide API failures. By understanding the common pitfalls, validating schemas rigorously, and establishing recovery playbooks, you can operate CRDs with confidence. Start with an inventory of your current CRDs, implement structural schemas, and test every change in a staging environment. The next step is to pick one CRD in your cluster, run the diagnostics described here, and fix any issues you find. Building these habits now will save you from painful debugging later. A reliable workflow for CRDs is not just about avoiding mistakes; it is about making your extended Kubernetes API predictable, observable, and safe for everyone who uses it.