Kubernetes Custom Resource Definition Configuration Mistakes: A Practical Field Guide

Introduction

Custom Resource Definitions (CRDs) extend Kubernetes with domain-specific APIs, enabling teams to manage applications and infrastructure through declarative configurations. However, CRDs are powerful and complex. A single misconfigured field can break existing resources, cause silent API failures, or make upgrades impossible. This guide examines frequent CRD configuration mistakes through practical examples, showing how to detect, prevent, and recover from them in production clusters. You will learn to validate schemas, avoid common pitfalls, and establish reliable operational workflows.

Version and Environment Inventory

Before changing any CRD, know exactly what you are running. Capture the Kubernetes version, CRD API version, and the controller version consuming the CRD. Some behaviors differ between Kubernetes releases, especially around structural schemas and pruning.

Run these read-only commands to record the environment:

kubectl version --short
kubectl get crd <crd-name> -o yaml
kubectl get deployment -n <namespace> <controller-name> -o yaml

Expected output includes the API version (apiextensions.k8s.io/v1 or v1beta1), the CRD's stored versions, and the controller image tag. For example:

API Version: apiextensions.k8s.io/v1
Kind: CustomResourceDefinition
Spec:
  Group: example.com
  Versions:
    - Name: v1
      Served: true
      Storage: true

If the CRD uses apiextensions.k8s.io/v1beta1, plan a migration to v1 because v1beta1 is removed in Kubernetes 1.22 and later. Check that all fields required by the v1 schema are present, especially schema.openAPIV3Schema with structural schema requirements.

Practical environment test: apply a minimal custom resource and verify it appears in kubectl get <plural> without errors. For example:

apiVersion: example.com/v1
kind: MyResource
metadata:
  name: test-instance
spec:
  size: 1

Then:

kubectl apply -f test-instance.yaml
kubectl get myresources

If the resource is not listed, check controller logs and CRD conditions.

Safe Configuration Path

A safe CRD rollout minimizes the risk of breaking existing custom resources and controllers. Follow these steps:

1. Review current CRD: kubectl get crd <crd-name> -o yaml > crd-backup.yaml
2. Test schema changes locally: Use kubectl apply --dry-run=client and --dry-run=server to validate without persisting.
3. Stage the new CRD: If adding a new version, ensure served: false initially, then gradually enable it.
4. Update the controller first: The controller should understand the new schema before you apply it to the cluster.
5. Apply the CRD change: kubectl apply -f crd-updated.yaml

Example: Adding a new version v2 to an existing CRD. Start with served: true, storage: false:

versions:
  - name: v1
    served: true
    storage: true
  - name: v2
    served: true
    storage: false

After migrating data, set storage: true on v2 and storage: false on v1 gradually.

Common mistake: Changing the storage version without a migration path. This can cause data loss or API errors. Kubernetes does not automatically convert stored objects; you must provide a conversion webhook or handle it manually.

Verification: After applying, check the CRD status conditions:

kubectl get crd <crd-name> -o jsonpath='{.status.conditions}' | jq .

Look for Established: True and NamesAccepted: True. Also try creating a resource with both old and new fields to ensure compatibility.

Verification and Diagnostics

Effective diagnostics start with observing the system's response to your configuration. Use kubectl describe to surface events and status details.

Validate schema correctness by applying an intentionally invalid resource:

apiVersion: example.com/v1
kind: MyResource
metadata:
  name: bad-instance
spec:
  size: "large"   # expected integer

Expected output from kubectl apply:

The request is invalid: spec.size: Invalid value: "string": spec.size in body must be of type integer: "string"

If you do not see a validation error, the schema may be missing type: integer or the field is not marked as required. Review the OpenAPI v3 schema under spec.versions[].schema.openAPIV3Schema.

Diagnose controller issues: Fetch logs for the controller:

kubectl logs -n <namespace> deployment/<controller-name> --tail=50

Look for repeated reconciliation errors, such as unable to parse resource or field not recognized. These indicate a mismatch between the CRD schema and the controller's expectations.

Check CRD status conditions:

kubectl get crd <crd-name> -o yaml | grep -A 10 conditions:

Conditions to monitor:

• NamesAccepted: true if the names are valid.
• Established: true when the REST endpoint is active.
• NonStructuralSchema: present if the schema is not structural.

Example: A CRD with a non-structural schema will trigger this warning:

message: 'spec.validation.openAPIV3Schema.type: Required value: must not be empty at the root'

Fix by specifying type: object at the root and adding properties for each field.

Failure Modes and Recovery

CRD misconfigurations can cause partial failures, silent data loss, or complete API unavailability. Prepare recovery procedures for the most likely scenarios.

Scenario 1: Invalid Schema Breaks Existing Resources

Symptom: After updating a CRD, existing resources fail validation on update or controller operations.

Cause: The new schema has stricter requirements (e.g., new required fields) that old resources do not satisfy.

Recovery:

1. Revert to the previous CRD version using your backup:

   kubectl apply -f crd-backup.yaml

1. Fix the schema to be backward compatible, e.g., by not adding required fields without defaults.
2. Reapply the updated CRD.

Scenario 2: Storage Version Changed Without Conversion

Symptom: kubectl get <resource> returns errors like no kind is registered for the version "v2" or conversion errors.

Cause: You changed the storage version but did not provide a conversion webhook, and existing objects are stored in the old version.

Recovery:

1. Set the storage version back to the previous one:

   kubectl patch crd <crd-name> --type='json' -p='[{"op": "replace", "path": "/spec/versions/0/storage", "value": true},{"op": "replace", "path": "/spec/versions/1/storage", "value": false}]'

(Adjust indices to match your versions.)

1. Implement a conversion webhook or manually migrate objects using kubectl get --raw and kubectl replace.
2. Then repeat the storage version change.

Scenario 3: Deleted CRD Orphaned Custom Resources

Symptom: You delete a CRD and all its custom resources are deleted immediately, causing data loss.

Cause: Cascading deletion is default. You intended to keep the data for later migration.

Recovery: Restore from etcd backup or any snapshots. To prevent this in the future, set spec.preserveUnknownFields: false is not enough; you need to export resources before deletion:

  kubectl get <plural> -o yaml > resources-backup.yaml

After recreating the CRD, apply the resources back.

Scenario 4: Controller Not Updated for New Fields

Symptom: New fields are ignored or cause controller crashes.

Cause: The controller's Go structs do not include the new fields, or the controller uses unstructured code with strict decoding.

Recovery: Update the controller to match the new schema, then restart it. To avoid this, use a conversion webhook to keep multiple API versions or coordinate CRD and controller releases.

Common Mistakes and How to Avoid Them

1. Omitting Structural Schema

Why it happens: Early CRDs often used preserveUnknownFields: true or minimal schemas for flexibility.

Risk: Kubernetes cannot prune unknown fields, leading to API server memory bloat and incompatibility with admission webhooks and CRD validation tools.

Avoidance: Always define a complete openAPIV3Schema with type: object and properties for all known fields. Set preserveUnknownFields: false (default in v1).

Example minimal structural schema:

schema:
  openAPIV3Schema:
    type: object
    properties:
      spec:
        type: object
        properties:
          size:
            type: integer
            minimum: 1

2. Forgetting Default Values for New Fields

Why it happens: Adding new fields without default can break existing objects that do not have that field set.

Risk: Controllers may become nil pointer to a field, causing crashes. Or admission webhooks reject old objects.

Avoidance: Define a default for every new field, or make the controller handle missing values gracefully.

Example:

properties:
  replicas:
    type: integer
    default: 1

3. Changing Field Types Incompatibly

Why it happens: Developers assume changing a field from string to integer is easy.

Risk: It breaks API compatibility. Existing clients and stored objects may fail.

Avoidance: Never change a field's type in the same API version. Introduce a new version (v2) with the changed type and use a conversion webhook.

4. Not Testing with kubectl explain

Why it happens: Teams rush to apply CRDs without checking documentation generation.

Risk: Users cannot discover fields properly; kubectl explain gives incomplete information.

Avoidance: After applying the CRD, run kubectl explain <resource>.<field> to ensure descriptions appear. Add description fields in the schema.

Example:

kubectl explain myresource.spec.size

Expected output includes description and type.

5. Ignoring Conversion Webhooks

Why it happens: Teams serve only one version initially, then later add a new version without a webhook.

Risk: When multiple versions are served, the API server cannot convert between them, causing errors for clients that use a specific version.

Avoidance: Implement a conversion webhook as soon as you have more than one version. Use tools like controller-runtime to generate webhooks.

Example webhook configuration in CRD:

conversion:
  strategy: Webhook
  webhook:
    conversionReviewVersions: ["v1", "v1beta1"]
    clientConfig:
      service:
        name: conversion-webhook
        namespace: default
        path: /convert

6. Pruning Custom Fields Accidentally

Why it happens: You add a field but do not include it in the schema, and the API server prunes it because unknown fields are dropped by default.

Risk: Data silently disappears when a user submits a resource with that field.

Avoidance: Ensure the schema includes every field your controller expects. Use x-kubernetes-preserve-unknown-fields: true only in specific nested contexts if necessary.

Example of preserving nested unknown fields:

properties:
  spec:
    type: object
    x-kubernetes-preserve-unknown-fields: true

But use sparingly; better to model all fields.

Operations Checklist

Maintain a repeatable CRD operations checklist. Assign an owner for each major step and review the checklist quarterly or after incidents.

Use this checklist as a starting point: adapt owners and frequencies to your team size and release cadence.

Conclusion

CRD configuration mistakes can have far-reaching consequences, from silent data loss to cluster-wide API failures. By understanding the common pitfalls, validating schemas rigorously, and establishing recovery playbooks, you can operate CRDs with confidence. Start with an inventory of your current CRDs, implement structural schemas, and test every change in a staging environment. The next step is to pick one CRD in your cluster, run the diagnostics described here, and fix any issues you find. Building these habits now will save you from painful debugging later.

A reliable workflow for CRDs is not just about avoiding mistakes; it is about making your extended Kubernetes API predictable, observable, and safe for everyone who uses it.