## Intro

Kubernetes Custom Resource Definitions (CRDs) extend the Kubernetes API, allowing platform teams and developers to model domain-specific resources. However, managing CRDs across CI/CD pipelines introduces unique challenges: schema evolution, controller dependency ordering, API version conversion, and rollback safety. This article provides practical, operational guidance for automating CRD lifecycle management, from initial validation to staged rollout and recovery.

We will cover version and environment inventory, safe configuration patterns, verification and diagnostics, failure modes with recovery runbooks, and a production-readiness operations checklist. Each section includes concrete kubectl commands, manifest examples, expected outputs, and decision points. The practices here are aimed at developers, DevOps consultants, and technical startup teams who need to ship CRD changes confidently.

## Version and Environment Inventory

Before automating any CRD change, establish a baseline of your cluster's current state. This inventory prevents assumptions and enables rollback decisions.

### Identify Cluster Version and CRD API Compatibility

Start by recording the Kubernetes server version and the version of any relevant controllers or operators.

kubectl version --short
# Example output:
# Client Version: v1.28.2
# Server Version: v1.27.5 
 Check the apiextensions.k8s.io API version supported for CRDs. In Kubernetes 1.16+, apiextensions.k8s.io/v1 is stable; older versions may use v1beta1 . Use kubectl api-versions to list available API groups.

kubectl api-versions | grep apiextensions
# apiextensions.k8s.io/v1 

### Inventory Existing CRDs and Their Storage Versions

 List all CRDs and their current storage version. The storage version determines how objects are persisted.

kubectl get crd -o custom-columns=NAME:.metadata.name,GROUP:.spec.group,STORAGE:.spec.versions[*].storage
# Example output:
# NAME GROUP STORAGE
# appconfigs.example.com example.com true,false
# backups.example.com example.com true 
 For a specific CRD, inspect its version details and preserve unknown fields.

kubectl get crd appconfigs.example.com -o yaml | grep -A5 versions:
# versions:
# - name: v1alpha1
# served: true
# storage: false
# - name: v1beta1
# served: true
# storage: true 

### Collect Prerequisite Resources and RBAC

 CRD changes often require updated RBAC rules for controllers. Capture current roles and cluster roles that interact with the CRD's API group.

kubectl get clusterrole,role -l app=crd-controller -o yaml
# or by API group:
kubectl get clusterrole -o yaml | grep -B2 'example.com' 
 Check if any admission webhooks are configured for the CRD, as they can block updates.

kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations -o yaml | grep -B3 'example.com' 

### Document Current State with Timestamps

 Create a snapshot directory with timestamp.

TIMESTAMP=$(date +%Y%m%d%H%M%S)
mkdir -p inventory-$TIMESTAMP
kubectl get crd -o yaml > inventory-$TIMESTAMP/crds.yaml
kubectl get deploy,sts,ds -n crd-controller-system -o yaml > inventory-$TIMESTAMP/controllers.yaml
kubectl get cm,secret -n crd-controller-system -o yaml > inventory-$TIMESTAMP/configs.yaml 
 This snapshot is your rollback reference. Store it in version control or object storage.

<style>
.eno-quiz-widget{margin:2rem 0;padding:1.5rem;border-radius:12px;background:var(--eno-surface-lowest,#f7f7f8);border:1px solid var(--eno-border-soft,#e2e2e6);font-family:var(--eno-font-body,Inter,sans-serif)}
.eno-quiz-widget .eno-quiz-kicker{font-family:var(--eno-font-label,"JetBrains Mono",monospace);font-size:.75rem;letter-spacing:.05em;text-transform:uppercase;color:var(--eno-text-muted,#5f5f68);margin:0 0 .5rem}
.eno-quiz-widget .eno-quiz-question{font-family:var(--eno-font-heading,"Hanken Grotesk",sans-serif);font-size:1.0625rem;font-weight:600;margin:0 0 1rem;color:var(--eno-text-strong,#1a1a1f)}
.eno-quiz-widget .eno-quiz-options{list-style:none;margin:0;padding:0;display:flex;flex-direction:column;gap:.5rem}
.eno-quiz-widget .eno-quiz-option{display:block;width:100%;min-height:44px;text-align:left;padding:.625rem .875rem;border-radius:8px;border:1.5px solid var(--eno-border-soft,#e2e2e6);background:#fff;font-size:.9375rem;cursor:pointer;transition:border-color var(--eno-motion-base,180ms ease-out),background var(--eno-motion-base,180ms ease-out)}
.eno-quiz-widget .eno-quiz-option:hover{border-color:var(--eno-primary,#0059bb)}
.eno-quiz-widget .eno-quiz-option:focus-visible{outline:none;box-shadow:0 0 0 3px rgb(0 89 187 / 0.15)}
.eno-quiz-widget .eno-quiz-option[aria-pressed="true"]{border-color:var(--eno-primary,#0059bb);background:rgb(0 89 187 / 0.06)}
.eno-quiz-widget .eno-quiz-option[data-correct="true"].eno-quiz-revealed{border-color:var(--eno-success,#17803a);background:rgb(23 128 58 / 0.08)}
.eno-quiz-widget .eno-quiz-option[data-correct="false"].eno-quiz-revealed.eno-quiz-was-selected{border-color:var(--eno-error,#ba1a1a);background:var(--eno-error-soft,#ffdad6)}
.eno-quiz-widget .eno-quiz-option-icon{display:inline-block;width:1.1em;margin-right:.4em;font-weight:700}
.eno-quiz-widget .eno-quiz-submit{margin-top:1rem;min-height:44px;padding:.5rem 1.25rem;border-radius:8px;border:none;background:var(--eno-primary,#0059bb);color:#fff;font-weight:600;font-size:.9375rem;cursor:pointer;transition:opacity var(--eno-motion-base,180ms ease-out)}
.eno-quiz-widget .eno-quiz-submit:disabled{opacity:.5;cursor:not-allowed}
.eno-quiz-widget .eno-quiz-submit:focus-visible{outline:none;box-shadow:0 0 0 3px rgb(0 89 187 / 0.15)}
.eno-quiz-widget .eno-quiz-explanation{margin-top:1rem;padding:.875rem 1rem;border-radius:8px;font-size:.9375rem;line-height:1.5;display:none}
.eno-quiz-widget .eno-quiz-explanation.eno-quiz-visible{display:block}
.eno-quiz-widget .eno-quiz-explanation.eno-quiz-correct{background:rgb(23 128 58 / 0.08);color:var(--eno-success,#17803a)}
.eno-quiz-widget .eno-quiz-explanation.eno-quiz-incorrect{background:var(--eno-error-soft,#ffdad6);color:var(--eno-error,#ba1a1a)}
@media (prefers-reduced-motion: reduce){.eno-quiz-widget *{transition:none!important}}
</style><div class="eno-quiz-widget" role="group" aria-label="Quick check question">
Quick check 1 of 2

What does the article recommend as the first step before automating any CRD change?

- Create a backup of the cluster
- Establish a baseline of the cluster's current state
- Install a CRD controller
- Define a conversion webhook

<button type="button" class="eno-quiz-submit" disabled>Submit</button>
<div class="eno-quiz-explanation">The article states: &#39;Before automating any CRD change, establish a baseline of your cluster&#39;s current state. This inventory prevents assumptions and enables rollback decisions.&#39;</div>
</div>
<script>
document.addEventListener('DOMContentLoaded', function () {
  document.querySelectorAll('.eno-quiz-widget:not([data-eno-quiz-bound])').forEach(function (widget) {
    widget.setAttribute('data-eno-quiz-bound', '1');
    var options = Array.prototype.slice.call(widget.querySelectorAll('.eno-quiz-option'));
    var submitBtn = widget.querySelector('.eno-quiz-submit');
    var explanation = widget.querySelector('.eno-quiz-explanation');
    var selected = null;
    options.forEach(function (opt) {
      opt.addEventListener('click', function () {
        if (widget.hasAttribute('data-eno-quiz-answered')) return;
        options.forEach(function (o) { o.setAttribute('aria-pressed', 'false'); });
        opt.setAttribute('aria-pressed', 'true');
        selected = opt;
        submitBtn.disabled = false;
      });
    });
    submitBtn.addEventListener('click', function () {
      if (!selected || widget.hasAttribute('data-eno-quiz-answered')) return;
      widget.setAttribute('data-eno-quiz-answered', '1');
      submitBtn.disabled = true;
      var correct = selected.getAttribute('data-correct') === 'true';
      options.forEach(function (o) {
        o.classList.add('eno-quiz-revealed');
        if (o === selected) o.classList.add('eno-quiz-was-selected');
        var icon = o.querySelector('.eno-quiz-option-icon');
        if (o.getAttribute('data-correct') === 'true') icon.textContent = '\u2713';
        else if (o === selected) icon.textContent = '\u2717';
      });
      explanation.classList.add('eno-quiz-visible', correct ? 'eno-quiz-correct' : 'eno-quiz-incorrect');
    });
  });
});
</script>

## Safe Configuration Path

A safe configuration path ensures that every change is scoped, reviewable, and reversible. For CRDs, this involves versioning, schema validation, and gradual rollout.

### Use Versioned CRD Manifests with Structural Schemas

Always specify a structural schema in your CRD. This enables validation and future conversion. Example minimal CRD:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
 name: appconfigs.example.com
spec:
 group: example.com
 names:
 kind: AppConfig
 listKind: AppConfigList
 plural: appconfigs
 singular: appconfig
 scope: Namespaced
 versions:
 - name: v1alpha1
 served: true
 storage: true
 schema:
 openAPIV3Schema:
 type: object
 properties:
 spec:
 type: object
 properties:
 replicas:
 type: integer
 minimum: 1
 image:
 type: string
 required: ["replicas", "image"] 
 Use kubectl apply --dry-run=server to validate without persisting.

kubectl apply -f crd.yaml --dry-run=server
# Expected: customresourcedefinition.apiextensions.k8s.io/appconfigs.example.com created (server dry run) 

### Maintain Multiple API Versions with Conversion Strategy

 If you need to support clients on different versions, define multiple versions and conversion. For simple field changes, None conversion may suffice.

spec:
 versions:
 - name: v1alpha1
 served: true
 storage: false
 schema: ...
 - name: v1beta1
 served: true
 storage: true
 schema: ...
 conversion:
 strategy: None 
 For complex changes, use a conversion webhook and test it rigorously.

### Enforce Least Privilege RBAC

Create dedicated roles for CI/CD pipelines that only allow specific operations on the CRD and its resources.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
 namespace: default
 name: crd-deployer
rules:
- apiGroups: ["apiextensions.k8s.io"]
 resources: ["customresourcedefinitions"]
 verbs: ["get", "list", "watch", "create", "patch"]
- apiGroups: ["example.com"]
 resources: ["appconfigs", "appconfigs/status"]
 verbs: ["get", "list", "watch", "create", "update", "patch"] 
 Avoid cluster-admin for pipeline service accounts.

### Stage Rollouts with Canary and Progressive Delivery

Apply CRD changes to a staging namespace or cluster first. Use kubectl apply --server-side for atomic updates if needed.

kubectl apply -f crd.yaml --server-side --field-manager=ci-pipeline 
 Monitor controller health before promoting.

kubectl rollout status deployment/crd-controller -n crd-controller-system --timeout=60s
# Expected: deployment "crd-controller" successfully rolled out 

## Verification and Diagnostics

 Verification ensures that your CRD change behaves as expected. Use a combination of declarative checks and real resource creation.

### Validate CRD Existence and Schema

After applying, confirm the CRD is established and accepted.

kubectl get crd appconfigs.example.com -o jsonpath='{.status.conditions[?(@.type=="Established")].status}'
# Expected: True

kubectl get crd appconfigs.example.com -o jsonpath='{.status.conditions[?(@.type=="NamesAccepted")].status}'
# Expected: True 
 Check for stored versions.

kubectl get crd appconfigs.example.com -o jsonpath='{.status.storedVersions}'
# Expected: ["v1beta1"] if v1beta1 is storage 

### Create a Test Custom Resource

 Create a sample resource to verify validation and controller behavior.

apiVersion: example.com/v1beta1
kind: AppConfig
metadata:
 name: test-appconfig
 namespace: default
spec:
 replicas: 2
 image: nginx:1.21 
 kubectl apply -f test-cr.yaml
kubectl get appconfig test-appconfig -o yaml 
 If the CRD controller sets status, check it.

kubectl get appconfig test-appconfig -o jsonpath='{.status.conditions}' 

### Test Validation Failures

 Intentionally create an invalid resource to ensure schema enforcement.

apiVersion: example.com/v1beta1
kind: AppConfig
metadata:
 name: bad-appconfig
spec:
 replicas: 0 # violates minimum 1 
 kubectl apply -f bad-cr.yaml
# Expected error:
# The AppConfig "bad-appconfig" is invalid: spec.replicas: Invalid value: 0: spec.replicas in body should be greater than or equal to 1 

### Diagnose Controller Issues

 If resources are not reconciled, inspect controller logs.

kubectl logs -n crd-controller-system deploy/crd-controller --tail=50
# Look for errors like "failed to list appconfigs" or conversion errors. 
 Check events in the namespace.

kubectl get events -n default --sort-by=.lastTimestamp | grep appconfig 

 Quick check 2 of 2

According to the article, what is the purpose of using `kubectl apply --dry-run=server` when applying a CRD?

- To apply the CRD without persisting it
- To apply the CRD and immediately delete it
- To apply the CRD in a dry run mode for testing
- To apply the CRD and ignore any errors

<button type="button" class="eno-quiz-submit" disabled>Submit</button>
<div class="eno-quiz-explanation">The text says: &#39;Use kubectl apply --dry-run=server to validate without persisting.&#39;</div>
</div>
<script>
document.addEventListener('DOMContentLoaded', function () {
  document.querySelectorAll('.eno-quiz-widget:not([data-eno-quiz-bound])').forEach(function (widget) {
    widget.setAttribute('data-eno-quiz-bound', '1');
    var options = Array.prototype.slice.call(widget.querySelectorAll('.eno-quiz-option'));
    var submitBtn = widget.querySelector('.eno-quiz-submit');
    var explanation = widget.querySelector('.eno-quiz-explanation');
    var selected = null;
    options.forEach(function (opt) {
      opt.addEventListener('click', function () {
        if (widget.hasAttribute('data-eno-quiz-answered')) return;
        options.forEach(function (o) { o.setAttribute('aria-pressed', 'false'); });
        opt.setAttribute('aria-pressed', 'true');
        selected = opt;
        submitBtn.disabled = false;
      });
    });
    submitBtn.addEventListener('click', function () {
      if (!selected || widget.hasAttribute('data-eno-quiz-answered')) return;
      widget.setAttribute('data-eno-quiz-answered', '1');
      submitBtn.disabled = true;
      var correct = selected.getAttribute('data-correct') === 'true';
      options.forEach(function (o) {
        o.classList.add('eno-quiz-revealed');
        if (o === selected) o.classList.add('eno-quiz-was-selected');
        var icon = o.querySelector('.eno-quiz-option-icon');
        if (o.getAttribute('data-correct') === 'true') icon.textContent = '\u2713';
        else if (o === selected) icon.textContent = '\u2717';
      });
      explanation.classList.add('eno-quiz-visible', correct ? 'eno-quiz-correct' : 'eno-quiz-incorrect');
    });
  });
});
</script>

## Failure Modes and Recovery

Plan for failures before they happen. The following are common CRD CI/CD failure modes and recovery steps.

### Failure: CRD Update Rejected Due to Invalid Schema

Symptom: kubectl apply returns 422 or 400 error with details.

Recovery:

- Validate the manifest with kubectl apply --dry-run=server .

- Correct the schema based on error messages.

- Re-apply.

Example:

The CustomResourceDefinition "appconfigs.example.com" is invalid: spec.versions[0].schema.openAPIV3Schema.properties.spec.properties.replicas.type: Required value: must not be empty for specified object fields 

### Failure: Controller Unable to Reconcile New Storage Version

 Symptom: CRD applies, but status.conditions shows NonStructuralSchema or controller logs show conversion errors.

Recovery:

- Check kubectl get crd appconfigs.example.com -o yaml for status conditions.

- If NonStructuralSchema is true, inspect the schema for unsupported fields (e.g., nullable: true in certain positions) and fix.

- If conversion errors, ensure conversion webhook is reachable and returns proper responses.

- If necessary, revert to previous CRD manifest: kubectl apply -f backup/crds.yaml .

### Failure: Breaking Change Removes Served Version Still in Use

Symptom: Clients using old API version get 404 or no kind is registered .

Recovery:

- Identify clients via audit logs or admission requests.

- Restore the served version temporarily: set served: true for the old version and re-apply.

- Coordinate client migration, then deprecate again in a future release.

### Rollback Procedure for CRDs

Because CRDs are cluster-scoped, rollback must be careful.

- Restore the backup CRD manifest.

kubectl apply -f inventory-$TIMESTAMP/crds.yaml 

- Ensure the storage version is set to the previous one.

- Wait for Established condition True.

- Verify controllers are healthy.

- Delete any custom resources that were created with a now-unserved version (if necessary, after backup).

### Failure: Pipeline Service Account Lacks Permissions

 Symptom: CI job fails with forbidden: User "system:serviceaccount:ci:deployer" cannot create resource "customresourcedefinitions" .

Recovery:

- Review the Role/RoleBinding used by the pipeline.

- Add required API groups and verbs.

- Apply RBAC changes and re-run pipeline.

## Operations Checklist

Use this checklist before and after CRD automation changes to ensure operational safety.

### Pre-Deployment Checklist

- [ ] Cluster version and API compatibility documented.

- [ ] Existing CRDs and storage versions inventoried.

- [ ] Backup of current CRD YAMLs stored.

- [ ] New CRD manifest uses apiextensions.k8s.io/v1 and structural schema.

- [ ] Schema changes validated with kubectl apply --dry-run=server .

- [ ] RBAC updated for controller and pipeline.

- [ ] Conversion strategy defined if multiple versions.

- [ ] Rollback plan documented with exact commands.

### Deployment Execution

# 1. Apply the CRD
kubectl apply -f crd.yaml --server-side

# 2. Wait for CRD to be established
kubectl wait --for=condition=Established --timeout=60s crd/appconfigs.example.com

# 3. Deploy/update controller if needed
kubectl apply -f controller-deployment.yaml
kubectl rollout status deploy/crd-controller -n crd-controller-system --timeout=120s

# 4. Create smoke test CR
kubectl apply -f test-cr.yaml
kubectl wait --for=condition=Ready --timeout=30s appconfig/test-appconfig 

### Post-Deployment Verification

- [ ] CRD condition Established is True.

- [ ] Test CR created and reconciled.

- [ ] Invalid resource correctly rejected.

- [ ] Controller logs have no new errors.

- [ ] Existing custom resources still accessible and healthy.

- [ ] Metrics/alerts show no spike in API errors.

### Continuous Monitoring

 Set up alerts for:

- CRD apiextensions.k8s.io/v1 API errors ( apiserver_request_total with group=example.com and code >= 400).

- Controller reconciliation failures.

- Webhook latency if conversion webhooks are used.

Example Prometheus alert rule:

- alert: CRDControllerDown
 expr: up{job="crd-controller"} == 0
 for: 5m
 labels:
 severity: critical
 annotations:
 summary: "CRD controller is down" 

## Conclusion

 Automating Kubernetes CRD CI/CD requires discipline across versioning, validation, deployment, and recovery. By inventorying your current state, following a safe configuration path, verifying with concrete signals, and preparing runbooks for common failures, you reduce the risk of API inconsistency and controller disruption.

Start small: pick one low-risk CRD improvement, apply the checklist, and measure the outcome. As your confidence grows, integrate these practices into your GitOps pipelines with automated tests and progressive delivery. The goal is not just to deploy CRDs, but to operate them safely and reversibly.