Kubernetes Custom Resource Definition CI/CD Automation: Practical Examples and Operational Runbooks

Intro

Kubernetes Custom Resource Definitions (CRDs) extend the Kubernetes API, allowing platform teams and developers to model domain-specific resources. However, managing CRDs across CI/CD pipelines introduces unique challenges: schema evolution, controller dependency ordering, API version conversion, and rollback safety. This article provides practical, operational guidance for automating CRD lifecycle management, from initial validation to staged rollout and recovery.

We will cover version and environment inventory, safe configuration patterns, verification and diagnostics, failure modes with recovery runbooks, and a production-readiness operations checklist. Each section includes concrete kubectl commands, manifest examples, expected outputs, and decision points. The practices here are aimed at developers, DevOps consultants, and technical startup teams who need to ship CRD changes confidently.

Version and Environment Inventory

Before automating any CRD change, establish a baseline of your cluster's current state. This inventory prevents assumptions and enables rollback decisions.

Identify Cluster Version and CRD API Compatibility

Start by recording the Kubernetes server version and the version of any relevant controllers or operators.

kubectl version --short
# Example output:
# Client Version: v1.28.2
# Server Version: v1.27.5

Check the apiextensions.k8s.io API version supported for CRDs. In Kubernetes 1.16+, apiextensions.k8s.io/v1 is stable; older versions may use v1beta1. Use kubectl api-versions to list available API groups.

kubectl api-versions | grep apiextensions
# apiextensions.k8s.io/v1

Inventory Existing CRDs and Their Storage Versions

List all CRDs and their current storage version. The storage version determines how objects are persisted.

kubectl get crd -o custom-columns=NAME:.metadata.name,GROUP:.spec.group,STORAGE:.spec.versions[*].storage
# Example output:
# NAME                          GROUP                 STORAGE
# appconfigs.example.com        example.com           true,false
# backups.example.com           example.com           true

For a specific CRD, inspect its version details and preserve unknown fields.

kubectl get crd appconfigs.example.com -o yaml | grep -A5 versions:
# versions:
# - name: v1alpha1
#   served: true
#   storage: false
# - name: v1beta1
#   served: true
#   storage: true

Collect Prerequisite Resources and RBAC

CRD changes often require updated RBAC rules for controllers. Capture current roles and cluster roles that interact with the CRD's API group.

kubectl get clusterrole,role -l app=crd-controller -o yaml
# or by API group:
kubectl get clusterrole -o yaml | grep -B2 'example.com'

Check if any admission webhooks are configured for the CRD, as they can block updates.

kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations -o yaml | grep -B3 'example.com'

Document Current State with Timestamps

Create a snapshot directory with timestamp.

TIMESTAMP=$(date +%Y%m%d%H%M%S)
mkdir -p inventory-$TIMESTAMP
kubectl get crd -o yaml > inventory-$TIMESTAMP/crds.yaml
kubectl get deploy,sts,ds -n crd-controller-system -o yaml > inventory-$TIMESTAMP/controllers.yaml
kubectl get cm,secret -n crd-controller-system -o yaml > inventory-$TIMESTAMP/configs.yaml

This snapshot is your rollback reference. Store it in version control or object storage.

Safe Configuration Path

A safe configuration path ensures that every change is scoped, reviewable, and reversible. For CRDs, this involves versioning, schema validation, and gradual rollout.

Use Versioned CRD Manifests with Structural Schemas

Always specify a structural schema in your CRD. This enables validation and future conversion. Example minimal CRD:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: appconfigs.example.com
spec:
  group: example.com
  names:
    kind: AppConfig
    listKind: AppConfigList
    plural: appconfigs
    singular: appconfig
  scope: Namespaced
  versions:
    - name: v1alpha1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                replicas:
                  type: integer
                  minimum: 1
                image:
                  type: string
              required: ["replicas", "image"]

Use kubectl apply --dry-run=server to validate without persisting.

kubectl apply -f crd.yaml --dry-run=server
# Expected: customresourcedefinition.apiextensions.k8s.io/appconfigs.example.com created (server dry run)

Maintain Multiple API Versions with Conversion Strategy

If you need to support clients on different versions, define multiple versions and conversion. For simple field changes, None conversion may suffice.

spec:
  versions:
    - name: v1alpha1
      served: true
      storage: false
      schema: ...
    - name: v1beta1
      served: true
      storage: true
      schema: ...
  conversion:
    strategy: None

For complex changes, use a conversion webhook and test it rigorously.

Enforce Least Privilege RBAC

Create dedicated roles for CI/CD pipelines that only allow specific operations on the CRD and its resources.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: crd-deployer
rules:
- apiGroups: ["apiextensions.k8s.io"]
  resources: ["customresourcedefinitions"]
  verbs: ["get", "list", "watch", "create", "patch"]
- apiGroups: ["example.com"]
  resources: ["appconfigs", "appconfigs/status"]
  verbs: ["get", "list", "watch", "create", "update", "patch"]

Avoid cluster-admin for pipeline service accounts.

Stage Rollouts with Canary and Progressive Delivery

Apply CRD changes to a staging namespace or cluster first. Use kubectl apply --server-side for atomic updates if needed.

kubectl apply -f crd.yaml --server-side --field-manager=ci-pipeline

Monitor controller health before promoting.

kubectl rollout status deployment/crd-controller -n crd-controller-system --timeout=60s
# Expected: deployment "crd-controller" successfully rolled out

Verification and Diagnostics

Verification ensures that your CRD change behaves as expected. Use a combination of declarative checks and real resource creation.

Validate CRD Existence and Schema

After applying, confirm the CRD is established and accepted.

kubectl get crd appconfigs.example.com -o jsonpath='{.status.conditions[?(@.type=="Established")].status}'
# Expected: True

kubectl get crd appconfigs.example.com -o jsonpath='{.status.conditions[?(@.type=="NamesAccepted")].status}'
# Expected: True

Check for stored versions.

kubectl get crd appconfigs.example.com -o jsonpath='{.status.storedVersions}'
# Expected: ["v1beta1"] if v1beta1 is storage

Create a Test Custom Resource

Create a sample resource to verify validation and controller behavior.

apiVersion: example.com/v1beta1
kind: AppConfig
metadata:
  name: test-appconfig
  namespace: default
spec:
  replicas: 2
  image: nginx:1.21

kubectl apply -f test-cr.yaml
kubectl get appconfig test-appconfig -o yaml

If the CRD controller sets status, check it.

kubectl get appconfig test-appconfig -o jsonpath='{.status.conditions}'

Test Validation Failures

Intentionally create an invalid resource to ensure schema enforcement.

apiVersion: example.com/v1beta1
kind: AppConfig
metadata:
  name: bad-appconfig
spec:
  replicas: 0   # violates minimum 1

kubectl apply -f bad-cr.yaml
# Expected error:
# The AppConfig "bad-appconfig" is invalid: spec.replicas: Invalid value: 0: spec.replicas in body should be greater than or equal to 1

Diagnose Controller Issues

If resources are not reconciled, inspect controller logs.

kubectl logs -n crd-controller-system deploy/crd-controller --tail=50
# Look for errors like "failed to list appconfigs" or conversion errors.

Check events in the namespace.

kubectl get events -n default --sort-by=.lastTimestamp | grep appconfig

Failure Modes and Recovery

Plan for failures before they happen. The following are common CRD CI/CD failure modes and recovery steps.

Failure: CRD Update Rejected Due to Invalid Schema

Symptom: kubectl apply returns 422 or 400 error with details.

Recovery:

1. Validate the manifest with kubectl apply --dry-run=server.
2. Correct the schema based on error messages.
3. Re-apply.

Example:

The CustomResourceDefinition "appconfigs.example.com" is invalid: spec.versions[0].schema.openAPIV3Schema.properties.spec.properties.replicas.type: Required value: must not be empty for specified object fields

Failure: Controller Unable to Reconcile New Storage Version

Symptom: CRD applies, but status.conditions shows NonStructuralSchema or controller logs show conversion errors.

Recovery:

1. Check kubectl get crd appconfigs.example.com -o yaml for status conditions.
2. If NonStructuralSchema is true, inspect the schema for unsupported fields (e.g., nullable: true in certain positions) and fix.
3. If conversion errors, ensure conversion webhook is reachable and returns proper responses.
4. If necessary, revert to previous CRD manifest: kubectl apply -f backup/crds.yaml.

Failure: Breaking Change Removes Served Version Still in Use

Symptom: Clients using old API version get 404 or no kind is registered.

Recovery:

1. Identify clients via audit logs or admission requests.
2. Restore the served version temporarily: set served: true for the old version and re-apply.
3. Coordinate client migration, then deprecate again in a future release.

Rollback Procedure for CRDs

Because CRDs are cluster-scoped, rollback must be careful.

1. Restore the backup CRD manifest.

kubectl apply -f inventory-$TIMESTAMP/crds.yaml

1. Ensure the storage version is set to the previous one.
2. Wait for Established condition True.
3. Verify controllers are healthy.
4. Delete any custom resources that were created with a now-unserved version (if necessary, after backup).

Failure: Pipeline Service Account Lacks Permissions

Symptom: CI job fails with forbidden: User "system:serviceaccount:ci:deployer" cannot create resource "customresourcedefinitions".

Recovery:

1. Review the Role/RoleBinding used by the pipeline.
2. Add required API groups and verbs.
3. Apply RBAC changes and re-run pipeline.

Operations Checklist

Use this checklist before and after CRD automation changes to ensure operational safety.

Pre-Deployment Checklist

• [ ] Cluster version and API compatibility documented.
• [ ] Existing CRDs and storage versions inventoried.
• [ ] Backup of current CRD YAMLs stored.
• [ ] New CRD manifest uses apiextensions.k8s.io/v1 and structural schema.
• [ ] Schema changes validated with kubectl apply --dry-run=server.
• [ ] RBAC updated for controller and pipeline.
• [ ] Conversion strategy defined if multiple versions.
• [ ] Rollback plan documented with exact commands.

Deployment Execution

# 1. Apply the CRD
kubectl apply -f crd.yaml --server-side

# 2. Wait for CRD to be established
kubectl wait --for=condition=Established --timeout=60s crd/appconfigs.example.com

# 3. Deploy/update controller if needed
kubectl apply -f controller-deployment.yaml
kubectl rollout status deploy/crd-controller -n crd-controller-system --timeout=120s

# 4. Create smoke test CR
kubectl apply -f test-cr.yaml
kubectl wait --for=condition=Ready --timeout=30s appconfig/test-appconfig

Post-Deployment Verification

• [ ] CRD condition Established is True.
• [ ] Test CR created and reconciled.
• [ ] Invalid resource correctly rejected.
• [ ] Controller logs have no new errors.
• [ ] Existing custom resources still accessible and healthy.
• [ ] Metrics/alerts show no spike in API errors.

Continuous Monitoring

Set up alerts for:

• CRD apiextensions.k8s.io/v1 API errors (apiserver_request_total with group=example.com and code >= 400).
• Controller reconciliation failures.
• Webhook latency if conversion webhooks are used.

Example Prometheus alert rule:

- alert: CRDControllerDown
  expr: up{job="crd-controller"} == 0
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "CRD controller is down"

Conclusion

Automating Kubernetes CRD CI/CD requires discipline across versioning, validation, deployment, and recovery. By inventorying your current state, following a safe configuration path, verifying with concrete signals, and preparing runbooks for common failures, you reduce the risk of API inconsistency and controller disruption.

Start small: pick one low-risk CRD improvement, apply the checklist, and measure the outcome. As your confidence grows, integrate these practices into your GitOps pipelines with automated tests and progressive delivery. The goal is not just to deploy CRDs, but to operate them safely and reversibly.