Kubernetes Custom Resource Definition Local Lab Setup with Practical Examples

Intro

Kubernetes Custom Resource Definitions (CRDs) let you extend the Kubernetes API with your own resource types. This article walks through a complete local lab setup for CRDs, targeted at developers, DevOps consultants, and technical startup teams. It connects Kubernetes CRD setup, testing, examples, and development to specific commands, expected output, failure signals, and recovery decisions.

The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached. By the end, you will have a reproducible local environment, working CRD examples, and troubleshooting patterns you can apply to your own operators.

Version and Environment Inventory

Before applying any CRD, you must know your environment. This section covers supported version ranges, prerequisites, read-only observation, the smallest justified change, and verification commands.

Prerequisites and Version Checks

For local CRD development, you need a Kubernetes cluster. The easiest way is to use kind (Kubernetes in Docker), minikube, or Docker Desktop's Kubernetes. We will use kind in this article because it is lightweight and scriptable. Ensure you have the following installed:

• kubectl (v1.25 or later recommended)
• kind (v0.17.0 or later)
• Docker (for kind nodes)
• jq (optional, for filtering JSON output)

Verify versions:

kubectl version --client
# Example output: Client Version: v1.27.3

kind version
# Example output: kind v0.20.0 go1.20.4 linux/amd64

docker version --format '{{.Server.Version}}'
# Example output: 24.0.2

Read-Only Observation

Before making changes, observe the current cluster state. If you don't have a cluster yet, create one, but first check if any cluster is already configured:

kubectl config current-context
# Example output: kind-kind (if already exists) or error if none

If you see a context, inspect its nodes and API server version:

kubectl get nodes
# Example output: NAME                 STATUS   ROLES           AGE   VERSION
#                  kind-control-plane   Ready    control-plane   10m   v1.27.3

kubectl version --short
# Example output: Client Version: v1.27.3
#                 Server Version: v1.27.3

If no cluster exists, create one for this lab:

kind create cluster --name crd-lab --image kindest/node:v1.27.3
# Example output:
# Creating cluster "crd-lab" ...
#  ✓ Ensuring node image (kindest/node:v1.27.3) 🖼
#  ✓ Preparing nodes 📦
#  ✓ Writing configuration 📜
#  ✓ Starting control-plane 🕹️
#  ✓ Installing CNI 🔌
#  ✓ Installing StorageClass 💾
# Set kubectl context to "kind-crd-lab"

Now set the context to the new cluster:

kubectl config use-context kind-crd-lab
# Example output: Switched to context "kind-crd-lab".

Smallest Justified Change

Now that we have a cluster, we will add a CustomResourceDefinition. A CRD defines a new kind and its schema. We'll start with a minimal CRD for a fictional Widget resource.

Create a file widget-crd.yaml:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: widgets.example.com
spec:
  group: example.com
  names:
    kind: Widget
    listKind: WidgetList
    plural: widgets
    singular: widget
  scope: Namespaced
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                size:
                  type: integer
                  minimum: 1
                  maximum: 10
                color:
                  type: string
                  enum: [red, green, blue]
              required: [size, color]

This CRD defines a namespaced resource widgets in the example.com group, with a schema that requires size (integer between 1 and 10) and color (one of red, green, blue). Apply it:

kubectl apply -f widget-crd.yaml
# Example output: customresourcedefinition.apiextensions.k8s.io/widgets.example.com created

Verify the CRD is established:

kubectl get crd widgets.example.com
# Example output:
# NAME                    CREATED AT
# widgets.example.com     2023-07-01T12:00:00Z

Check its conditions:

kubectl get crd widgets.example.com -o jsonpath='{.status.conditions[?(@.type=="Established")].status}'
# Example output: True

Now the cluster has a new API endpoint. You can list the custom resources (currently none):

kubectl get widgets
# Example output: No resources found in default namespace.

Safe Configuration Path

In this section, we expand the previous CRD into a more realistic scenario: a namespaced custom resource with validation defaults, additional printer columns, and a simple controller pattern simulated via a shell script. We also cover how to make safe changes and roll back if needed.

Namespaced CRD with Status Subresource

Add a status subresource and printer columns to make kubectl get more informative. Modify widget-crd.yaml (or create a new file widget-crd-v2.yaml):

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: widgets.example.com
spec:
  group: example.com
  names:
    kind: Widget
    listKind: WidgetList
    plural: widgets
    singular: widget
    shortNames:
      - wd
  scope: Namespaced
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                size:
                  type: integer
                  minimum: 1
                  maximum: 10
                color:
                  type: string
                  enum: [red, green, blue]
              required: [size, color]
            status:
              type: object
              properties:
                phase:
                  type: string
                  enum: [Pending, Running, Failed]
                message:
                  type: string
      subresources:
        status: {}
      additionalPrinterColumns:
        - name: Size
          type: integer
          jsonPath: .spec.size
        - name: Color
          type: string
          jsonPath: .spec.color
        - name: Phase
          type: string
          jsonPath: .status.phase

Apply the updated CRD:

kubectl apply -f widget-crd-v2.yaml
# Example output: customresourcedefinition.apiextensions.k8s.io/widgets.example.com configured

Note: updating a CRD is a safe change if you only add fields, subresources, or printer columns. Removing fields or changing validation may affect existing custom resources.

Now create a sample Widget:

cat <<EOF | kubectl apply -f -
apiVersion: example.com/v1
kind: Widget
metadata:
  name: my-widget
  namespace: default
spec:
  size: 5
  color: blue
EOF
# Example output: widget.example.com/my-widget created

Check it with custom columns:

kubectl get widgets
# Example output:
# NAME        SIZE   COLOR   PHASE
# my-widget   5      blue

Simulating a Controller with a Bash Watch Loop

A real controller watches for changes and updates status. For safety, we can simulate a minimal controller using a shell loop that lists all widgets and sets a status if none exists. This is not production-grade, but it demonstrates the pattern.

First, create a small script simple-controller.sh:

#!/usr/bin/env bash
set -euo pipefail

while true; do
  for widget in $(kubectl get widgets -o jsonpath='{.items[*].metadata.name}'); do
    phase=$(kubectl get widget "$widget" -o jsonpath='{.status.phase}')
    if [ -z "$phase" ]; then
      echo "Setting status for $widget to Pending"
      kubectl patch widget "$widget" --type merge -p '{"status":{"phase":"Pending","message":"Processing"}}'
    fi
  done
  sleep 10
done

Make it executable and run it in a separate terminal (or background):

chmod +x simple-controller.sh
./simple-controller.sh

After a few seconds, check the widget status:

kubectl get widget my-widget -o yaml
# Expected output snippet:
# status:
#   message: Processing
#   phase: Pending

This demonstrates how a controller can manage status updates safely using kubectl patch with merge strategy.

Rollback and Recovery

If you apply a CRD change that breaks existing resources, you can sometimes revert by applying a previous manifest. Keep versioned manifests in a Git repository. For example, to revert to v1:

kubectl apply -f widget-crd.yaml
# Example output: customresourcedefinition.apiextensions.k8s.io/widgets.example.com configured

If a CRD update invalidates existing custom resources, you may need to edit or delete those resources first. Always test in a local lab before touching production.

Verification and Diagnostics

Verification is not just about kubectl apply; it's about confirming the resource behaves as expected. This section provides concrete checks, common failure signals, and diagnostic commands.

Functional Verification of the CRD Schema

The schema we defined enforces constraints. Try creating a widget with an invalid color:

cat <<EOF | kubectl apply -f -
apiVersion: example.com/v1
kind: Widget
metadata:
  name: bad-widget
spec:
  size: 7
  color: purple
EOF
# Expected error:
# The Widget "bad-widget" is invalid: spec.color: Unsupported value: "purple": supported values: "red", "green", "blue"

The API server rejects the request before storing it. This confirms schema validation works.

Test size bounds:

cat <<EOF | kubectl apply -f -
apiVersion: example.com/v1
kind: Widget
metadata:
  name: big-widget
spec:
  size: 11
  color: red
EOF
# Expected error:
# The Widget "big-widget" is invalid: spec.size: Invalid value: 11: spec.size in body should be less than or equal to 10

Diagnostic Commands for CRD Issues

If your CRD isn't establishing, check its status conditions and events.

kubectl describe crd widgets.example.com
# Look for Events at the bottom, e.g.:
# Events:
#   Type     Reason                 Age   From                            Message
#   Warning  NonStructuralSchema    2m    customresourcedefinition-api   spec.validation.openAPIV3Schema.type: Required value: must not be empty at the root

Common failure signals:

• NonStructuralSchema: The schema is not structural (e.g., missing type: object).
• NoServedVersions: The versions list is empty or all served: false.
• Terminating: The CRD is stuck in deletion due to finalizers.

If you see Terminating, inspect the finalizers:

kubectl get crd widgets.example.com -o jsonpath='{.metadata.finalizers}'
# Example output: ["customresourcecleanup.apiextensions.k8s.io"]

If you need to force deletion (only in lab, not production), remove the finalizer:

kubectl patch crd widgets.example.com -p '{"metadata":{"finalizers":[]}}' --type=merge
# Example output: customresourcedefinition.apiextensions.k8s.io/widgets.example.com patched

Then check if it disappears.

Verifying Controller Behavior

If your custom controller isn't picking up changes, check its logs. Since our simple controller runs in a local shell, its output goes to the terminal. In a real pod, you'd use:

kubectl logs <controller-pod-name>

Common controller issues:

• RBAC misconfiguration: The controller doesn't have permission to get/list/watch widgets. Check with kubectl auth can-i list widgets --as=system:serviceaccount:default:controller-sa
• CRD not served: kubectl get widgets returns error "the server could not find the requested resource". Ensure CRD is applied and served: true.
• API deprecation: If using an old API version, ensure apiextensions.k8s.io/v1 (not v1beta1).

Failure Modes and Recovery

Operators must plan for failure. This section details common failure modes for CRDs and custom resources, along with recovery steps.

Failure Mode: CRD Deletion Leaves Custom Resources Orphaned

When you delete a CRD, all custom resources of that type are deleted immediately (garbage collected). If you accidentally delete a CRD with important data, you may lose resources. Recovery requires backups.

Before deleting a CRD, back up all custom resources:

kubectl get widgets -o yaml > widgets-backup.yaml

If you delete the CRD and need to restore, recreate the CRD first, then reapply the backup:

kubectl delete crd widgets.example.com
# Example output: customresourcedefinition.apiextensions.k8s.io "widgets.example.com" deleted

# Now recreate CRD (assuming your manifest is saved)
kubectl apply -f widget-crd.yaml
# Then restore resources
kubectl apply -f widgets-backup.yaml

But note: if the custom resources had status fields, they may not be preserved because backups captured before deletion might lack status. Best practice: use a backup tool like Velero.

Failure Mode: Schema Too Restrictive Blocks Updates

If a schema prevents legitimate updates because of a new field, you can relax the schema by adding the field as optional. Example: add a description field to the schema:

spec:
  description:
    type: string

Apply the updated CRD, and existing resources are unaffected because the field is optional.

To avoid breaking changes, follow these rules:

• Never remove a field from the schema.
• Never change a field's type.
• Only add new optional fields.

For major changes, introduce a new API version (e.g., v2) and let old resources migrate gradually.

Failure Mode: Conversion Webhook Misconfiguration

If you use conversion webhooks for multiple versions, a misconfigured webhook can break all API calls for the CRD. To recover, patch the CRD to remove the conversion strategy or fix the webhook service.

Example: check conversion settings:

kubectl get crd widgets.example.com -o jsonpath='{.spec.conversion}'
# Example output: {"strategy":"Webhook","webhook":{"clientConfig":{"service":{"name":"conversion-webhook","namespace":"default","path":"/convert","port":443}},"conversionReviewVersions":["v1"]}}

If the webhook is down, API requests may fail with conversion webhook not found. Temporarily change strategy to None (if only one version served) to restore access:

kubectl patch crd widgets.example.com --type merge -p '{"spec":{"conversion":{"strategy":"None"}}}'

Note: this requires all versions to be served by the API server without conversion, so ensure you have only one version or that all versions are structurally identical.

Recovery Verification

After any recovery action, verify by listing and describing resources, checking CRD conditions, and ensuring controllers are running.

kubectl get crd widgets.example.com -o jsonpath='{.status.conditions}' | jq .

Operations Checklist

Use this checklist before and after any CRD change in your local lab (or as a template for production). Each item includes a concrete example.

Conclusion

Kubernetes Custom Resource Definition local lab setup with practical examples is essential for safe development of custom operators. This article provided a step-by-step guide: environment inventory, minimal CRD, safe configuration with subresources and printer columns, verification commands, and failure recovery scenarios.

To reinforce learning, practice these tasks:

1. Extend the Widget CRD with a new optional field and a validation pattern.
2. Implement a second API version (v2) with a conversion strategy (None or Webhook) and test migration.
3. Deploy a real controller using Kubernetes Deployment and ServiceAccount with proper RBAC.
4. Use kubectl explain widgets to explore the generated API documentation.

Remember the operational safety principles: observe before changing, limit blast radius, use placeholders, verify, and document recovery. With a solid local lab, you can iterate faster and reduce production risk.

Next step: choose one low-risk verification from the checklist (e.g., schema validation), record current state, run the check, compare with expected output, and review dependencies like RBAC roles for your controller.

A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision.