## Intro

Kubernetes Custom Resource Definitions (CRDs) let you extend the Kubernetes API with your own resource types. This article walks through a complete local lab setup for CRDs, targeted at developers, DevOps consultants, and technical startup teams. It connects Kubernetes CRD setup, testing, examples, and development to specific commands, expected output, failure signals, and recovery decisions.

The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached. By the end, you will have a reproducible local environment, working CRD examples, and troubleshooting patterns you can apply to your own operators.

## Version and Environment Inventory

Before applying any CRD, you must know your environment. This section covers supported version ranges, prerequisites, read-only observation, the smallest justified change, and verification commands.

### Prerequisites and Version Checks

For local CRD development, you need a Kubernetes cluster. The easiest way is to use `kind` (Kubernetes in Docker), `minikube`, or Docker Desktop's Kubernetes. We will use `kind` in this article because it is lightweight and scriptable. Ensure you have the following installed:

- `kubectl` (v1.25 or later recommended)
- `kind` (v0.17.0 or later)
- Docker (for `kind` nodes)
- `jq` (optional, for filtering JSON output)

Verify versions:

```bash
kubectl version --client
# Example output: Client Version: v1.27.3

kind version
# Example output: kind v0.20.0 go1.20.4 linux/amd64

docker version --format '{{.Server.Version}}'
# Example output: 24.0.2
```

### Read-Only Observation

Before making changes, observe the current cluster state. If you don't have a cluster yet, create one, but first check if any cluster is already configured:

```bash
kubectl config current-context
# Example output: kind-kind (if already exists) or error if none
```

If you see a context, inspect its nodes and API server version:

```bash
kubectl get nodes
# Example output: NAME                 STATUS   ROLES           AGE   VERSION
#                  kind-control-plane   Ready    control-plane   10m   v1.27.3

kubectl version --short
# Example output: Client Version: v1.27.3
#                 Server Version: v1.27.3
```

If no cluster exists, create one for this lab:

```bash
kind create cluster --name crd-lab --image kindest/node:v1.27.3
# Example output:
# Creating cluster "crd-lab" ...
#  ✓ Ensuring node image (kindest/node:v1.27.3) 🖼
#  ✓ Preparing nodes 📦
#  ✓ Writing configuration 📜
#  ✓ Starting control-plane 🕹️
#  ✓ Installing CNI 🔌
#  ✓ Installing StorageClass 💾
# Set kubectl context to "kind-crd-lab"
```

Now set the context to the new cluster:

```bash
kubectl config use-context kind-crd-lab
# Example output: Switched to context "kind-crd-lab".
```

### Smallest Justified Change

Now that we have a cluster, we will add a CustomResourceDefinition. A CRD defines a new kind and its schema. We'll start with a minimal CRD for a fictional `Widget` resource.

Create a file `widget-crd.yaml`:

```yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: widgets.example.com
spec:
  group: example.com
  names:
    kind: Widget
    listKind: WidgetList
    plural: widgets
    singular: widget
  scope: Namespaced
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                size:
                  type: integer
                  minimum: 1
                  maximum: 10
                color:
                  type: string
                  enum: [red, green, blue]
              required: [size, color]
```

This CRD defines a namespaced resource `widgets` in the `example.com` group, with a schema that requires `size` (integer between 1 and 10) and `color` (one of red, green, blue). Apply it:

```bash
kubectl apply -f widget-crd.yaml
# Example output: customresourcedefinition.apiextensions.k8s.io/widgets.example.com created
```

Verify the CRD is established:

```bash
kubectl get crd widgets.example.com
# Example output:
# NAME                    CREATED AT
# widgets.example.com     2023-07-01T12:00:00Z
```

Check its conditions:

```bash
kubectl get crd widgets.example.com -o jsonpath='{.status.conditions[?(@.type=="Established")].status}'
# Example output: True
```

Now the cluster has a new API endpoint. You can list the custom resources (currently none):

```bash
kubectl get widgets
# Example output: No resources found in default namespace.
```

## Safe Configuration Path

In this section, we expand the previous CRD into a more realistic scenario: a namespaced custom resource with validation defaults, additional printer columns, and a simple controller pattern simulated via a shell script. We also cover how to make safe changes and roll back if needed.

### Namespaced CRD with Status Subresource

Add a status subresource and printer columns to make `kubectl get` more informative. Modify `widget-crd.yaml` (or create a new file `widget-crd-v2.yaml`):

```yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: widgets.example.com
spec:
  group: example.com
  names:
    kind: Widget
    listKind: WidgetList
    plural: widgets
    singular: widget
    shortNames:
      - wd
  scope: Namespaced
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                size:
                  type: integer
                  minimum: 1
                  maximum: 10
                color:
                  type: string
                  enum: [red, green, blue]
              required: [size, color]
            status:
              type: object
              properties:
                phase:
                  type: string
                  enum: [Pending, Running, Failed]
                message:
                  type: string
      subresources:
        status: {}
      additionalPrinterColumns:
        - name: Size
          type: integer
          jsonPath: .spec.size
        - name: Color
          type: string
          jsonPath: .spec.color
        - name: Phase
          type: string
          jsonPath: .status.phase
```

Apply the updated CRD:

```bash
kubectl apply -f widget-crd-v2.yaml
# Example output: customresourcedefinition.apiextensions.k8s.io/widgets.example.com configured
```

Note: updating a CRD is a safe change if you only add fields, subresources, or printer columns. Removing fields or changing validation may affect existing custom resources.

Now create a sample Widget:

```bash
cat <<EOF | kubectl apply -f -
apiVersion: example.com/v1
kind: Widget
metadata:
  name: my-widget
  namespace: default
spec:
  size: 5
  color: blue
EOF
# Example output: widget.example.com/my-widget created
```

Check it with custom columns:

```bash
kubectl get widgets
# Example output:
# NAME        SIZE   COLOR   PHASE
# my-widget   5      blue
```

### Simulating a Controller with a Bash Watch Loop

A real controller watches for changes and updates status. For safety, we can simulate a minimal controller using a shell loop that lists all widgets and sets a status if none exists. This is not production-grade, but it demonstrates the pattern.

First, create a small script `simple-controller.sh`:

```bash
#!/usr/bin/env bash
set -euo pipefail

while true; do
  for widget in $(kubectl get widgets -o jsonpath='{.items[*].metadata.name}'); do
    phase=$(kubectl get widget "$widget" -o jsonpath='{.status.phase}')
    if [ -z "$phase" ]; then
      echo "Setting status for $widget to Pending"
      kubectl patch widget "$widget" --type merge -p '{"status":{"phase":"Pending","message":"Processing"}}'
    fi
  done
  sleep 10
done
```

Make it executable and run it in a separate terminal (or background):

```bash
chmod +x simple-controller.sh
./simple-controller.sh
```

After a few seconds, check the widget status:

```bash
kubectl get widget my-widget -o yaml
# Expected output snippet:
# status:
#   message: Processing
#   phase: Pending
```

This demonstrates how a controller can manage status updates safely using `kubectl patch` with merge strategy.

### Rollback and Recovery

If you apply a CRD change that breaks existing resources, you can sometimes revert by applying a previous manifest. Keep versioned manifests in a Git repository. For example, to revert to v1:

```bash
kubectl apply -f widget-crd.yaml
# Example output: customresourcedefinition.apiextensions.k8s.io/widgets.example.com configured
```

If a CRD update invalidates existing custom resources, you may need to edit or delete those resources first. Always test in a local lab before touching production.

## Verification and Diagnostics

Verification is not just about `kubectl apply`; it's about confirming the resource behaves as expected. This section provides concrete checks, common failure signals, and diagnostic commands.

### Functional Verification of the CRD Schema

The schema we defined enforces constraints. Try creating a widget with an invalid color:

```bash
cat <<EOF | kubectl apply -f -
apiVersion: example.com/v1
kind: Widget
metadata:
  name: bad-widget
spec:
  size: 7
  color: purple
EOF
# Expected error:
# The Widget "bad-widget" is invalid: spec.color: Unsupported value: "purple": supported values: "red", "green", "blue"
```

The API server rejects the request before storing it. This confirms schema validation works.

Test size bounds:

```bash
cat <<EOF | kubectl apply -f -
apiVersion: example.com/v1
kind: Widget
metadata:
  name: big-widget
spec:
  size: 11
  color: red
EOF
# Expected error:
# The Widget "big-widget" is invalid: spec.size: Invalid value: 11: spec.size in body should be less than or equal to 10
```

### Diagnostic Commands for CRD Issues

If your CRD isn't establishing, check its status conditions and events.

```bash
kubectl describe crd widgets.example.com
# Look for Events at the bottom, e.g.:
# Events:
#   Type     Reason                 Age   From                            Message
#   Warning  NonStructuralSchema    2m    customresourcedefinition-api   spec.validation.openAPIV3Schema.type: Required value: must not be empty at the root
```

Common failure signals:
- `NonStructuralSchema`: The schema is not structural (e.g., missing `type: object`).
- `NoServedVersions`: The `versions` list is empty or all `served: false`.
- `Terminating`: The CRD is stuck in deletion due to finalizers.

If you see `Terminating`, inspect the finalizers:

```bash
kubectl get crd widgets.example.com -o jsonpath='{.metadata.finalizers}'
# Example output: ["customresourcecleanup.apiextensions.k8s.io"]
```

If you need to force deletion (only in lab, not production), remove the finalizer:

```bash
kubectl patch crd widgets.example.com -p '{"metadata":{"finalizers":[]}}' --type=merge
# Example output: customresourcedefinition.apiextensions.k8s.io/widgets.example.com patched
```

Then check if it disappears.

### Verifying Controller Behavior

If your custom controller isn't picking up changes, check its logs. Since our simple controller runs in a local shell, its output goes to the terminal. In a real pod, you'd use:

```bash
kubectl logs <controller-pod-name>
```

Common controller issues:
- RBAC misconfiguration: The controller doesn't have permission to get/list/watch widgets. Check with `kubectl auth can-i list widgets --as=system:serviceaccount:default:controller-sa`
- CRD not served: `kubectl get widgets` returns error "the server could not find the requested resource". Ensure CRD is applied and `served: true`.
- API deprecation: If using an old API version, ensure `apiextensions.k8s.io/v1` (not v1beta1).

## Failure Modes and Recovery

Operators must plan for failure. This section details common failure modes for CRDs and custom resources, along with recovery steps.

### Failure Mode: CRD Deletion Leaves Custom Resources Orphaned

When you delete a CRD, all custom resources of that type are deleted immediately (garbage collected). If you accidentally delete a CRD with important data, you may lose resources. Recovery requires backups.

Before deleting a CRD, back up all custom resources:

```bash
kubectl get widgets -o yaml > widgets-backup.yaml
```

If you delete the CRD and need to restore, recreate the CRD first, then reapply the backup:

```bash
kubectl delete crd widgets.example.com
# Example output: customresourcedefinition.apiextensions.k8s.io "widgets.example.com" deleted

# Now recreate CRD (assuming your manifest is saved)
kubectl apply -f widget-crd.yaml
# Then restore resources
kubectl apply -f widgets-backup.yaml
```

But note: if the custom resources had status fields, they may not be preserved because backups captured before deletion might lack status. Best practice: use a backup tool like Velero.

### Failure Mode: Schema Too Restrictive Blocks Updates

If a schema prevents legitimate updates because of a new field, you can relax the schema by adding the field as optional. Example: add a `description` field to the schema:

```yaml
spec:
  description:
    type: string
```

Apply the updated CRD, and existing resources are unaffected because the field is optional.

To avoid breaking changes, follow these rules:
- Never remove a field from the schema.
- Never change a field's type.
- Only add new optional fields.

For major changes, introduce a new API version (e.g., v2) and let old resources migrate gradually.

### Failure Mode: Conversion Webhook Misconfiguration

If you use conversion webhooks for multiple versions, a misconfigured webhook can break all API calls for the CRD. To recover, patch the CRD to remove the conversion strategy or fix the webhook service.

Example: check conversion settings:

```bash
kubectl get crd widgets.example.com -o jsonpath='{.spec.conversion}'
# Example output: {"strategy":"Webhook","webhook":{"clientConfig":{"service":{"name":"conversion-webhook","namespace":"default","path":"/convert","port":443}},"conversionReviewVersions":["v1"]}}
```

If the webhook is down, API requests may fail with `conversion webhook not found`. Temporarily change strategy to `None` (if only one version served) to restore access:

```bash
kubectl patch crd widgets.example.com --type merge -p '{"spec":{"conversion":{"strategy":"None"}}}'
```

Note: this requires all versions to be served by the API server without conversion, so ensure you have only one version or that all versions are structurally identical.

### Recovery Verification

After any recovery action, verify by listing and describing resources, checking CRD conditions, and ensuring controllers are running.

```bash
kubectl get crd widgets.example.com -o jsonpath='{.status.conditions}' | jq .
```

## Operations Checklist

Use this checklist before and after any CRD change in your local lab (or as a template for production). Each item includes a concrete example.

| # | Checklist Item | Example Command / Action | Expected Result |
|---|----------------|--------------------------|-----------------|
| 1 | Record current cluster and CRD versions | `kubectl version --short` and `kubectl get crd widgets.example.com -o yaml \| grep -A1 'name: v'` | Example: v1.27.3, CRD version v1 only |
| 2 | Back up existing custom resources | `kubectl get widgets -o yaml > widgets-backup-$(date +%Y%m%d).yaml` | File saved (e.g., widgets-backup-20230701.yaml) |
| 3 | Apply new CRD manifest | `kubectl apply -f widget-crd-v2.yaml` | Output: customresourcedefinition configured |
| 4 | Verify CRD established | `kubectl get crd widgets.example.com -o jsonpath='{.status.conditions[?(@.type=="Established")].status}'` | True |
| 5 | Test schema validation with an invalid object | `echo '{"apiVersion":"example.com/v1","kind":"Widget","metadata":{"name":"test"},"spec":{"size":5,"color":"purple"}}' \| kubectl apply -f -` | Error: Unsupported value: "purple" |
| 6 | Create a valid custom resource | Apply a manifest with size=3, color=green | Output: widget.example.com/test created |
| 7 | Check custom resource fields via `kubectl get` | `kubectl get widgets test` | Shows SIZE=3, COLOR=green, PHASE empty |
| 8 | Simulate controller status update | Run `simple-controller.sh` | Patches widget with phase Pending |
| 9 | Verify status update | `kubectl get widget test -o jsonpath='{.status.phase}'` | Pending |
| 10 | Test rollback | `kubectl apply -f widget-crd.yaml` (old CRD) | CRD configured without status subresource |
| 11 | Clean up resources | `kubectl delete crd widgets.example.com` after backup | CRD deleted, all custom resources deleted |

## Conclusion

Kubernetes Custom Resource Definition local lab setup with practical examples is essential for safe development of custom operators. This article provided a step-by-step guide: environment inventory, minimal CRD, safe configuration with subresources and printer columns, verification commands, and failure recovery scenarios.

To reinforce learning, practice these tasks:
1. Extend the Widget CRD with a new optional field and a validation pattern.
2. Implement a second API version (v2) with a conversion strategy (None or Webhook) and test migration.
3. Deploy a real controller using Kubernetes Deployment and ServiceAccount with proper RBAC.
4. Use `kubectl explain widgets` to explore the generated API documentation.

Remember the operational safety principles: observe before changing, limit blast radius, use placeholders, verify, and document recovery. With a solid local lab, you can iterate faster and reduce production risk.

Next step: choose one low-risk verification from the checklist (e.g., schema validation), record current state, run the check, compare with expected output, and review dependencies like RBAC roles for your controller.

A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision.