## Intro Kubernetes Custom Resource Definitions (CRDs) extend the Kubernetes API, but versioning introduces security risks that are often overlooked. A CRD may have multiple versions (e.g., v1alpha1, v1beta1, v1), and each version may have a different schema, validation rules, and conversion logic. If not properly secured, versioning can lead to privilege escalation, data corruption, or unintended access. This guide provides a practical approach to hardening CRD versioning, with concrete commands, configuration examples, and verification steps. The target audience includes platform engineers, DevOps consultants, and technical startup teams responsible for maintaining Kubernetes clusters. The focus is on operational safety: observe before changing, minimize blast radius, use placeholders instead of secrets, verify every step, and document recovery procedures. Throughout this guide, we use a sample CRD called widgets.example.com to illustrate concepts. We assume a cluster running Kubernetes 1.25+ with kubectl configured and appropriate permissions to create CRDs and access the API server. ## Version and Environment Inventory Before making any changes, inventory the current state of your CRDs and their versions. This includes identifying installed CRDs, their versions, storage status, and conversion strategies. Start by listing all CRDs in the cluster: kubectl get crd Expected output includes a list of CRD names, their created dates. To see details for a specific CRD, use: kubectl get crd widgets.example.com -o yaml This outputs the full specification. Look for the spec.versions field to see all versions defined. For example: spec: versions: - name: v1beta1 served: true storage: false schema: ... - name: v1 served: true storage: true schema: ... Pay attention to: - Which version is the storage version (only one can be storage). - Which versions are served (clients can use them). - Whether a conversion strategy is defined (e.g., None or Webhook ). Check if any conversion webhooks are configured: kubectl get crd widgets.example.com -o jsonpath='{.spec.conversion}' If the output includes strategy: Webhook , then conversion is handled by a webhook. Note the webhook's service and path, as this is a potential security risk if not properly authenticated. Also inspect existing custom resources of that kind to see which versions are in use: kubectl get widgets.example.com --all-namespaces -o wide The -o wide may include the version in the VERSION column. If not, use -o custom-columns to display it: kubectl get widgets.example.com --all-namespaces -o custom-columns=NAME:.metadata.name,NAMESPACE:.metadata.namespace,APIVERSION:.apiVersion Record the current state and timestamps. This inventory helps you understand the impact of any versioning changes. Keep a copy of the CRD definition and a list of resources before making modifications. Practical environment inventory checklist: - kubectl version to confirm client and server versions. - kubectl get crd widgets.example.com -o yaml > crd-backup.yaml to back up the current definition. - kubectl get widgets.example.com -A -o json > widgets-backup.json to back up all instances. - kubectl get apiservice | grep widgets.example.com to see the aggregated API service status. - kubectl describe crd widgets.example.com to see events and status conditions. ## Safe Configuration Path When configuring CRD versioning security, follow the principle of least privilege and validate everything. The safe path involves defining proper schemas, RBAC, and using conversion webhooks securely. ### 1. Define Strict Schemas Each version of your CRD should have a schema that validates the structure and constraints. Use OpenAPI v3 schema validation. A weak schema can allow arbitrary fields, which might be exploited. Example for v1 : schema: openAPIV3Schema: type: object required: ["spec"] properties: spec: type: object required: ["size"] properties: size: type: integer minimum: 1 maximum: 100 replicas: type: integer minimum: 0 maximum: 10 additionalProperties: false additionalProperties: false Set additionalProperties: false to reject unknown fields. This prevents attackers from injecting unexpected data. ### 2. Implement RBAC for CRD Versions RBAC can control access to specific API versions. By default, permissions granted on a resource apply to all versions. To restrict access to certain versions, you can use resourceNames or separate roles. For example, to allow only reading v1 resources but not v1beta1 , create a role: apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: namespace: default name: widget-v1-reader rules: - apiGroups: ["example.com"] resources: ["widgets"] verbs: ["get", "list", "watch"] # No way to specify version directly, but you can use resourceNames or rely on aggregation. Actually, Kubernetes RBAC does not support version-specific authorization directly. The API group is specified, but not the version. To enforce version restrictions, you need to use admission webhooks or separate API groups per version. Alternatively, you can use OPA/Gatekeeper policies. We'll cover admission webhooks later. ### 3. Use Conversion Webhooks Securely If you have multiple versions, conversion webhooks allow Kubernetes to convert resources between versions. The webhook must be served over HTTPS with a valid certificate. Configure the CRD conversion with caBundle to trust the webhook's CA. Example: conversion: strategy: Webhook webhook: conversionReviewVersions: ["v1", "v1beta1"] clientConfig: service: namespace: conversion-webhook name: conversion-service path: /convert caBundle: Always set conversionReviewVersions to the versions your webhook supports. Ensure the webhook service is only accessible within the cluster and uses mutual TLS if possible. ### 4. Apply Changes Gradually When modifying CRD versioning, avoid removing or changing served versions abruptly. Follow a gradual process: - First, add a new version with served: true and storage: false . - Allow clients to migrate, then set storage: true after testing. - Deprecate the old version by setting served: false later. - Finally, remove the old version if no resources use it. To check which resources are still using an old version: kubectl get widgets.example.com --all-namespaces --field-selector apiVersion=example.com/v1beta1 Or use: kubectl get widgets.example.com --all-namespaces -o json | jq '.items[] | select(.apiVersion=="example.com/v1beta1")' ### Example: Safe CRD Update Assume you want to add v2 as the new storage version. Steps: - Backup current CRD: kubectl get crd widgets.example.com -o yaml > crd-backup.yaml - Edit CRD to add v2 with served: true and storage: false : kubectl edit crd widgets.example.com Add under spec.versions : - name: v2 served: true storage: false schema: ... Save and exit. - Verify v2 is served but not storage: kubectl get crd widgets.example.com -o jsonpath='{.spec.versions[*].name} {"\n"}' kubectl get crd widgets.example.com -o jsonpath='{.spec.versions[?(@.storage==true)].name}' Expected first output lists all versions including v2, second output still shows v1. - Test creating a v2 resource: kubectl apply -f - < or if using managed Kubernetes, use cloud provider logging. ### Diagnose Conversion Webhook Failures If conversion webhook is misconfigured, you may see errors like: Check webhook pod logs: - conversion webhook not reachable - x509: certificate signed by unknown authority kubectl logs -n conversion-webhook deploy/conversion-service Test webhook endpoint directly from within cluster: kubectl run -it --rm debug --image=curlimages/curl -- sh # inside pod: curl -k https://conversion-service.conversion-webhook.svc/convert -d '{"apiVersion":"apiextensions.k8s.io/v1","kind":"ConversionReview",...}' Ensure caBundle is correctly base64-encoded. Use base64 -w0 on Linux. ## Failure Modes and Recovery Understand common failure modes and how to recover. ### Failure: CRD with invalid schema causes API server errors If you apply a CRD with a non-structural schema or invalid OpenAPI, the API server may reject it. Example error: The CustomResourceDefinition "widgets.example.com" is invalid: spec.validation.openAPIV3Schema.properties[spec].type: Required value: must not be empty at root Recovery: correct the schema and reapply. If the CRD was previously working and you made a mistake, restore from backup: kubectl apply -f crd-backup.yaml ### Failure: Removing a version still in use If you set served: false for a version that has resources, those resources become inaccessible via the API, though they remain in etcd. You must migrate them before removing serving. To recover, set served: true again and perform migration. ### Failure: Conversion webhook down If the conversion webhook is down, requests that require conversion will fail. Example error: conversion webhook not reachable . Recovery: fix the webhook deployment. If webhook cannot be restored quickly, you can change conversion strategy to None temporarily, but that may cause data inconsistency if multiple versions exist. Better to restore webhook. ### Failure: Storage version change causes data loss Changing storage version without proper conversion can lead to data loss if fields are dropped. Always ensure conversion webhook is in place and tested before switching storage. If data loss occurs, restore from etcd backup if available. ### Recovery Steps - Identify the failing operation and error messages. - Check cluster events: kubectl get events --sort-by='.lastTimestamp' . - Check API server logs for detailed errors (requires access to control plane). - Backup current state: kubectl get crd widgets.example.com -o yaml > crd-current.yaml . - Attempt to revert to previous known good CRD: kubectl apply -f crd-backup.yaml . - If resources are missing or corrupted, restore from etcd snapshot (see etcd disaster recovery). - After recovery, verify with read-only queries before enabling writes. ## Operations Checklist Use this checklist before and after making versioning changes.

#	Check	Command / Action	Expected Result
1	Backup CRD	`kubectl get crd widgets.example.com -o yaml > crd-backup-$(date +%Y%m%d).yaml`	File created with current CRD definition

2	List existing resources by version	kubectl get widgets.example.com -A -o json \| jq -r '.items[] \| .apiVersion' \| sort \| uniq -c	Count of resources per version
3	Check conversion strategy	kubectl get crd widgets.example.com -o jsonpath='{.spec.conversion.strategy}'	None or Webhook
4	Validate schema offline	Use kubectl apply --dry-run=client -f crd-new.yaml	No schema errors
5	Test on staging cluster first	Apply changes to a non-production cluster	No unexpected behavior
6	Monitor audit logs	Tail audit logs for example.com group	Only expected requests
7	Verify each version's serving	kubectl get crd widgets.example.com -o jsonpath='{range .spec.versions[*]}{.name}{" served="}{.served}{" storage="}{.storage}{"\n"}{end}'	Correct served/storage flags
8	Test resource CRUD in all served versions	Script that creates, gets, lists, deletes a test resource in each version	Success without errors
9	Check for deprecated version usage	kubectl get widgets.example.com -A -o json \| jq '.items[] \| select(.apiVersion \| contains("v1beta1"))'	Empty list (no resources using deprecated version)
10	Document rollback procedure	Write runbook for reverting CRD changes	Clear steps for recovery

## Conclusion Securing Kubernetes CRD versioning is essential for maintaining a robust and safe API extension system. By following the practices in this guide—inventorying versions, defining strict schemas, controlling access, using conversion webhooks securely, and verifying changes—you can prevent many common security and operational issues. Start with a low-risk verification: inventory your current CRDs, identify their versions and conversion settings, and run the commands in the checklist. Then, gradually implement improvements, testing each change in a staging environment. Remember to always back up before making changes and have a rollback plan. A reliable workflow makes failures visible, protects sensitive data, limits changes to intended resources, and defines recovery before an incident forces the decision. Apply these principles to harden your Kubernetes CRD versioning today.