Kubernetes Custom Resource Definition Versioning Security Hardening: A Practical Guide

Intro

Kubernetes Custom Resource Definitions (CRDs) extend the Kubernetes API, but versioning introduces security risks that are often overlooked. A CRD may have multiple versions (e.g., v1alpha1, v1beta1, v1), and each version may have a different schema, validation rules, and conversion logic. If not properly secured, versioning can lead to privilege escalation, data corruption, or unintended access. This guide provides a practical approach to hardening CRD versioning, with concrete commands, configuration examples, and verification steps.

The target audience includes platform engineers, DevOps consultants, and technical startup teams responsible for maintaining Kubernetes clusters. The focus is on operational safety: observe before changing, minimize blast radius, use placeholders instead of secrets, verify every step, and document recovery procedures.

Throughout this guide, we use a sample CRD called widgets.example.com to illustrate concepts. We assume a cluster running Kubernetes 1.25+ with kubectl configured and appropriate permissions to create CRDs and access the API server.

Version and Environment Inventory

Before making any changes, inventory the current state of your CRDs and their versions. This includes identifying installed CRDs, their versions, storage status, and conversion strategies.

Start by listing all CRDs in the cluster:

kubectl get crd

Expected output includes a list of CRD names, their created dates. To see details for a specific CRD, use:

kubectl get crd widgets.example.com -o yaml

This outputs the full specification. Look for the spec.versions field to see all versions defined. For example:

spec:
  versions:
    - name: v1beta1
      served: true
      storage: false
      schema: ...
    - name: v1
      served: true
      storage: true
      schema: ...

Pay attention to:

• Which version is the storage version (only one can be storage).
• Which versions are served (clients can use them).
• Whether a conversion strategy is defined (e.g., None or Webhook).

Check if any conversion webhooks are configured:

kubectl get crd widgets.example.com -o jsonpath='{.spec.conversion}'

If the output includes strategy: Webhook, then conversion is handled by a webhook. Note the webhook's service and path, as this is a potential security risk if not properly authenticated.

Also inspect existing custom resources of that kind to see which versions are in use:

kubectl get widgets.example.com --all-namespaces -o wide

The -o wide may include the version in the VERSION column. If not, use -o custom-columns to display it:

kubectl get widgets.example.com --all-namespaces -o custom-columns=NAME:.metadata.name,NAMESPACE:.metadata.namespace,APIVERSION:.apiVersion

Record the current state and timestamps. This inventory helps you understand the impact of any versioning changes. Keep a copy of the CRD definition and a list of resources before making modifications.

Practical environment inventory checklist:

• kubectl version to confirm client and server versions.
• kubectl get crd widgets.example.com -o yaml > crd-backup.yaml to back up the current definition.
• kubectl get widgets.example.com -A -o json > widgets-backup.json to back up all instances.
• kubectl get apiservice | grep widgets.example.com to see the aggregated API service status.
• kubectl describe crd widgets.example.com to see events and status conditions.

Safe Configuration Path

When configuring CRD versioning security, follow the principle of least privilege and validate everything. The safe path involves defining proper schemas, RBAC, and using conversion webhooks securely.

1. Define Strict Schemas

Each version of your CRD should have a schema that validates the structure and constraints. Use OpenAPI v3 schema validation. A weak schema can allow arbitrary fields, which might be exploited. Example for v1:

schema:
  openAPIV3Schema:
    type: object
    required: ["spec"]
    properties:
      spec:
        type: object
        required: ["size"]
        properties:
          size:
            type: integer
            minimum: 1
            maximum: 100
          replicas:
            type: integer
            minimum: 0
            maximum: 10
        additionalProperties: false
    additionalProperties: false

Set additionalProperties: false to reject unknown fields. This prevents attackers from injecting unexpected data.

2. Implement RBAC for CRD Versions

RBAC can control access to specific API versions. By default, permissions granted on a resource apply to all versions. To restrict access to certain versions, you can use resourceNames or separate roles. For example, to allow only reading v1 resources but not v1beta1, create a role:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: widget-v1-reader
rules:
- apiGroups: ["example.com"]
  resources: ["widgets"]
  verbs: ["get", "list", "watch"]
  # No way to specify version directly, but you can use resourceNames or rely on aggregation.

Actually, Kubernetes RBAC does not support version-specific authorization directly. The API group is specified, but not the version. To enforce version restrictions, you need to use admission webhooks or separate API groups per version. Alternatively, you can use OPA/Gatekeeper policies. We'll cover admission webhooks later.

3. Use Conversion Webhooks Securely

If you have multiple versions, conversion webhooks allow Kubernetes to convert resources between versions. The webhook must be served over HTTPS with a valid certificate. Configure the CRD conversion with caBundle to trust the webhook's CA. Example:

conversion:
  strategy: Webhook
  webhook:
    conversionReviewVersions: ["v1", "v1beta1"]
    clientConfig:
      service:
        namespace: conversion-webhook
        name: conversion-service
        path: /convert
      caBundle: <base64-encoded CA certificate>

Always set conversionReviewVersions to the versions your webhook supports. Ensure the webhook service is only accessible within the cluster and uses mutual TLS if possible.

4. Apply Changes Gradually

When modifying CRD versioning, avoid removing or changing served versions abruptly. Follow a gradual process:

• First, add a new version with served: true and storage: false.
• Allow clients to migrate, then set storage: true after testing.
• Deprecate the old version by setting served: false later.
• Finally, remove the old version if no resources use it.

To check which resources are still using an old version:

kubectl get widgets.example.com --all-namespaces --field-selector apiVersion=example.com/v1beta1

Or use:

kubectl get widgets.example.com --all-namespaces -o json | jq '.items[] | select(.apiVersion=="example.com/v1beta1")'

Example: Safe CRD Update

Assume you want to add v2 as the new storage version. Steps:

1. Backup current CRD:

kubectl get crd widgets.example.com -o yaml > crd-backup.yaml

1. Edit CRD to add v2 with served: true and storage: false:

kubectl edit crd widgets.example.com

Add under spec.versions:

- name: v2
  served: true
  storage: false
  schema: ...

Save and exit.

1. Verify v2 is served but not storage:

kubectl get crd widgets.example.com -o jsonpath='{.spec.versions[*].name} {"\n"}'
kubectl get crd widgets.example.com -o jsonpath='{.spec.versions[?(@.storage==true)].name}'

Expected first output lists all versions including v2, second output still shows v1.

1. Test creating a v2 resource:

kubectl apply -f - <<EOF
apiVersion: example.com/v2
kind: Widget
metadata:
  name: test-widget-v2
spec:
  size: 10
EOF

1. After successful test, switch storage to v2:

kubectl patch crd widgets.example.com --type='json' -p='[{"op": "replace", "path": "/spec/versions/1/storage", "value": true}, {"op": "replace", "path": "/spec/versions/0/storage", "value": false}]'

(Assuming versions array order: index 0 = v1, index 1 = v2). Adjust paths accordingly.

1. Verify storage version changed:

kubectl get crd widgets.example.com -o jsonpath='{.spec.versions[?(@.storage==true)].name}'

Output: v2.

Verification and Diagnostics

After making changes, verify that everything works as expected and diagnose any issues.

Check CRD Status

Use kubectl describe crd to see status conditions:

kubectl describe crd widgets.example.com

Look for Established condition true and no NonStructuralSchema warnings. If the schema is not structural, Kubernetes may reject the CRD or fail to serve it.

Test Resource Operations

Attempt to create, get, and list resources in each version:

# Create v1 resource
kubectl apply -f - <<EOF
apiVersion: example.com/v1
kind: Widget
metadata:
  name: test-v1
spec:
  size: 5
EOF

# Get via v1
kubectl get widgets.v1.example.com test-v1 -o yaml

# Get via v2 (conversion should happen if webhook configured)
kubectl get widgets.v2.example.com test-v1 -o yaml

If conversion is not set, the v2 get will fail because storage version is v2 but the object was created with v1? Actually, if storage is v2 and you create a v1 resource, Kubernetes will convert it to v2 for storage if a conversion webhook exists, or if no conversion, it will reject. Verify expected behavior.

Check Audit Logs

Enable Kubernetes audit logging to monitor access to CRD versions. Look for requests to /apis/example.com/v1beta1/widgets if you are trying to detect usage of deprecated versions. Example audit policy snippet:

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: Metadata
  resources:
  - group: "example.com"
    resources: ["widgets"]

Then check audit logs with tools like kubectl logs -n kube-system kube-apiserver-<node> or if using managed Kubernetes, use cloud provider logging.

Diagnose Conversion Webhook Failures

If conversion webhook is misconfigured, you may see errors like:

Check webhook pod logs:

• conversion webhook not reachable
• x509: certificate signed by unknown authority

kubectl logs -n conversion-webhook deploy/conversion-service

Test webhook endpoint directly from within cluster:

kubectl run -it --rm debug --image=curlimages/curl -- sh
# inside pod:
curl -k https://conversion-service.conversion-webhook.svc/convert -d '{"apiVersion":"apiextensions.k8s.io/v1","kind":"ConversionReview",...}'

Ensure caBundle is correctly base64-encoded. Use base64 -w0 on Linux.

Failure Modes and Recovery

Understand common failure modes and how to recover.

Failure: CRD with invalid schema causes API server errors

If you apply a CRD with a non-structural schema or invalid OpenAPI, the API server may reject it. Example error:

The CustomResourceDefinition "widgets.example.com" is invalid: spec.validation.openAPIV3Schema.properties[spec].type: Required value: must not be empty at root

Recovery: correct the schema and reapply. If the CRD was previously working and you made a mistake, restore from backup:

kubectl apply -f crd-backup.yaml

Failure: Removing a version still in use

If you set served: false for a version that has resources, those resources become inaccessible via the API, though they remain in etcd. You must migrate them before removing serving. To recover, set served: true again and perform migration.

Failure: Conversion webhook down

If the conversion webhook is down, requests that require conversion will fail. Example error: conversion webhook not reachable. Recovery: fix the webhook deployment. If webhook cannot be restored quickly, you can change conversion strategy to None temporarily, but that may cause data inconsistency if multiple versions exist. Better to restore webhook.

Failure: Storage version change causes data loss

Changing storage version without proper conversion can lead to data loss if fields are dropped. Always ensure conversion webhook is in place and tested before switching storage. If data loss occurs, restore from etcd backup if available.

Recovery Steps

1. Identify the failing operation and error messages.
2. Check cluster events: kubectl get events --sort-by='.lastTimestamp'.
3. Check API server logs for detailed errors (requires access to control plane).
4. Backup current state: kubectl get crd widgets.example.com -o yaml > crd-current.yaml.
5. Attempt to revert to previous known good CRD: kubectl apply -f crd-backup.yaml.
6. If resources are missing or corrupted, restore from etcd snapshot (see etcd disaster recovery).
7. After recovery, verify with read-only queries before enabling writes.

Operations Checklist

Use this checklist before and after making versioning changes.

Conclusion

Securing Kubernetes CRD versioning is essential for maintaining a robust and safe API extension system. By following the practices in this guide—inventorying versions, defining strict schemas, controlling access, using conversion webhooks securely, and verifying changes—you can prevent many common security and operational issues.

Start with a low-risk verification: inventory your current CRDs, identify their versions and conversion settings, and run the commands in the checklist. Then, gradually implement improvements, testing each change in a staging environment. Remember to always back up before making changes and have a rollback plan.

A reliable workflow makes failures visible, protects sensitive data, limits changes to intended resources, and defines recovery before an incident forces the decision. Apply these principles to harden your Kubernetes CRD versioning today.