## Intro Kubernetes Dynamic Resource Allocation (DRA) is a relatively new feature, introduced as alpha in 1.26 and moving toward beta, that generalizes how Pods request and access specialized hardware such as GPUs, FPGAs, and network adapters. Unlike traditional device plugins, DRA allows cluster administrators to define resource classes and let users request resources by name, with the scheduler and a resource driver handling the allocation lifecycle. This flexibility is powerful, but it also makes CI/CD automation more complex: resource availability can vary by node, allocation is asynchronous, and drivers may need installation and configuration. This article provides a practical guide to building CI/CD pipelines for workloads that use Kubernetes Dynamic Resource Allocation. We will cover version and environment discovery, safe configuration approaches, verification and diagnostics, failure modes and recovery, and an operations checklist. Every section includes concrete commands, expected outputs, and decision points. By the end, you will be able to create automated pipelines that deploy DRA-dependent applications safely, with clear rollback paths when allocations fail. The target audience includes DevOps engineers, platform teams, and technical leads who need to manage stateful, hardware-accelerated workloads in Kubernetes. We assume familiarity with basic Kubernetes concepts, kubectl, and CI/CD tools like GitHub Actions or GitLab CI. ## Version and Environment Inventory Before automating any DRA interaction, confirm that your cluster supports the feature and that the required components are installed. DRA relies on two main parts: - The Kubernetes control plane with the `DynamicResourceAllocation` feature gate enabled (for alpha in 1.26, beta in 1.27+, and stable in later releases). - A resource driver that implements the DRA API. Examples include the NVIDIA GPU DRA driver, the Intel Device Plugins DRA driver, or custom drivers for your hardware. Run the following read-only checks to inventory your environment: ```bash kubectl version --short kubectl get nodes -o wide kubectl get pods -n kube-system | grep -E 'dra|resource-driver' kubectl get resourceclasses.resource.k8s.io kubectl get resourceslices.resource.k8s.io ``` Expected outputs: - `kubectl version` should show server version 1.26 or later. - `kubectl get nodes` should list all worker nodes with their status. - The `kubectl get pods -n kube-system` command will list the DRA driver Pods; if none appear, the driver is not installed. - `resourceclasses` and `resourceslices` are custom resources; if the APIs are unavailable, the feature gate is off or the CRDs are missing. Example from a healthy cluster: ``` $ kubectl get resourceclasses.resource.k8s.io NAME AGE gpu.nvidia.com 5d ``` If you see `error: the server doesn't have a resource type "resourceclasses"`, enable the feature gate on the API server and controller manager, or upgrade to a version where DRA is beta and enabled by default. **Compatibility notes:** - DRA is alpha in 1.26, so it requires the feature gate `DynamicResourceAllocation=true` on kube-apiserver, kube-controller-manager, and kube-scheduler. - In 1.27 and 1.28, DRA is beta and enabled by default, but the API may still be under the `resource.k8s.io/v1alpha2` group. - Check your driver's documentation for compatibility; some drivers may only work with specific Kubernetes versions. **Security consideration:** Use a service account with read-only permissions for these checks in CI. For example, create a `dra-reader` ClusterRole with `get`, `list`, and `watch` on resourceclasses, resourceslices, pods, and nodes. Once the environment is verified, capture it in a pipeline stage. For GitHub Actions, a simple inventory job could be: ```yaml name: Environment Inventory on: [workflow_dispatch] jobs: inventory: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Set up kubectl uses: azure/setup-kubectl@v3 with: version: 'v1.28.0' - name: Check DRA resources run: | kubectl version --short kubectl get resourceclasses.resource.k8s.io kubectl get resourceslices.resource.k8s.io ``` This stage runs before any deployment and fails if the DRA resources are missing, preventing later failures. ## Safe Configuration Path With DRA, configuration must be versioned and applied incrementally. The key objects are: - `ResourceClass`: defines a class of resource (e.g., `gpu.nvidia.com`) and references a driver. - `ResourceSlice`: published by the driver to advertise available resources on nodes. - `ResourceClaim`: a request for a resource from a specific class, created by a user or Pod. - Pod spec using the claim: the Pod references the ResourceClaim to get the allocated resource. A safe configuration approach is to use a GitOps model where all DRA manifests are in a repository and applied through a pipeline with review steps. Here is an example of a minimal DRA configuration: **ResourceClass:** ```yaml apiVersion: resource.k8s.io/v1alpha2 kind: ResourceClass metadata: name: gpu.nvidia.com driverName: dra.nvidia.com ``` **ResourceClaim (template):** ```yaml apiVersion: resource.k8s.io/v1alpha2 kind: ResourceClaimTemplate metadata: name: gpu-claim-template spec: metadata: labels: app: myapp spec: resourceClassName: gpu.nvidia.com ``` **Pod using the claim template:** ```yaml apiVersion: v1 kind: Pod metadata: name: myapp-with-gpu spec: containers: - name: app image: myapp:latest resources: claims: - name: gpu resourceClaims: - name: gpu source: resourceClaimTemplateName: gpu-claim-template ``` When you apply these manifests, the controller creates a ResourceClaim and the driver allocates a suitable resource on a node. The Pod will only be scheduled after the allocation succeeds. **Smallest justified change:** Always apply the ResourceClass and driver first, test with a static ResourceClaim, then move to templates. Avoid changing driver configurations without a full canary deployment. **Pipeline example with approval stage:** In GitHub Actions, you can use environments with required reviewers to gate the apply step: ```yaml jobs: deploy-staging: runs-on: ubuntu-latest environment: staging steps: - name: Apply DRA configuration run: | kubectl apply -f dra/ --dry-run=client kubectl apply -f dra/ kubectl get resourceclaims ``` The `dry-run=client` validates syntax before applying. After applying, check the claim status: ```bash kubectl get resourceclaims NAME STATE myapp-gpu-claim allocated ``` If the state remains `pending`, check driver logs and node capacity. ## Verification and Diagnostics Verification goes beyond checking Pod status. Since DRA involves asynchronous allocation, you must verify the entire chain: claim created, resource allocated, driver bound, and container started with the resource visible. **Step-by-step verification commands:** 1. Check Pod events: ```bash kubectl describe pod myapp-with-gpu ``` Look for events like: ``` Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 10s default-scheduler Successfully assigned default/myapp-with-gpu to node1 Normal Pulled 9s kubelet Container image "myapp:latest" already present on machine Normal Created 9s kubelet Created container app Normal Started 9s kubelet Started container app ``` 2. Inspect the ResourceClaim: ```bash kubectl describe resourceclaim myapp-gpu-claim ``` Output should show `Allocation: allocated` and the name of the ResourceSlice. 3. Verify resource inside the container: ```bash kubectl exec myapp-with-gpu -- nvidia-smi ``` Expected output includes GPU information, e.g., `NVIDIA GeForce RTX 3090`. 4. Check driver logs if allocation failed: ```bash kubectl logs -n kube-system -l app=nvidia-dra-driver --tail=50 ``` **Common failure signals:** - Pod stays in `Pending` with event `FailedScheduling: no nodes available for resource claim`. - Claim status `pending` with no allocation after several minutes. - Driver logs show errors like `insufficient resources` or `driver not ready`. **Diagnostic script for CI:** Add a verification job that runs after deployment and fails if any check fails: ```yaml - name: Verify DRA allocation run: | # Wait for Pod to be Running kubectl wait --for=condition=Ready pod/myapp-with-gpu --timeout=300s # Check claim status CLAIM_STATE=$(kubectl get resourceclaim myapp-gpu-claim -o jsonpath='{.status.allocation.result}') if [ "$CLAIM_STATE" != "allocated" ]; then echo "Claim not allocated" exit 1 fi # Verify resource in container kubectl exec myapp-with-gpu -- nvidia-smi | grep -q 'NVIDIA' ``` This script uses `kubectl wait` to avoid race conditions and then confirms allocation and device visibility. ## Failure Modes and Recovery DRA introduces new failure modes beyond typical scheduling issues. Understanding these helps design effective rollback strategies. **Common failure modes:** 1. **Driver not installed or misconfigured**: ResourceClasses exist but no ResourceSlices are published. - Symptom: `kubectl get resourceslices.resource.k8s.io` returns no resources. - Recovery: Install/upgrade the driver, ensure it runs on all nodes, and check its RBAC. 2. **Resource exhaustion**: All resources of a class are allocated to other claims. - Symptom: New claims stay `pending`; ResourceSlices show `allocated` equals `capacity`. - Recovery: Release unused claims manually: ```bash kubectl delete resourceclaim myapp-gpu-claim ``` Then redeploy. Or scale down other workloads that hold claims. 3. **Node failure**: The node holding the allocated resource goes down. - Symptom: Pod is evicted or stuck in `Terminating`; claim may be in `reserved` state. - Recovery: Delete the Pod and claim; the scheduler will allocate a new resource on a healthy node if available. 4. **Driver version mismatch**: Driver does not support the current Kubernetes DRA API version. - Symptom: Controller logs show API errors; claims never get allocated. - Recovery: Upgrade driver to a compatible version; check driver release notes for Kubernetes version support. **Rollback strategy:** In CI/CD, always test rollback before production. A simple rollback procedure for a DRA-dependent Deployment: ```bash # Rollback Deployment kubectl rollout undo deployment/myapp # If that fails, delete the Pod and claim kubectl delete pod myapp-with-gpu kubectl delete resourceclaim myapp-gpu-claim # Redeploy previous version kubectl apply -f previous-manifests/ ``` For automated rollback in GitHub Actions, use a job that runs on failure: ```yaml on: workflow_run: workflows: ["Deploy"] types: - completed jobs: rollback-on-failure: if: ${{ github.event.workflow_run.conclusion == 'failure' }} runs-on: ubuntu-latest steps: - run: | kubectl rollout undo deployment/myapp kubectl delete resourceclaim myapp-gpu-claim ``` **Recovery verification:** After rollback, ensure the previous version runs correctly: ```bash kubectl rollout status deployment/myapp kubectl get pods -l app=myapp ``` ## Operations Checklist Use this checklist before and after any CI/CD pipeline run that involves DRA. It is designed to be pasted into an issue or runbook. **Pre-deployment checklist:** 1. Kubernetes version is 1.26+ (for alpha) or 1.27+ (for beta); feature gate is enabled if required. - Command: `kubectl version --short` 2. DRA driver is deployed and healthy. - Command: `kubectl get pods -n kube-system | grep dra` (all Running) 3. ResourceClass and ResourceSlice objects exist. - Command: `kubectl get resourceclasses.resource.k8s.io`, `kubectl get resourceslices.resource.k8s.io` 4. Available capacity is sufficient for the new workload. - Command: `kubectl describe resourceslice | grep -A5 'Allocated resources'` 5. ResourceClaim manifests are validated. - Command: `kubectl apply --dry-run=client -f manifests/` 6. Rollback plan is documented and tested in staging. - Example: `kubectl rollout undo deployment/myapp` **During deployment:** - Monitor claim status: `watch kubectl get resourceclaims -l app=myapp` - Watch Pod events: `kubectl describe pod myapp-with-gpu` - If claim stays pending for more than 5 minutes, check driver logs. **Post-deployment verification:** - Pod is Running and Ready. - ResourceClaim state is `allocated`. - Device is visible inside container (e.g., `nvidia-smi` output). - Application performance is acceptable (no excessive waits on resource). **Example of a filled checklist for a specific deployment:** | Item | Check | Expected | Owner | |---|---|---|---| | Kubernetes version | `kubectl version --short` | Server v1.28.0 | Priya Shah, Engineering Lead | | Driver health | `kubectl get pods -n kube-system -l app=nvidia-dra-driver` | 3/3 Running | DevOps Bot | | ResourceClass exists | `kubectl get resourceclass gpu.nvidia.com` | Age < 24h | Platform Team | | Capacity | `kubectl describe resourceslice node1-gpu-slice` | Allocated 2/4 GPUs | SRE | | Dry-run | `kubectl apply --dry-run=client -f manifests/` | No errors | CI pipeline | | Rollback plan | `kubectl rollout undo deployment/myapp` | Tested in staging | Release Manager | ## Conclusion Kubernetes Dynamic Resource Allocation unlocks flexible hardware sharing, but it demands careful automation. By inventorying your environment, applying changes incrementally, verifying allocations with concrete commands, and preparing for failure modes, you can build reliable CI/CD pipelines for GPU-accelerated and other specialized workloads. The key takeaways are: - Always check version compatibility and driver health before deploying anything that uses DRA. - Use read-only checks and dry-runs first; never apply without validation. - Monitor the entire allocation chain: ResourceClaim, ResourceSlice, Pod scheduling. - Have a clear rollback path and test it in staging. - Automate verification steps to catch failures early. Start by implementing the Environment Inventory stage in your pipeline. Then add the safe apply with dry-run and approval. Finally, integrate the verification and rollback jobs. With these in place, you can confidently manage DRA in production.