Kubernetes Cluster Role Binding Backup and Restore: A Practical Guide

Intro

Kubernetes Role-Based Access Control (RBAC) governs who can do what inside a cluster. ClusterRoleBindings are cluster-scoped grants that attach a ClusterRole to subjects such as users, groups, or ServiceAccounts. Losing or corrupting a ClusterRoleBinding can lock administrators out of critical namespaces or silently remove permissions from applications. Restoring bindings by hand is error-prone and slow. This guide explains how to back up and restore ClusterRoleBindings safely, validate the results, and recover from common failure modes.

You will learn how to inventory existing bindings, export them as YAML, use dry-run modes, apply backups with rollback options, and verify restored permissions. Every step includes concrete commands and expected output snippets so you can follow along in your own cluster.

Before you begin, ensure you have kubectl installed and configured with access to a test cluster. Do not run these procedures against production until you have validated them in a staging environment.

Version and Environment Inventory

Start by confirming your cluster version and the available API resources for RBAC. ClusterRoleBinding is part of the rbac.authorization.k8s.io/v1 API group, available in Kubernetes 1.8 and later. Check your server version:

kubectl version --short

Expected output shows client and server versions. If the server is older than 1.8, you must use the v1beta1 API, but for modern clusters, v1 is the standard.

List all ClusterRoleBindings currently in the cluster:

kubectl get clusterrolebindings

Example output:

NAME                            ROLE                                    AGE
cluster-admin                   cluster-admin                           13d
system:basic-user               system:basic-user                      13d
system:discovery                system:discovery                       13d
system:node                     system:node                            13d
system:node-proxier             system:node-proxier                    13d
my-app-binding                  cluster-admin                          2d

Note the my-app-binding entry: it is a custom binding created by a user. That is the kind of binding you need to back up.

Use kubectl get clusterrolebindings -o wide to see the subjects and role references in a table:

kubectl get clusterrolebindings -o wide

Expected output:

NAME                ROLE                AGE   USERS   GROUPS   SERVICEACCOUNTS
cluster-admin       cluster-admin       13d
my-app-binding      cluster-admin       2d    jane    devs     default/my-app-sa

To inspect a specific binding in detail, use kubectl describe:

kubectl describe clusterrolebinding my-app-binding

This shows the role, subjects, and creation timestamp. For example:

Name:         my-app-binding
Labels:       <none>
Annotations:  <none>
Role:
  Kind:  ClusterRole
  Name:  cluster-admin
Subjects:
  Kind            Name        Namespace
  ----            ----        ---------
  User            jane
  Group           devs
  ServiceAccount  my-app-sa  default

Always capture timestamps and the current state before making changes. Run kubectl get clusterrolebindings my-app-binding -o yaml > my-app-binding-backup-$(date +%Y%m%d).yaml to save the exact manifest. This file becomes your primary backup.

Safe Configuration Path

A safe configuration change follows a pattern: observe, export, modify, dry-run, apply, verify. Never edit a live object directly without a backup. Start by exporting the current binding:

kubectl get clusterrolebinding my-app-binding -o yaml > my-app-binding-original.yaml

Inspect the file. It contains metadata, roleRef, and subjects. A minimal binding looks like:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: my-app-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: User
  name: jane
  apiGroup: rbac.authorization.k8s.io
- kind: Group
  name: devs
  apiGroup: rbac.authorization.k8s.io
- kind: ServiceAccount
  name: my-app-sa
  namespace: default

To safely create a new binding, use kubectl apply --dry-run=client to validate syntax without persisting:

kubectl auth reconcile -f my-app-binding-original.yaml --dry-run=client

Output:

clusterrolebinding.rbac.authorization.k8s.io/my-app-binding created (dry run)

If you need to test a modification, copy the file, change the subject or role, and apply with dry-run. For example, to add a new user bob:

# my-app-binding-modified.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: my-app-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: User
  name: jane
  apiGroup: rbac.authorization.k8s.io
- kind: User
  name: bob
  apiGroup: rbac.authorization.k8s.io
- kind: Group
  name: devs
  apiGroup: rbac.authorization.k8s.io
- kind: ServiceAccount
  name: my-app-sa
  namespace: default

Dry-run it:

kubectl auth reconcile -f my-app-binding-modified.yaml --dry-run=client

Output confirms the change would be adopted. If you see warnings about missing fields, fix them before applying for real.

Use kubectl auth reconcile to ensure the live object matches your desired file. This command applies the differences without removing subjects that are not in the file unless you use --remove-extra-subjects. For example, to keep the binding exactly as specified in my-app-binding-original.yaml:

kubectl auth reconcile -f my-app-binding-original.yaml

Output:

clusterrolebinding.rbac.authorization.k8s.io/my-app-binding reconciled

If you want to remove any subjects not in the file, add --remove-extra-subjects:

kubectl auth reconcile -f my-app-binding-original.yaml --remove-extra-subjects

This is useful when restoring from a backup and you want to reset the binding to a known state.

Backup Procedures

Backing up ClusterRoleBindings involves exporting their manifests. You can back up all bindings in one command by iterating over the list.

First, list all binding names:

kubectl get clusterrolebindings -o name

Output:

clusterrolebinding.rbac.authorization.k8s.io/cluster-admin
clusterrolebinding.rbac.authorization.k8s.io/my-app-binding
...

Then, loop through and save each to a timestamped directory:

mkdir -p clusterrolebinding-backup-$(date +%Y%m%d)
for binding in $(kubectl get clusterrolebindings -o name); do
  name=$(echo $binding | cut -d'/' -f2)
  kubectl get clusterrolebinding $name -o yaml > clusterrolebinding-backup-$(date +%Y%m%d)/$name.yaml
done

This creates individual YAML files for each binding. Alternatively, you can save all bindings in a single file using kubectl get clusterrolebindings -o yaml > all-bindings.yaml. However, individual files are easier to restore selectively.

For automated backup schedules, consider integrating this command into a cron job or a Kubernetes management tool. But manual exports are sufficient for most small to medium clusters.

Check the backup files for completeness:

ls -l clusterrolebinding-backup-20250315/

Expected output lists each YAML file. Verify one file contains the correct API version and kind:

head -5 clusterrolebinding-backup-20250315/my-app-binding.yaml

Output:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: my-app-binding

Store backups in a secure location, such as an encrypted object store or a Git repository with access controls. Avoid keeping them on local disks only.

Restore Procedures

Restoring a binding from a backup file is straightforward with kubectl apply, but safety checks prevent accidental overwrites.

If the binding was deleted, apply the file to recreate it:

kubectl apply -f clusterrolebinding-backup-20250315/my-app-binding.yaml

Output:

clusterrolebinding.rbac.authorization.k8s.io/my-app-binding created

If the binding exists but has been modified, kubectl apply will update it to match the file. However, be cautious: apply may not remove subjects that were added later. To restore the exact original state including removal of extra subjects, use kubectl auth reconcile:

kubectl auth reconcile -f clusterrolebinding-backup-20250315/my-app-binding.yaml --remove-extra-subjects

Output:

clusterrolebinding.rbac.authorization.k8s.io/my-app-binding reconciled

After restoring, verify the binding details match the backup:

kubectl get clusterrolebinding my-app-binding -o yaml

Compare the output with the backup file. Look for differences in roleRef and subjects.

For a bulk restore, loop over the backup directory:

for file in clusterrolebinding-backup-20250315/*.yaml; do
  kubectl apply -f $file
done

Be aware that this will recreate or update all bindings, including system ones. It is usually safe because system bindings are recreated by Kubernetes if deleted, but exercise caution in production.

Verification and Diagnostics

After backup or restore, verify that permissions work as expected. The primary command is kubectl auth can-i to test access from a subject's perspective.

To check if user jane can list pods in the default namespace:

kubectl auth can-i list pods --as jane

Expected output:

yes

For a ServiceAccount, use --as=system:serviceaccount:default:my-app-sa:

kubectl auth can-i list pods --as=system:serviceaccount:default:my-app-sa -n default

If the binding grants cluster-admin, the output is yes for any verb and resource. If the binding is misconfigured, the output is no.

To diagnose binding issues, inspect the binding's YAML and the role's permissions. For example, if jane cannot list pods, check if the binding still references the correct role:

kubectl get clusterrolebinding my-app-binding -o jsonpath='{.roleRef.name}'

Expected output:

cluster-admin

If the role name is empty or wrong, fix the binding. Also, check that the subject is correctly listed:

kubectl get clusterrolebinding my-app-binding -o jsonpath='{.subjects[*].name}'

Expected output:

jane devs my-app-sa

Use kubectl describe clusterrolebinding my-app-binding for a human-readable summary of the same information.

For cluster-wide diagnostics, list all bindings and see if any are missing or have unusual roles:

kubectl get clusterrolebindings -o custom-columns=NAME:.metadata.name,ROLE:.roleRef.name,SUBJECTS:.subjects[*].name

Example output:

NAME                ROLE                SUBJECTS
cluster-admin       cluster-admin       jane, devs, my-app-sa
...

If a binding is missing, that explains access denials. Restore it from backup as described.

Failure Modes and Recovery

Common failure modes include accidental deletion, overwriting with wrong subjects, and API deprecation issues.

Accidental Deletion

If a ClusterRoleBinding is deleted, users lose access immediately. To recover, you need a backup. If no backup exists, you can recreate the binding from documentation or from a similar cluster. For example, to restore the my-app-binding if deleted:

kubectl apply -f clusterrolebinding-backup-20250315/my-app-binding.yaml

If you do not have a backup, you must manually reconstruct the binding based on your team's records. This is why regular backups are critical.

Overwriting with Wrong Subjects

If someone applies a new binding file that removes necessary subjects, you can restore from your backup. Suppose the current binding only has user bob but should also have jane and the ServiceAccount. Restore with kubectl auth reconcile as above.

API Version Deprecation

On older clusters, you might see errors like:

error: unable to recognize "my-app-binding.yaml": no matches for kind "ClusterRoleBinding" in version "rbac.authorization.k8s.io/v1"

This indicates your client is trying to use v1 but the server only supports v1beta1. Check supported versions:

kubectl api-versions | grep rbac

Output might include rbac.authorization.k8s.io/v1 and rbac.authorization.k8s.io/v1beta1. If only v1beta1 is available, convert your manifest by changing apiVersion to rbac.authorization.k8s.io/v1beta1, or upgrade the cluster.

Loss of Cluster Admin Access

If you accidentally remove all cluster-admin bindings, you may lose administrative access. Recovery requires a user with direct access to the cluster's API server or a break-glass mechanism. Some clusters configure static tokens or a super-admin user in the kubeconfig that bypasses RBAC. Otherwise, you may need to restore from etcd backup or contact your cloud provider. This scenario underscores the importance of having multiple admin users and off-cluster backups.

Rollback Strategies

Always keep a history of changes. Use kubectl rollout undo for Deployments, but for RBAC objects there is no built-in rollback. Instead, maintain multiple versions of binding files in Git and apply the desired version to roll back. For example, if a recent change broke access, check out the previous commit and run:

git checkout HEAD~1 -- clusterrolebinding-backup-20250315/my-app-binding.yaml
kubectl auth reconcile -f clusterrolebinding-backup-20250315/my-app-binding.yaml --remove-extra-subjects

This sets the binding back to the prior state.

Operations Checklist

Use this checklist before and after any backup or restore operation on ClusterRoleBindings.

Run this checklist in a staging cluster first, then adapt for production. Always have a rollback plan before applying changes.

Conclusion

Backing up and restoring Kubernetes ClusterRoleBindings is a foundational practice for RBAC management. With the commands and procedures in this guide, you can protect your cluster against accidental permission loss and recover quickly. Remember to export bindings regularly, store backups safely, test restores with dry-runs, and verify access after every change. Include ClusterRoleBindings in your disaster recovery plan alongside workloads and configuration. By following these practices, you ensure that your cluster's access controls remain reliable and recoverable.