## Intro Kubernetes Role-Based Access Control (RBAC) is the primary mechanism for regulating access to cluster resources. While Roles and RoleBindings scope permissions to a single namespace, ClusterRoles and ClusterRoleBindings grant permissions across the entire cluster. A single overly permissive ClusterRoleBinding can expose sensitive resources, enable privilege escalation, or lead to full cluster compromise. This guide focuses on practical steps to harden ClusterRoleBindings. You will learn how to assess existing bindings, apply least-privilege principles, verify changes, and recover from mistakes. Every section includes concrete commands and configuration examples that you can run in your own environment, whether it is a local development cluster, a managed cloud offering like EKS or GKE, or an on-premises production system. We assume you have basic familiarity with Kubernetes concepts such as pods, namespaces, and service accounts. If you are new to RBAC, the official Kubernetes documentation provides a good foundation. This article goes beyond the basics to show real-world hardening techniques that are often missed. ## Version and Environment Inventory Before making any changes, you need a clear picture of your cluster and the tools available. This section lists prerequisites and shows how to gather version information and current RBAC state. ### Prerequisites - A running Kubernetes cluster (version 1.19 or later recommended; RBAC is stable in v1 and the `rbac.authorization.k8s.io/v1` API is available since 1.8). - `kubectl` command-line tool configured with cluster-admin or sufficient privileges to view and modify RBAC objects. - `jq` (optional but strongly recommended for processing JSON output). - Access to audit logs, if available, for diagnosis. ### Check Kubernetes and kubectl Versions Run the following command to see both client and server versions: ```bash kubectl version --short ``` Example output: ``` Client Version: v1.25.3 Server Version: v1.25.3 ``` If your cluster version is older than 1.19, some API fields may differ. The examples in this guide use `rbac.authorization.k8s.io/v1`, which is stable since Kubernetes 1.8. ### List Existing ClusterRoleBindings To see all cluster-wide role bindings: ```bash kubectl get clusterrolebindings ``` This lists every binding, including default ones like `cluster-admin` and various `system:*` bindings. Take note of any that are not part of the default set. Custom bindings created by users or third-party tools often have names like `my-app-binding` or `developer-access`. ### Check API Resources for RBAC Confirm that the RBAC API group is available: ```bash kubectl api-resources | grep rbac ``` Expected output includes `clusterrolebindings`, `clusterroles`, `roles`, and `rolebindings` under the `rbac.authorization.k8s.io` group. ### Snapshot the Current State Before modifying anything, create a backup of all ClusterRoleBindings. This is crucial for recovery. ```bash kubectl get clusterrolebindings -o yaml > clusterrolebindings-backup-$(date +%Y%m%d).yaml ``` Store this file in a version-controlled repository or secure location. You will thank yourself later if something goes wrong. ## Safe Configuration Path This section provides a step-by-step approach to hardening ClusterRoleBindings. We start by reviewing existing bindings, then apply least-privilege modifications, and finally verify the results. ### Step 1: Identify Overly Permissive Bindings The built-in `cluster-admin` ClusterRoleBinding grants superuser rights to any bound subject. Often it is bound to the default service account in the `kube-system` namespace or to a generic user. Check who has this binding: ```bash kubectl get clusterrolebinding cluster-admin -o yaml ``` Examine the `subjects` field. For example, if you see a service account from the `default` namespace, that is a red flag. The output might include: ```yaml subjects: - kind: ServiceAccount name: default namespace: default ``` This means any pod running with the default service account in the default namespace can perform any action in the cluster. That is almost certainly not intended. Also list all bindings that reference the `cluster-admin` ClusterRole: ```bash kubectl get clusterrolebindings -o json | jq '.items[] | select(.roleRef.name == "cluster-admin") | .metadata.name' ``` Review each result. Some may be legitimate, such as a break-glass admin binding, but each should be justified. ### Step 2: Use More Specific Roles Instead of binding users to `cluster-admin`, create or use narrower ClusterRoles. For instance, a user who only needs to view nodes can be bound to a custom role: ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: node-viewer rules: - apiGroups: [""] resources: ["nodes"] verbs: ["get", "list", "watch"] ``` Create the role: ```bash kubectl apply -f node-viewer-clusterrole.yaml ``` Then create a ClusterRoleBinding that binds this role to a specific user or group: ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: node-viewer-binding roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: node-viewer subjects: - kind: User name: jane@example.com apiGroup: rbac.authorization.k8s.io ``` Apply it: ```bash kubectl apply -f node-viewer-binding.yaml ``` Now `jane@example.com` can list nodes but cannot modify them or access other resources. Similarly, if a user only needs read access to pods across all namespaces, create a `pod-reader` ClusterRole with `get`, `list`, and `watch` verbs on `pods`, and bind it accordingly. ### Step 3: Audit Service Account Bindings Service accounts are often over-privileged because default settings may grant broad access, or applications may request more than they need. Check all service accounts bound to cluster roles: ```bash kubectl get clusterrolebindings -o json | jq '.items[] | select(.subjects[]?.kind == "ServiceAccount") | .metadata.name' ``` For each binding, inspect the details: ```bash kubectl get clusterrolebinding -o yaml ``` Ask these questions: - Does the service account actually need cluster-wide permissions, or would a RoleBinding in a specific namespace suffice? - If cluster-wide access is necessary, is the role limited to the necessary resources and verbs? - Is the service account still in use? Check deployments and statefulsets that reference it: `kubectl get pods --all-namespaces -o json | jq '.items[] | select(.spec.serviceAccountName == "") | .metadata.name'` Reduce permissions where possible. For example, if a service account only needs to read configmaps in the `kube-system` namespace, use a Role and RoleBinding instead of a ClusterRoleBinding. ### Step 4: Remove Unused Bindings List bindings sorted by creation date to find old or unused ones: ```bash kubectl get clusterrolebindings --sort-by=.metadata.creationTimestamp ``` Older bindings are candidates for removal. Before deleting, verify they are not referenced by any active subject. For user bindings, check if the user still exists in your identity provider. For service accounts, check if any workloads use them (as shown above). Delete any that are no longer needed: ```bash kubectl delete clusterrolebinding unused-binding-name ``` ### Step 5: Use Groups Instead of Individual Users Binding roles to groups simplifies management. Instead of creating a binding for each user, bind a role to a group and manage group membership in your identity provider (e.g., Active Directory, LDAP, OIDC). Example ClusterRoleBinding for a developer group: ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: developer-binding roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: developer-cluster-role subjects: - kind: Group name: developers apiGroup: rbac.authorization.k8s.io ``` This binds all members of the `developers` group to the `developer-cluster-role`, which should be scoped to typical developer tasks like creating deployments, viewing logs, and accessing configmaps. When a developer leaves, remove them from the group; no cluster changes are needed. ### Step 6: Limit Use of Wildcards Wildcards (`*`) in RBAC rules are dangerous. Avoid them whenever possible. For example, a rule like: ```yaml rules: - apiGroups: ["*"] resources: ["*"] verbs: ["*"] ``` grants full access to all resources, effectively equivalent to cluster-admin. Even a seemingly limited wildcard can be risky: `resources: ["*"]` includes future resources that may be added to the cluster, potentially expanding access unintentionally. Instead, explicitly list the API groups, resources, and verbs needed. If you must use a wildcard, restrict it to a specific API group or resource type and document the rationale. ## Verification and Diagnostics After applying changes, verify that permissions are as intended and that no unintended access is granted. ### Check Effective Permissions Use `kubectl auth can-i` to test permissions for a user or service account. For example, check if user `jane@example.com` can list pods: ```bash kubectl auth can-i list pods --as jane@example.com ``` Expected output: `yes` or `no`. For a service account, use the `--as` flag with the fully qualified name: ```bash kubectl auth can-i get nodes --as=system:serviceaccount:default:my-sa ``` You can also test specific verbs on specific resources: ```bash kubectl auth can-i create deployments --as jane@example.com --namespace my-namespace ``` To list all permissions for a subject, use `--list`: ```bash kubectl auth can-i --list --as jane@example.com ``` This shows a table of resources and allowed verbs. ### List All Roles for a Subject There is no direct command to list all roles for a subject, but you can use `kubectl` and `jq` to extract bindings for a specific user or group. Example for user `jane@example.com`: ```bash kubectl get clusterrolebindings -o json | jq '.items[] | select(.subjects[]?.kind == "User" and .name == "jane@example.com") | .roleRef.name' ``` For a service account, replace the subject kind and name accordingly. Also check RoleBindings in relevant namespaces, as those are separate. ### Review Audit Logs If audit logging is enabled, check logs for denied requests that may indicate overly restrictive permissions or potential attack attempts. Look for entries with `responseStatus.code` in the 403 range and `user.name` matching the affected subject. Most managed Kubernetes services provide audit logs. For self-managed clusters, ensure the audit policy includes RBAC-related events. Example audit policy snippet: ```yaml apiVersion: audit.k8s.io/v1 kind: Policy rules: - level: Metadata verbs: ["create", "update", "patch", "delete"] resources: - group: "rbac.authorization.k8s.io" resources: ["clusterroles", "clusterrolebindings", "roles", "rolebindings"] ``` ### Run a RBAC Audit Tool Tools like `kubescape` or `kube-bench` can scan for overly permissive RBAC configurations. Example with `kubescape`: ```bash kubescape scan framework nsa --exclude-namespaces kube-system ``` Review the RBAC-related findings. `kubescape` flags rules that grant excessive permissions and suggests remediation. ## Failure Modes and Recovery Mistakes in ClusterRoleBindings can lock out users or break applications. This section covers common failure modes and how to recover. ### Accidental Removal of Admin Access If you delete or modify the `cluster-admin` binding and lock yourself out, recovery depends on having another admin path. **Prevention:** Maintain at least one emergency admin account that is not part of the default bindings. For example, create a dedicated service account `break-glass-admin` in a secure namespace, bind it to `cluster-admin`, and store its credentials in a sealed secret or a password manager with limited access. Only use it in emergencies. Review its use quarterly. The platform security lead (e.g., "Priya Shah, Engineering Lead") owns this account and must ensure it is rotated every 90 days. **Recovery:** If you are locked out and have no emergency account, use cloud provider or infrastructure emergency access. For managed clusters (EKS, GKE, AKS), you can often use the cloud IAM to reset the cluster admin role. For on-prem clusters, you may need to access the etcd backup or use a static token file. Always have a procedure documented. ### Overly Permissive Binding Leads to Security Incident If a binding is too broad and a breach occurs, immediately revoke the binding: ```bash kubectl delete clusterrolebinding over-permissive-binding ``` Then investigate the scope: check audit logs for actions performed by the compromised subject, inspect pods and workloads for unauthorized changes, and rotate any secrets that may have been exposed. After the incident, conduct a root cause analysis with the security team and update RBAC policies. ### Application Service Account Loses Necessary Permissions If an application starts failing with `403 Forbidden`, check the service account's bindings. ```bash kubectl describe clusterrolebinding ``` Audit logs can show the denied requests. Look for `responseStatus.code: 403` and `serviceAccountName` in the request. Restore necessary permissions by applying the correct binding or adding the missing verb. Test with `kubectl auth can-i --as=system:serviceaccount:: ` before redeploying. ### Rollback Strategy Always back up existing bindings before making changes, as described in the snapshot step. To restore a specific binding from the backup file, extract it using `kubectl` with a filter or manually edit the YAML. Example using `yq` (if installed): ```bash yq eval 'select(.metadata.name == "my-binding")' clusterrolebindings-backup-20240101.yaml | kubectl apply -f - ``` If you don't have `yq`, use `kubectl get -f` with a label selector, or open the backup file and copy the relevant object. Be cautious: applying the entire backup may reintroduce unwanted bindings. Only restore the specific objects you need. ## Common Pitfalls and How to Avoid Them Beyond the failure modes above, here are recurring mistakes teams make with ClusterRoleBindings and how to prevent them. ### 1. Using ClusterRoleBinding When a Namespaced RoleBinding Would Suffice **Pitfall:** Developers often request cluster-wide access because it is easier than determining the exact namespaces needed. This broadens the attack surface unnecessarily. **Why it happens:** Lack of awareness about namespaced RBAC objects, or default tooling that encourages cluster-wide roles. **Avoidance:** Before creating a ClusterRoleBinding, ask: "Does this subject need access in all namespaces?" If only a few namespaces are needed, use a Role and RoleBinding in each namespace, or use a ClusterRole with a RoleBinding to grant cluster-scoped permissions in a single namespace. **Recovery:** Convert the ClusterRoleBinding to a RoleBinding and delete the cluster-wide binding. Test with `kubectl auth can-i` for the specific namespaces. ### 2. Overuse of Wildcards **Pitfall:** Using `*` for apiGroups, resources, or verbs is convenient but dangerous. It often grants permissions the subject does not need, including access to future resources. **Why it happens:** Time pressure, lack of RBAC knowledge, or copying examples from the internet. **Avoidance:** Explicitly list required resources and verbs. Use tools like `kubectl auth can-i --list` to determine the minimal set needed. Enforce policies with OPA or Kyverno that reject wildcard rules in production. **Recovery:** Replace wildcard rules with explicit lists. Test thoroughly to ensure the application still works. ### 3. Forgetting to Remove Bindings for Departed Users **Pitfall:** When a user leaves the organization, their user binding remains, potentially allowing unauthorized access if their credentials are compromised. **Why it happens:** Manual offboarding processes often miss RBAC cleanup, especially for cluster-wide bindings that are not visible in namespace-scoped dashboards. **Avoidance:** Integrate RBAC cleanup into the offboarding checklist. Use group bindings so removing a user from a group automatically revokes access. The IT security team should manage group membership in the identity provider. **Recovery:** Regularly audit bindings for inactive users. You can script a check that compares user subjects against your HR system. ### 4. Binding to the Default Service Account **Pitfall:** Binding a cluster role to the default service account in a namespace means every pod in that namespace inherits the permissions unless `automountServiceAccountToken` is disabled. This is a common mistake when developers want to give a single pod extra permissions. **Why it happens:** The default service account is easy to reference, and novice users may not realize the implication. **Avoidance:** Always create a dedicated service account for workloads that need specific permissions, and bind the role to that service account only. Disable automounting of service account tokens in pods that do not need Kubernetes API access. **Recovery:** Remove the binding from the default service account, create a new service account for the workload, and update the deployment to use it. Test with `kubectl auth can-i`. ### 5. Not Reviewing Third-Party Installations **Pitfall:** Helm charts or operators may install ClusterRoleBindings with excessive permissions. The `cluster-admin` binding is sometimes included by default. **Why it happens:** Vendors prioritize ease of installation over security best practices. **Avoidance:** Before installing, inspect the chart's RBAC manifests. Use `helm template` to preview the resources. After installation, audit all new ClusterRoleBindings and adjust as needed. **Recovery:** If you find an overly permissive binding from a third-party installation, contact the vendor or modify the binding to use a more limited role if possible. Some tools may not function correctly with reduced permissions, so test in a staging environment first. ## Operations Checklist Use this checklist regularly (e.g., monthly or after major changes) to maintain ClusterRoleBinding security. The platform security lead (e.g., Priya Shah, Engineering Lead) owns this checklist and reviews it every 30 days. | Task | Command / Action | |------|------------------| | List all ClusterRoleBindings | `kubectl get clusterrolebindings` | | Identify subjects with cluster-admin | `kubectl get clusterrolebinding cluster-admin -o yaml` | | Check service accounts bound to cluster roles | `kubectl get clusterrolebindings -o json \| jq '.items[] \| select(.subjects[]?.kind == "ServiceAccount") \| .metadata.name'` | | Remove unused bindings | Review list and delete with `kubectl delete clusterrolebinding ` | | Test user permissions | `kubectl auth can-i --list --as=` | | Backup current bindings | `kubectl get clusterrolebindings -o yaml > clusterrolebindings-backup.yaml` | | Review audit logs for denied requests | Check audit backend logs for 403 errors and unexpected access | | Run automated RBAC scanner | `kubescape scan framework nsa` | | Review emergency admin account | Check last usage and rotate credentials if needed | | Inspect third-party bindings | `kubectl get clusterrolebindings -l app.kubernetes.io/managed-by=Helm` | ## Conclusion Hardening Kubernetes ClusterRoleBindings is essential for cluster security. By following the steps in this guide, you can reduce the attack surface, enforce least privilege, and maintain a clear inventory of who can do what across the cluster. Start by auditing your current bindings, then apply the safe configuration changes described. Always verify permissions with `kubectl auth can-i` and have a rollback plan. Use the operations checklist regularly to keep your cluster secure over time. Remember that RBAC is a continuous process: review permissions quarterly, adapt to new workloads, and stay informed about Kubernetes security best practices.