## Intro Kubernetes Role-Based Access Control (RBAC) is the primary mechanism for regulating access to cluster resources. It answers two fundamental questions: who can do what, and where can they do it. Without a solid RBAC strategy, a cluster can quickly become a security incident waiting to happen, whether from a compromised container, a disgruntled employee, or a simple human error. A well-implemented RBAC setup ensures that users and applications have only the permissions they need, nothing more. This principle of least privilege is not just a nice-to-have; it is a requirement for compliance frameworks like SOC 2, ISO 27001, and PCI DSS. It also makes day-to-day operations safer because an over-privileged service account cannot accidentally delete a production namespace. This article is a practical guide to Kubernetes RBAC. We will explain the core components, demonstrate how they interact, and walk through a safe implementation path with concrete examples. You will learn how to verify your RBAC configuration, diagnose common issues, and recover from failures. By the end, you will have an operational checklist to keep your cluster secure and compliant. ## Version and Environment Inventory Before diving into RBAC configuration, take stock of your Kubernetes environment. RBAC has been stable and enabled by default since Kubernetes v1.8, but newer versions may introduce new API groups or deprecate old ones. For this guide, we assume: - Kubernetes cluster version 1.24 or later. - kubectl client version matching or newer than the cluster. - Access to the cluster with cluster-admin privileges for initial setup. - A clear understanding of your user base: human users (via authentication providers) and service accounts for applications. Check your cluster version: ```bash kubectl version --short ``` Expected output (example): ```text Client Version: v1.28.0 Server Version: v1.28.0 ``` Identify the authentication mechanism in place. Kubernetes does not manage user accounts natively; identity comes from external systems like X.509 client certificates, OpenID Connect (OIDC) providers, or cloud IAM. Service accounts, however, are managed by Kubernetes itself. List existing service accounts in the default namespace: ```bash kubectl get serviceaccounts ``` Expected output (example): ```text NAME SECRETS AGE default 0 10d ``` RBAC is additive: there are no deny rules. If a user has multiple role bindings, their effective permissions are the union of all granted permissions. This means you must be careful not to accidentally grant more access than intended through overlapping roles. Plan your role design around actual job functions: developers, operators, auditors, CI/CD systems, and so on. Each function should map to a clear set of allowed actions on specific resources in specific namespaces. Finally, ensure you have a break-glass access method before making any RBAC changes. This could be a cloud provider's root account, a static kubeconfig with cluster-admin, or an emergency user defined in your identity provider. Losing all administrative access to a cluster is a painful recovery scenario that is entirely avoidable with proper planning. ## Safe Configuration Path A safe approach to RBAC is incremental. Start with minimal roles and expand only when a clear need arises. Avoid wildcard permissions such as `*` for resources or verbs unless absolutely necessary, and always scope permissions to the smallest possible boundary, typically a namespace. ### Core RBAC Components RBAC rests on four object types: - **Role**: defines permissions within a single namespace. - **ClusterRole**: defines permissions cluster-wide or for cluster-scoped resources like nodes or persistent volumes. A ClusterRole can also be used to grant access across all namespaces. - **RoleBinding**: grants a Role to subjects (users, groups, or service accounts) within a specific namespace. - **ClusterRoleBinding**: grants a ClusterRole to subjects cluster-wide. These objects are namespaced or cluster-scoped: - Role and RoleBinding are namespaced; they apply only to the namespace in which they are created. - ClusterRole and ClusterRoleBinding are cluster-scoped; they apply across the entire cluster. Understanding this distinction is critical to designing a secure permission model. ### Step-by-Step Example: Restricting Access to a Namespace Suppose a development team needs to deploy and manage applications in the `dev` namespace. They should not see or modify resources in other namespaces. We will create a Role with permissions to manage common workload resources and then bind it to both a group of human users and a service account. #### Create the Role Create a file `dev-role.yaml`: ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: namespace: dev name: developer rules: - apiGroups: ["", "apps", "batch"] resources: ["pods", "deployments", "services", "configmaps", "secrets", "jobs"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] - apiGroups: [""] resources: ["pods/log"] verbs: ["get", "list"] ``` Apply it: ```bash kubectl apply -f dev-role.yaml ``` Expected output: ```text role.rbac.authorization.k8s.io/developer created ``` This Role grants read/write access to pods, deployments, services, configmaps, secrets, and jobs in the `dev` namespace, but nothing else. Note that `secrets` are included because deploying applications often requires reading secrets, but consider whether your developers actually need to read secret data or only reference them. If they only need to mount secrets into pods, they do not need `get` or `list` permissions on secrets. Limiting secret access is a common hardening step. #### Bind the Role to Subjects Create `dev-rolebinding.yaml`: ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: developer-binding namespace: dev subjects: - kind: Group name: dev-team apiGroup: rbac.authorization.k8s.io - kind: ServiceAccount name: deployer namespace: dev roleRef: kind: Role name: developer apiGroup: rbac.authorization.k8s.io ``` Apply it: ```bash kubectl apply -f dev-rolebinding.yaml ``` Expected output: ```text rolebinding.rbac.authorization.k8s.io/developer-binding created ``` Now any member of the `dev-team` group (as defined in your authentication provider) and any pod running with the `deployer` service account in the `dev` namespace can manage those resources. They cannot, however, access any resource in other namespaces. ### ClusterRole for Read-Only Access Across All Namespaces Sometimes you need a view-only role that spans the entire cluster, for example for auditors or monitoring dashboards. A ClusterRole is appropriate here because the permission boundary is cluster-wide. Create `view-clusterrole.yaml`: ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: view-only rules: - apiGroups: ["*"] resources: ["*"] verbs: ["get", "list", "watch"] ``` Apply: ```bash kubectl apply -f view-clusterrole.yaml ``` Expected output: ```text clusterrole.rbac.authorization.k8s.io/view-only created ``` This ClusterRole allows reading all resources in all API groups. It does not allow `create`, `update`, or `delete`. However, it does allow reading secrets, which may be too permissive for some auditors. In practice, you may want to exclude secrets from this role or create a separate role for sensitive data. We will discuss this in the pitfalls section. Bind it to a single user, say `auditor@example.com`: Create `view-clusterrolebinding.yaml`: ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: view-only-binding subjects: - kind: User name: auditor@example.com apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: view-only apiGroup: rbac.authorization.k8s.io ``` Apply: ```bash kubectl apply -f view-clusterrolebinding.yaml ``` Expected output: ```text clusterrolebinding.rbac.authorization.k8s.io/view-only-binding created ``` Now `auditor@example.com` can read all resources cluster-wide. If you need to grant the same read-only access within a single namespace, you would create a Role with identical rules and bind it with a RoleBinding in that namespace. This is a common pattern: define a ClusterRole once and then bind it in multiple namespaces using RoleBindings. For example, you could bind the `view-only` ClusterRole to a group in the `dev` namespace only: ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: dev-view-only namespace: dev subjects: - kind: Group name: dev-auditors apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: view-only apiGroup: rbac.authorization.k8s.io ``` This grants `dev-auditors` read access to all resources in `dev` only, even though the ClusterRole is cluster-scoped. This approach avoids duplicating role definitions. ## Verification and Diagnostics After creating roles and bindings, verify that permissions work as intended. The most direct tool is `kubectl auth can-i`, which simulates a request as a specific user or service account. ### Check Access as a User Check if `auditor@example.com` can list pods: ```bash kubectl auth can-i list pods --as auditor@example.com ``` Expected output: ```text yes ``` Check if they can delete pods: ```bash kubectl auth can-i delete pods --as auditor@example.com ``` Expected output: ```text no ``` The `--as` flag impersonates the given user for the authorization check. This works regardless of your current credentials, so you can test any user or group. ### Check Access for a Service Account Service accounts are identified by the prefix `system:serviceaccount:`. For a service account named `deployer` in the `dev` namespace, you would use: ```bash kubectl auth can-i create deployments --as system:serviceaccount:dev:deployer -n dev ``` Expected output: ```text yes ``` Check if the same service account can read pods in another namespace, say `kube-system`: ```bash kubectl auth can-i get pods --as system:serviceaccount:dev:deployer -n kube-system ``` Expected output: ```text no ``` This confirms that the Role is correctly scoped to the `dev` namespace. You can also list all permissions for a user or service account: ```bash kubectl auth can-i --list --as system:serviceaccount:dev:deployer -n dev ``` This prints a table of all resources and verbs allowed for that subject in that namespace. It is a quick way to audit effective permissions. ### Review RBAC Objects List all roles and role bindings in a namespace: ```bash kubectl get roles, rolebindings -n dev ``` Expected output (example): ```text NAME CREATED AT role.rbac.authorization.k8s.io/developer 2024-05-01T10:00:00Z NAME ROLE AGE rolebinding.rbac.authorization.k8s.io/developer-binding Role/developer 10m ``` Describe a role to see its full rules: ```bash kubectl describe role developer -n dev ``` Expected output snippet: ```text Name: developer Namespace: dev Labels: Annotations: PolicyRule: Resources Non-Resource URLs Resource Names Verbs --------- ----------------- -------------- ----- pods [] [] [get list watch create update patch delete] deployments [] [] [get list watch create update patch delete] services [] [] [get list watch create update patch delete] configmaps [] [] [get list watch create update patch delete] secrets [] [] [get list watch create update patch delete] jobs [] [] [get list watch create update patch delete] pods/log [] [] [get list] ``` This description shows exactly what the role permits. Review it carefully for unintended permissions, especially on sensitive resources like secrets. ## Common Pitfalls and How to Avoid Them Even experienced Kubernetes administrators make mistakes with RBAC. Here are the most frequent pitfalls, why they happen, and how to avoid or recover from them. ### 1. Overly Broad Permissions with Wildcards **Why it happens:** Using `*` for resources or verbs is convenient, especially when debugging. Developers may request "just give me all permissions" to get something working quickly. **How to avoid:** Never use `*` in production roles unless the role is genuinely cluster-wide and read-only for an auditor. Even then, exclude secrets. Instead, list specific resources and verbs. For example: ```yaml rules: - apiGroups: [""] resources: ["pods", "services", "configmaps"] verbs: ["get", "list", "watch"] ``` **Recovery:** If you discover a role with wildcard permissions, replace it with a specific role immediately. Audit all bindings to see who was granted that role and whether any misuse occurred via audit logs. ### 2. Missing or Incorrect Subjects in Bindings **Why it happens:** Typing a username incorrectly, using the wrong kind (e.g., `User` instead of `ServiceAccount`), or referencing a group that does not exist in your identity provider. **How to avoid:** Double-check subject names and kinds. Use `kubectl auth can-i --list --as ` to verify that the subject actually receives the expected permissions. **Recovery:** Fix the binding by correcting the subject. If the wrong subject was granted access, delete the binding and recreate it with the correct one. ### 3. Namespace Confusion **Why it happens:** Thinking that a Role can be used in a ClusterRoleBinding, or creating a Role in the wrong namespace. A Role only exists within its namespace, and a RoleBinding can only reference a Role in the same namespace. **How to avoid:** Clearly document which namespace each role belongs to. Use consistent naming conventions that include the namespace name, like `developer-dev` or `view-only-prod`. **Recovery:** If you referenced a Role from a ClusterRoleBinding (which is invalid), the binding will not function. Create a ClusterRole with the same rules and bind that instead. ### 4. Granting Secrets Access Unnecessarily **Why it happens:** The `view-only` wildcard role includes secrets. Many developers also request secrets access for convenience. **How to avoid:** Separate secrets into their own role with very limited subjects. For most workloads, pods only need secret references via volumes or environment variables, not direct `get` or `list` on secret objects. Only grant secrets read access to specific service accounts that truly need it, such as a secret management controller. **Recovery:** Remove secrets from broad roles. Create a dedicated `secrets-reader` role and bind it only to necessary subjects. ### 5. Accumulation of Unused Roles and Bindings **Why it happens:** Over time, teams change, projects end, and roles are left behind. Unused roles are a security risk because they may grant permissions that are no longer needed. **How to avoid:** Regularly audit roles and bindings. Delete any that are not referenced or whose subjects no longer exist. Use tools like `kubectl get rolebindings --all-namespaces -o json` to script audits. **Recovery:** Simply delete the unused objects. But first ensure no active subject relies on them by checking bindings. ### 6. Locking Yourself Out of the Cluster **Why it happens:** Modifying or deleting the `cluster-admin` ClusterRoleBinding, or removing all bindings for administrative users. **How to avoid:** Always maintain a break-glass account outside of Kubernetes RBAC, such as a cloud provider's root user or a static kubeconfig stored securely. Before making changes to critical admin bindings, test access with `kubectl auth can-i` using a non-admin user. **Recovery:** If locked out, use your break-glass method to restore admin access. Then recreate the necessary RoleBindings or ClusterRoleBindings. ### 7. Ignoring API Groups **Why it happens:** Forgetting that many resources are in non-core API groups like `apps/v1` for deployments or `networking.k8s.io/v1` for network policies. Using an empty string `""` for `apiGroups` only covers core resources like pods and services. **How to avoid:** Always specify the correct API group for each resource. For example: ```yaml - apiGroups: ["apps"] resources: ["deployments", "statefulsets"] verbs: ["get", "list", "watch"] ``` **Recovery:** If permission is denied for a resource, check its API group with `kubectl api-resources` and update the role accordingly. ## Operations Checklist Maintaining a secure RBAC posture requires regular reviews. The following checklist assigns an owner and frequency to each task, ensuring accountability. | Task | Owner | Command / Action | Frequency | |------|-------|------------------|-----------| | Review all cluster roles and bindings | Priya Shah, Security Lead | `kubectl get clusterrolebindings -o yaml` | Monthly | | List all roles in each namespace | Carlos Mendes, Platform Admin | `kubectl get roles --all-namespaces` | Monthly | | List all role bindings in each namespace | Carlos Mendes, Platform Admin | `kubectl get rolebindings --all-namespaces` | Monthly | | Check for wildcard permissions | Priya Shah, Security Lead | `kubectl get clusterroles -o yaml \| grep -B5 '\*'` | Weekly | | Verify service account usage | Janet Okafor, DevOps Engineer | `kubectl get serviceaccounts --all-namespaces` | Weekly | | Test access for critical users | Lin Wei, Platform Admin | `kubectl auth can-i --list --as ` | On change | | Rotate service account tokens | Automated system | Depending on cluster setup | Quarterly | | Review audit logs for denied requests | Priya Shah, Security Lead | Grep audit logs for `"decision":"forbid"` | Weekly | Additionally, adhere to these principles: - Always create roles with the least privilege required. - Prefer Role and RoleBinding for namespace-scoped access. - Use ClusterRole and ClusterRoleBinding only for cluster-wide resources or when access must span all namespaces. - Avoid using the default service account for applications; create dedicated service accounts with specific RBAC. - Regularly remove unused roles and bindings. - Document each role's purpose, its subjects, and the owner who requested it. For audit logs, configure your API server to log authorization decisions. A sample audit policy to log denied requests for pods: ```yaml apiVersion: audit.k8s.io/v1 kind: Policy rules: - level: Metadata verbs: ["*"] resources: - group: "" resources: ["pods"] omitStages: - "RequestReceived" ``` Then inspect logs for entries with `"decision":"forbid"` to identify unauthorized attempts. ## Conclusion Kubernetes RBAC is a powerful tool for securing your cluster, but it requires deliberate design and ongoing maintenance. By understanding its architecture and following good practices, you can ensure that every user and application has only the permissions necessary for their function. Start with a clear inventory of your environment. Implement roles and bindings incrementally, avoiding wildcards and over-permissive grants. Verify access regularly using `kubectl auth can-i`. Have a recovery plan for misconfigurations, and assign clear ownership for RBAC hygiene tasks. Use the provided checklist to maintain a secure posture over time. With these practices in place, you can confidently manage access in your Kubernetes clusters, reducing the risk of security breaches and operational incidents.