## Intro Kubernetes authorization determines what actions a user, group, or service account can perform on cluster resources. While authentication proves who you are, authorization controls what you are allowed to do. Misconfigured authorization is a leading cause of security incidents and operational surprises in Kubernetes clusters. This guide explains common authorization patterns, shows practical `kubectl` commands with realistic output, and demonstrates how to verify and troubleshoot access. It is written for developers, DevOps consultants, and technical startup teams who need to work safely and efficiently. The focus is on the default Role-Based Access Control (RBAC) system, but the concepts apply to other authorization modes as well. We assume you have a running Kubernetes cluster with `kubectl` configured and at least read access to some resources. All examples use illustrative names; replace them with values from your environment. ## Why Authorization Matters Consider a team onboarding a new developer. You create a kubeconfig file that grants full cluster-admin rights because it is easy. Months later, a misconfiguration deletes a production namespace. This is a common story. Authorization should follow the principle of least privilege: grant only the permissions required for a specific role. RBAC in Kubernetes is built on four key objects: - **Role**: defines permissions within a single namespace. - **ClusterRole**: defines permissions cluster-wide or for namespaced resources across all namespaces. - **RoleBinding**: grants a Role's permissions to users, groups, or service accounts within a namespace. - **ClusterRoleBinding**: grants a ClusterRole's permissions cluster-wide. These objects are managed via the `rbac.authorization.k8s.io/v1` API. ## Prerequisites and Version Check Before running authorization commands, verify your environment. Check the Kubernetes server version: ```bash kubectl version --short ``` Expected output includes both client and server versions, for example: ```text Client Version: v1.29.2 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.29.2 ``` RBAC has been stable since Kubernetes 1.8, but newer versions may add fields like `aggregationRule` or `conditions`. Ensure your API server is reachable: ```bash kubectl cluster-info ``` If you see `Unable to connect to the server`, your kubeconfig or network configuration needs attention. Also confirm your current identity and permissions: ```bash kubectl auth whoami ``` This command, available in Kubernetes 1.24+, shows the authenticated user and their groups. Example output: ```text ATTRIBUTE VALUE Username alice@example.com Groups [system:authenticated, dev-team] ``` ## Viewing Roles and Bindings Start by listing existing Roles in a namespace: ```bash kubectl get roles -n development ``` Sample output: ```text NAME CREATED AT pod-reader 2024-03-15T10:30:00Z ``` To see details of a specific Role: ```bash kubectl get role pod-reader -n development -o yaml ``` This outputs the full YAML definition, including rules. Example snippet: ```yaml rules: - apiGroups: [""] resources: ["pods"] verbs: ["get", "list", "watch"] ``` For cluster-wide permissions, list ClusterRoles: ```bash kubectl get clusterroles ``` You will see many system-defined roles. To focus on custom ones, filter by label or name. For example, to find roles related to monitoring: ```bash kubectl get clusterroles | grep monitoring ``` To view bindings: ```bash kubectl get rolebindings -n development ``` Output: ```text NAME ROLE AGE read-pods Role/pod-reader 2d ``` And cluster-wide bindings: ```bash kubectl get clusterrolebindings ``` To inspect a binding: ```bash kubectl get rolebinding read-pods -n development -o yaml ``` Relevant part of the output: ```yaml subjects: - kind: User name: alice@example.com apiGroup: rbac.authorization.k8s.io roleRef: kind: Role name: pod-reader apiGroup: rbac.authorization.k8s.io ``` This shows that user `alice@example.com` is bound to the `pod-reader` role in the `development` namespace. ## Checking Your Own Permissions The most practical command for authorization is `kubectl auth can-i`. It tests whether the current user can perform an action. For example: ```bash kubectl auth can-i create deployments --namespace development ``` Output: ```text yes ``` If denied, it returns `no`. Add `--list` to see all permissions: ```bash kubectl auth can-i --list --namespace development ``` This produces a table like: ```text Resources Non-Resource URLs Resource Names Verbs pods [] [] [get list watch] services [] [] [get list] ``` You can test permissions for another user or service account using `--as`: ```bash kubectl auth can-i get secrets --namespace production --as system:serviceaccount:production:default ``` This checks if the default service account in the production namespace can get secrets. Output is `yes` or `no`. Useful for troubleshooting when a user reports an access issue: run the same `can-i` command with `--as` their username to see the result from their perspective. ## Creating a Role and RoleBinding Example Let's create a practical example. We want to grant a developer named `bob@example.com` read access to pods in the `staging` namespace. First, create a Role manifest `pod-reader-role.yaml`: ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: namespace: staging name: pod-reader rules: - apiGroups: [""] resources: ["pods"] verbs: ["get", "list", "watch"] ``` Apply it: ```bash kubectl apply -f pod-reader-role.yaml ``` Then create a RoleBinding manifest `pod-reader-binding.yaml`: ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: read-pods namespace: staging subjects: - kind: User name: bob@example.com apiGroup: rbac.authorization.k8s.io roleRef: kind: Role name: pod-reader apiGroup: rbac.authorization.k8s.io ``` Apply it: ```bash kubectl apply -f pod-reader-binding.yaml ``` Now verify Bob's access: ```bash kubectl auth can-i list pods --namespace staging --as bob@example.com ``` Output: `yes`. And verify Bob cannot delete pods: ```bash kubectl auth can-i delete pods --namespace staging --as bob@example.com ``` Output: `no`. This demonstrates least privilege in action. ## Creating a ClusterRole and ClusterRoleBinding Example For cluster-wide read-only access, use a ClusterRole. Create `cluster-pod-reader.yaml`: ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: cluster-pod-reader rules: - apiGroups: [""] resources: ["pods"] verbs: ["get", "list", "watch"] ``` Apply it: ```bash kubectl apply -f cluster-pod-reader.yaml ``` Then bind it to a group `dev-team`: ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: read-pods-global subjects: - kind: Group name: dev-team apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: cluster-pod-reader apiGroup: rbac.authorization.k8s.io ``` Apply and test for a user in that group: ```bash kubectl auth can-i list pods --all-namespaces --as carol@example.com ``` If Carol is a member of `dev-team`, output is `yes`. Otherwise, `no`. ## Service Account Authorization Applications often use service accounts to interact with the Kubernetes API. To grant permissions to a pod, create a service account and bind a role to it. Create service account `app-reader` in namespace `webapp`: ```bash kubectl create serviceaccount app-reader -n webapp ``` Then create a RoleBinding that grants the existing `pod-reader` role to this service account: ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: app-reader-pod-reader namespace: webapp subjects: - kind: ServiceAccount name: app-reader namespace: webapp roleRef: kind: Role name: pod-reader apiGroup: rbac.authorization.k8s.io ``` Apply the binding, then verify by checking if the service account can list pods: ```bash kubectl auth can-i list pods --namespace webapp --as system:serviceaccount:webapp:app-reader ``` Expected output: `yes`. To allow a pod to use this service account, set `serviceAccountName: app-reader` in the pod spec. Then the pod's application can call the API server with the service account's token and have those permissions. ## Inspecting Access from the API Server's Perspective Sometimes you need to see whether an incoming request would be authorized. You can use the `--v=8` flag on the API server to log authorization decisions, but this requires access to the control plane. More commonly, use `kubectl auth reconcile` to ensure bindings match desired state. For example, before applying changes, you can reconcile RBAC objects from a file: ```bash kubectl auth reconcile -f rbac-policies.yaml ``` This command ensures the roles and bindings in the file match the cluster, adding or removing permissions as needed. It is safer than `kubectl apply` because it also removes permissions that are no longer in the file (with the `--remove-extra-permissions` flag). Use with caution. ```bash kubectl auth reconcile -f rbac-policies.yaml --remove-extra-permissions ``` Always review the diff first by adding `--dry-run=client`: ```bash kubectl auth reconcile -f rbac-policies.yaml --remove-extra-permissions --dry-run=client -o yaml ``` This shows what would change without applying. ## Troubleshooting Authorization Issues When a user or application reports "forbidden" errors, follow a systematic approach. ### Step 1: Identify the user and the requested action Ask for the exact error message. Kubernetes returns clear messages like: ```text Error from server (Forbidden): pods is forbidden: User "alice@example.com" cannot list resource "pods" in API group "" in the namespace "production" ``` This tells you the user (alice@example.com), the resource (pods), the namespace (production), and the verb (list). ### Step 2: Check existing bindings List role bindings in the namespace: ```bash kubectl get rolebindings -n production ``` And cluster bindings: ```bash kubectl get clusterrolebindings | grep alice ``` Examine the roles referenced by those bindings to see what verbs are allowed. ### Step 3: Use can-i with --as Simulate the user's access: ```bash kubectl auth can-i list pods --namespace production --as alice@example.com ``` If output is `no`, the user lacks that permission. ### Step 4: Review role rules for matches Check if the role includes the necessary API group, resource, and verb. A common mistake is using the wrong API group. For example, `deployments` are in the `apps` group, not the core group (`""`). A role that grants `pods` access does not grant `pods/log` access; that is a separate subresource. ### Step 5: Check for denial via RBAC or other admission controllers RBAC is one of several authorization and admission steps. If the user has the role but is still denied, consider other admission webhooks or Pod Security Policies (deprecated). Check the API server audit logs if available. ## Using Impersonation for Advanced Testing To test as another user without having their credentials, use impersonation. You need permission to impersonate users, often granted to cluster administrators. The `--as` flag in `kubectl` requests impersonation. Example: ```bash kubectl get pods --namespace development --as=alice@example.com ``` If you receive `forbidden`, it means the API server rejected the impersonation (you lack the `impersonate` verb on the user) or Alice does not have access. To impersonate a group: ```bash kubectl get pods --as=system:serviceaccount:webapp:app-reader ``` This is useful for verifying service account permissions. ## Auditing Authorization Changes Track changes to RBAC objects using `kubectl get events` or audit logs. Events are not detailed for RBAC, but you can view the history of a role via `kubectl rollout status` (not applicable) or by checking the object's metadata and annotations if you added them. Better practice is to enable Kubernetes audit logging. Configure the API server with an audit policy that logs `rbac.authorization.k8s.io` resource changes. Then, when troubleshooting, you can query the audit log to see who made changes and when. For example, a policy might include: ```yaml rules: - level: RequestResponse resources: - group: "rbac.authorization.k8s.io" resources: ["roles", "rolebindings", "clusterroles", "clusterrolebindings"] ``` Then use tools like `jq` or `grep` to filter the audit log for recent changes. ## Safe Configuration Practices When modifying authorization, follow these best practices: 1. **Use a dedicated service account for CI/CD** instead of your personal user. This limits the blast radius if credentials leak. 2. **Avoid cluster-admin for regular users.** Reserve it for break-glass accounts. 3. **Use groups for role bindings.** Bind permissions to groups, then manage group membership in your identity provider. This reduces the need to edit Kubernetes objects when people join or leave. 4. **Test changes in a non-production cluster first.** Apply the same RBAC manifests to a staging cluster and verify with `can-i`. 5. **Use `--dry-run=client` to preview changes.** For example: ```bash kubectl apply -f role.yaml --dry-run=client -o yaml ``` This shows the object that would be applied without actually applying it. 6. **Back up existing RBAC objects before deletion.** Export them: ```bash kubectl get rolebindings -n development -o yaml > rbac-backup.yaml ``` Then you can reapply if needed. 7. **Consider using `kubectl auth reconcile`** to manage RBAC declaratively, but only after reviewing the changes with `--dry-run`. ## Common Pitfalls and How to Avoid Them ### 1. Confusing Verbs and Resources **Mistake**: Granting `update` but not `patch`, or `list` but not `get`. Some controllers use `watch` along with `list` for efficient operation. Omitting necessary verbs causes intermittent failures. **Avoidance**: Test with `kubectl auth can-i --list` to see all permissions for a subject, and align the verb set with the actions the subject should perform. For read-only monitoring, `get`, `list`, and `watch` are often needed. **Recovery**: Add the missing verb to the role and reapply. Verify with `can-i`. ### 2. Using the Wrong API Group **Mistake**: Creating a role for `deployments` with `apiGroups: [""]` (core group). Deployments live in the `apps` group. **Avoidance**: Check the resource's API group with `kubectl api-resources | grep deployment`. The output shows the group, e.g., `deployments apps/v1 true Deployment`. **Recovery**: Fix the `apiGroups` field in the role to `["apps"]` and reapply. ### 3. Forgetting Subresources **Mistake**: Granting access to `pods` but not `pods/log`. A user can list pods but cannot view logs. **Avoidance**: Identify which subresources are needed and include them explicitly. For example, to allow log access, add `resources: ["pods/log"]` with the `get` verb. **Recovery**: Update the role to include the subresource and reapply. ### 4. Overly Broad Permissions **Mistake**: Using `*` for verbs or resources out of convenience. This violates least privilege and increases risk. **Avoidance**: Start with the smallest set of permissions and expand only when necessary. Use `kubectl auth can-i --list` to see what is actually granted. **Recovery**: Replace wildcards with explicit lists. For example, instead of `verbs: ["*"]`, use `verbs: ["get", "list", "watch"]` for read-only roles. ### 5. RoleBinding in Wrong Namespace **Mistake**: Binding a Role in the `default` namespace when the user needs access in `production`. RoleBindings are namespace-scoped; they do not grant access outside their namespace. **Avoidance**: Double-check the `namespace` field in both the Role and RoleBinding manifests. Use `kubectl -n ` consistently. **Recovery**: Create a new RoleBinding in the correct namespace, or use a ClusterRoleBinding if access is needed across all namespaces. ### 6. Not Testing with Service Account Context **Mistake**: A pod fails to access the API, but the error is buried in application logs. The developer assumes the RBAC is correct because they tested with their own user. **Avoidance**: Always test service account permissions with `kubectl auth can-i --as system:serviceaccount::`. Also, inspect the pod's service account and ensure the token is mounted. **Recovery**: Fix the service account name in the pod spec, or create the appropriate RoleBinding for the service account. ## Verification and Diagnostics Commands Summary Here is a cheat sheet of common authorization commands: | Command | Purpose | Example Output | |---|---|---| | `kubectl auth can-i ` | Check current user permission | `yes` or `no` | | `kubectl auth can-i --list` | List all permissions for current user | Table of resources and verbs | | `kubectl auth can-i --as ` | Check permission for another user | `yes` or `no` | | `kubectl auth can-i --as-group ` | Check permission for a group | `yes` or `no` | | `kubectl get roles -n ` | List roles in namespace | Names | | `kubectl get role -n -o yaml` | View role rules | YAML with rules | | `kubectl get rolebindings -n ` | List role bindings | Names and roles | | `kubectl get clusterroles` | List cluster roles | Many system roles | | `kubectl get clusterrolebindings` | List cluster role bindings | Names and subjects | | `kubectl auth reconcile -f --dry-run=client` | Preview RBAC reconciliation | YAML diff | | `kubectl create role ...` | Create a role | Role created | | `kubectl create rolebinding ...` | Create a role binding | Binding created | These commands cover most day-to-day authorization tasks. ## Operations Checklist for Authorization Maintenance As a cluster operator, schedule regular reviews of RBAC. Here is a practical checklist: - **Weekly**: Review new cluster-role bindings (e.g., `kubectl get clusterrolebindings | grep -v system:` to see non-system bindings) and revoke any that are unnecessary. - **Monthly**: Audit service accounts with high privileges using `kubectl get serviceaccounts --all-namespaces` and check if any are bound to cluster-admin or other powerful roles. - **Quarterly**: Perform a full RBAC review: export all roles and bindings, compare against a documented policy, and remove unused or overly broad permissions. - **On-demand**: When a security incident occurs, immediately inspect the audit logs and revoke risky permissions using `kubectl delete rolebinding -n ` or `kubectl delete clusterrolebinding `. Accountability: Assign a Kubernetes Security Owner (e.g., the platform team lead) who is responsible for the RBAC review process. That owner should be named in your documentation with a fallback delegate. ## Failure Modes and Recovery Authorization failures often manifest in application logs or user reports. Understanding common failure modes helps in quick recovery. ### Failure: User Suddenly Loses Access **Cause**: A RoleBinding was deleted or modified, possibly by an automated script or inadvertently. **Diagnosis**: Check recent changes with `kubectl get events --sort-by=.lastTimestamp` or audit logs. Look for deletion of bindings. **Recovery**: Recreate the binding from backup or Git history. If no backup, use `kubectl get rolebindings -n -o yaml` to see existing bindings and compare with expected. If necessary, create a new binding. ### Failure: New User Cannot Access Anything **Cause**: The user is not bound to any role, or the binding exists but the user's authentication identity does not match (e.g., email vs username). **Diagnosis**: Run `kubectl auth can-i --list --as=` to see no permissions. Confirm the user's identity with `kubectl auth whoami` (if they can run it) or ask them to run `kubectl config view` to see their current context user. **Recovery**: Create the appropriate RoleBinding for their username or group. ### Failure: Service Account Token Invalid **Cause**: The service account's token was rotated or the pod is not using the correct service account. **Diagnosis**: Describe the pod and check `serviceAccountName`. Check the service account's secrets (if auto-mount is enabled). Attempt to use the token manually with `curl` to the API server. **Recovery**: Ensure the pod spec has the correct `serviceAccountName`. If the token is compromised, delete the secret to force regeneration (in older versions) or rely on projected tokens (default in 1.24+). ### Failure: Permissions Seem Correct but Still Denied **Cause**: Another admission controller (like a validating webhook) is rejecting the request, or there is a network policy blocking the API server connection. **Diagnosis**: Check the API server logs for authorization decisions. If using webhooks, inspect their logs. Temporarily disable the webhook (if possible) to isolate the issue. **Recovery**: Fix the webhook configuration or adjust the network policy. ## Conclusion Kubernetes authorization is a critical component of cluster security. By using the practical commands and examples in this guide, you can effectively manage roles, bindings, and service accounts while adhering to the principle of least privilege. Regularly test permissions with `kubectl auth can-i`, review RBAC configurations, and keep an eye on audit logs to maintain a secure and well-functioning cluster. Remember: observe before changing, limit blast radius, use placeholders instead of secrets, verify results, and document recovery steps. With these practices, you can ensure that the right people and services have exactly the access they need.