## Intro Role-Based Access Control (RBAC) is the primary authorization mechanism in Kubernetes, controlling who can perform which actions on which resources. Misconfigured RBAC is one of the most common causes of access denied errors, failed deployments, and security audit findings. This guide provides a practical, command-driven approach to troubleshooting RBAC issues in Kubernetes clusters. You will learn how to systematically identify permission problems, verify configurations, apply least-privilege fixes, and avoid common pitfalls. This article is intended for developers, DevOps engineers, and platform operators who manage Kubernetes clusters and need to resolve authorization failures quickly without compromising security. Every command includes expected output or a clear signal of success, and every recommendation is scoped to a specific Kubernetes version (1.24+ unless otherwise noted). The goal is operational safety: understand the current state, make minimal changes, verify the result, and document recovery steps. ## Version and Environment Inventory Before troubleshooting RBAC, confirm the version and environment characteristics. RBAC is stable across all supported Kubernetes versions, but API group names and certain flags may vary. Run these read-only commands to capture the current state: - `kubectl version --short` (deprecated in 1.26, use `kubectl version` without `--short`) - `kubectl cluster-info` - `kubectl get nodes -o wide` Example output for `kubectl version` on a client: ``` Client Version: v1.27.3 Kustomize Version: v5.0.1 Server Version: v1.26.5 ``` Confirm that the server version matches your expectations. RBAC API is `rbac.authorization.k8s.io/v1` since Kubernetes 1.8, so no version-specific changes are needed for basic troubleshooting. Identify the exact component failing. Is it a user, a service account, or a group? Use `kubectl auth can-i --list` to see all permissions for the current user, or specify another user with `--as`: ```bash kubectl auth can-i --list --as=system:serviceaccount:default:my-app ``` This outputs a table of resources and allowed verbs. If you get no output, the user may have no permissions at all. For service accounts, confirm the namespace and existence: ```bash kubectl get serviceaccount my-app -n default ``` If not found, the pod referencing it will fail to start with a `CreateContainerConfigError`. Check events: ```bash kubectl describe pod my-pod -n default | grep -A5 Events ``` Always start with read-only observations and avoid making changes until you understand the blast radius. ## Safe Configuration Path RBAC configuration consists of four main object types: Role, ClusterRole, RoleBinding, and ClusterRoleBinding. A safe troubleshooting path begins with viewing existing objects and testing permissions before modifying anything. **Step 1: View current RBAC objects.** List roles and bindings in the relevant namespace: ```bash kubectl get roles,rolebindings -n my-namespace kubectl get clusterroles,clusterrolebindings | grep my-app ``` **Step 2: Test permissions without changing anything.** Use `kubectl auth can-i` to check specific actions for a user or service account: ```bash kubectl auth can-i create deployments --as=system:serviceaccount:default:my-app -n my-namespace ``` Expected output if allowed: ``` yes ``` If denied: ``` no ``` **Step 3: Inspect the effective rules.** Use `kubectl describe` to view the role's rules: ```bash kubectl describe role my-app-role -n my-namespace ``` Example output: ``` Name: my-app-role Labels: Annotations: PolicyRule: Resources Non-Resource URLs Resource Names Verbs --------- ----------------- -------------- ----- pods [] [] [get list watch] deployments [] [] [get list watch create update patch delete] ``` **Step 4: Make the smallest justified change.** If the role lacks a required verb, create a patch file or use `kubectl edit` with care. For example, to add `delete` permission for pods, create a YAML patch: ```yaml # add-delete-pods.yaml rules: - apiGroups: [""] resources: ["pods"] verbs: ["get", "list", "watch", "delete"] ``` Apply with: ```bash kubectl patch role my-app-role -n my-namespace --patch "$(cat add-delete-pods.yaml)" ``` **Step 5: Verify the change.** Re-run `kubectl auth can-i` to confirm the new permission is effective. Also, check that the binding still references the correct role and subject: ```bash kubectl get rolebinding my-app-binding -n my-namespace -o yaml ``` Never use wildcard `*` in verbs or resources unless absolutely necessary, and avoid granting cluster-admin to service accounts. ## Verification and Diagnostics After making a change, verify both the configuration and the actual behavior. Here are key diagnostic commands: **Check all permissions for a service account:** ```bash kubectl auth can-i --list --as=system:serviceaccount:default:my-app -n my-namespace ``` Example output snippet: ``` Resources Non-Resource URLs Resource Names Verbs pods [] [] [get list watch] deployments.apps [] [] [create delete get list patch update watch] ``` **Check a specific action on a specific resource:** ```bash kubectl auth can-i get secret/my-secret --as=system:serviceaccount:default:my-app -n my-namespace ``` **View RBAC-related events in the namespace:** ```bash kubectl get events -n my-namespace --field-selector reason=FailedToGetResource ``` **Examine API server audit logs (if enabled).** For managed clusters, check cloud provider logs. For self-managed, grep the audit log: ```bash grep "my-app" /var/log/kubernetes/audit.log | grep "forbidden" ``` Audit logs provide detailed reasons for denials, such as missing verbs or missing resource names. If you see `forbidden: User "system:serviceaccount:default:my-app" cannot get resource "pods"`, it indicates a missing RBAC permission. **Test pod execution with the service account.** Create a temporary pod with the service account and run `kubectl` inside it (if the image has kubectl) or use `kubectl run`: ```bash kubectl run test-pod --image=bitnami/kubectl:latest --restart=Never --serviceaccount=my-app -n my-namespace -- sleep 3600 ``` Then exec into it and run a command that should be allowed or denied: ```bash kubectl exec -it test-pod -n my-namespace -- kubectl get pods ``` If denied, you will see a `Forbidden` error. Clean up with `kubectl delete pod test-pod -n my-namespace`. ## Failure Modes and Recovery RBAC failures typically manifest as `Forbidden` errors in logs, failed API calls, or pods stuck in CrashLoopBackOff. Here are the most common failure modes and how to recover. **1. Missing permission for a service account used by a controller.** Symptoms: Controller logs show `403 Forbidden` when watching resources. Example: ``` E0120 10:00:00.123456 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.0.0/tools/cache/reflector.go:167: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:my-namespace:my-controller" cannot list resource "pods" in API group "" in the namespace "my-namespace" ``` Recovery: Create or update a Role granting `list` and `watch` on `pods`, and bind it to the service account. Apply and restart the controller pod to pick up changes (if not using dynamic reload). **2. User cannot perform action but has a role that seems sufficient.** Symptom: `kubectl auth can-i` returns `no` despite having a role with the verb. Possible causes: role binding in wrong namespace, subject name mismatch, or API group mismatch. Recovery: Verify binding details: ```bash kubectl get rolebinding my-binding -n my-namespace -o yaml ``` Check `subjects` and `roleRef`. Ensure the user's name is exactly as authenticated (case-sensitive). For cluster-scoped resources, ensure you used `ClusterRole` and `ClusterRoleBinding`, not namespaced ones. **3. Wildcard permissions causing unintended access.** Symptom: A service account has more access than intended, discovered in an audit. Recovery: Replace wildcard rules with explicit resources and verbs. Example bad rule: ```yaml rules: - apiGroups: ["*"] resources: ["*"] verbs: ["*"] ``` Replace with explicit list, e.g., only `pods` and `deployments` with specific verbs. Use `kubectl auth can-i --list` to document the reduced permissions. **4. ClusterRoleBinding created instead of RoleBinding for namespaced access.** Symptom: User can access all namespaces, not just intended one. Recovery: Delete the ClusterRoleBinding and create a RoleBinding in the specific namespace. Verify with `kubectl auth can-i --list --as=user -n target-ns`. **5. RBAC changes not taking effect.** Symptom: After applying a new role, access is still denied. RBAC is eventually consistent but changes should propagate within seconds. If not, check for typos or that the API server has the RBAC authorizer enabled (default). Verify with `kubectl get clusterrolebinding system:node -o yaml` to ensure RBAC is functioning. If issues persist, restart the API server (only for self-managed clusters). ## Operations Checklist Use this checklist during any RBAC troubleshooting session. Assign an owner for each item (e.g., the on-call engineer) and review the checklist after every incident or change. For ongoing RBAC hygiene, review permissions quarterly. | # | Item | Owner | Command or Action | Expected Result | Frequency / Review | |---|------|-------|-------------------|-----------------|-------------------| | 1 | Confirm cluster version and RBAC API | DevOps Lead | `kubectl version` | Server v1.24+ | Once per cluster setup | | 2 | Identify failing user or service account | On-call Engineer | `kubectl auth can-i --list --as=` | List of allowed resources | Per incident | | 3 | Check current roles and bindings | On-call Engineer | `kubectl get roles,rolebindings -n ` | Objects exist | Per incident | | 4 | Test specific permission | On-call Engineer | `kubectl auth can-i --as=` | `yes` or `no` | Per incident | | 5 | Inspect role rules | Security Engineer | `kubectl describe role -n ` | Rules match desired access | Quarterly audit | | 6 | Apply minimal RBAC change | DevOps Lead | `kubectl apply -f updated-role.yaml` | Change applied | Per change | | 7 | Verify permission after change | On-call Engineer | Re-run `kubectl auth can-i` | Output changes to `yes` if intended | Per change | | 8 | Review audit logs for denials | Security Engineer | `grep "forbidden" /var/log/kubernetes/audit.log` | No unexpected denials | Monthly | | 9 | Rotate service account tokens if compromised | Security Engineer | `kubectl create token -n ` | New token issued | As needed | | 10 | Document RBAC policies and changes | DevOps Lead | Update internal wiki | Up-to-date documentation | After each change | ## Common Pitfalls Avoid these frequent mistakes when managing Kubernetes RBAC. **Pitfall 1: Assigning cluster-admin to service accounts.** This is tempting for quick fixes but creates a massive security risk. If the pod is compromised, the attacker gains full cluster control. *Why it happens:* Developers often want to avoid permission issues during development. *How to avoid:* Use least privilege from the start. Create specific roles for each service account. Use tools like `kubectl auth can-i` to determine exact permissions needed. *Recovery:* Replace ClusterRoleBinding with a namespaced RoleBinding with minimal permissions. **Pitfall 2: Using wildcard `*` in verbs or resources.** Wildcards grant broad access that may violate security policies and make auditing harder. *Why it happens:* It is quicker than listing every resource. *How to avoid:* Spend the extra minutes to list precise resources and verbs. *Recovery:* Audit existing wildcards with `kubectl get clusterroles,roles -o yaml | grep -B5 -A5 '\*'` and replace them with explicit rules. **Pitfall 3: Forgetting that RBAC is additive only.** There is no "deny" rule in RBAC. If a user has any role allowing an action, they can perform it, even if another role seems to deny it. *Why it happens:* Misunderstanding RBAC semantics. *How to avoid:* Design roles to be mutually exclusive or use external policy engines like OPA if you need deny rules. *Recovery:* Review all bindings for the user and remove unwanted roles. **Pitfall 4: Mismatching API groups.** For example, granting `deployments` in the core API group instead of `apps` will not work. *Why it happens:* Confusion about resource groups. *How to avoid:* Use `kubectl api-resources` to find the correct group for a resource. *Recovery:* Update the role with the correct `apiGroups` field. **Pitfall 5: Ignoring namespace scoping.** Creating a RoleBinding instead of a ClusterRoleBinding for cluster-scoped resources or vice versa. *Why it happens:* Not checking whether the resource is namespaced or cluster-scoped. *How to avoid:* Use `kubectl api-resources --namespaced=true/false` to determine scope. *Recovery:* Convert the binding type appropriately. ## Conclusion Troubleshooting Kubernetes RBAC requires a systematic approach: identify the subject, inspect current permissions, test with `kubectl auth can-i`, make minimal changes, and verify. By following the practices in this guide, you can resolve access issues quickly while maintaining a secure, least-privilege posture. Always document changes and review RBAC policies regularly to prevent future incidents. As a next step, run `kubectl auth can-i --list` for your own user and for any service accounts in your cluster to identify overly broad permissions. Then, choose one service account and tighten its permissions using the safe configuration path described above. Record the current state, apply a minimal role update, and verify with a test command. Remember: the goal is not just to fix the immediate error but to ensure long-term security and auditability.