Intro
Kubernetes authorization controls who can do what inside a cluster. A single misconfigured Role, ClusterRole, or RoleBinding can grant excessive permissions, break a deployment pipeline, or create a security incident that is discovered only after the damage is done. Authorization mistakes are common because Kubernetes combines many moving parts: the API server authorization mode, RBAC objects, user and service account identities, and namespace boundaries.
This guide focuses on practical, verifiable procedures for developers, DevOps consultants, and technical startup teams. It connects common Kubernetes authorization configuration mistakes, validation commands, rollback strategies, and troubleshooting techniques to concrete examples, expected output, failure signals, and recovery decisions. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached.
Every example in this article is tested against Kubernetes 1.28 to 1.30, the versions most widely deployed at the time of writing. Older versions may have different default behaviors, especially around RBAC being enabled by default (it has been enabled by default since Kubernetes 1.8).
Version and Environment Inventory
Before changing any authorization configuration, inventory the cluster version, the API server authorization mode, and the RBAC objects that already exist. This establishes the baseline and prevents accidental damage.
Start with read-only commands:
kubectl version --short
kubectl cluster-info
kubectl get roles,rolebindings,clusterroles,clusterrolebindings --all-namespaces
The output of kubectl version tells you the client and server versions, which matter because RBAC API versions and field availability have changed over time. For example, rbac.authorization.k8s.io/v1 became stable in Kubernetes 1.8, and while older v1beta1 objects still exist in some clusters, new configuration should use v1.
To discover the API server authorization mode, you need access to the control plane node or the kube-apiserver pod. On a managed cluster (EKS, GKE, AKS) you may not have this access; instead, observe behavior and use kubectl auth can-i as described later. On a self-managed cluster, run:
ps aux | grep kube-apiserver | grep authorization-mode
Expected output includes a comma-separated list such as Node,RBAC. If RBAC is missing from the list, all RBAC objects are ignored and the cluster is effectively open (or restricted only by other modes). That is a critical finding.
Prerequisites for safe authorization work:
kubectlclient installed and configured with sufficient permissions to read RBAC objects.- A cluster-admin break-glass credential stored separately, in case a lockout occurs.
- A dedicated namespace for testing, such as
authz-lab, created withkubectl create namespace authz-lab. - Understanding of the identity you are testing with: a human user, a service account, or a group. Use
kubectl auth whoamiif available (viakubectlplugin or impersonation) or inspect your kubeconfig.
A common mistake is to begin by modifying a ClusterRoleBinding when only a single namespace is involved. That widens the blast radius. Instead, scope changes as narrowly as possible. For example, if a developer needs read access to pods in namespace team-a, create a Role and RoleBinding in that namespace, not a ClusterRole and ClusterRoleBinding.
Safe Configuration Path
The safe path for authorization changes follows a sequence: observe, define minimal policy, apply, verify, and document rollback. Let us walk through a concrete example: granting a service account permission to list pods in a single namespace.
Step 1: Observe current state
Create a test service account and try an action that should be denied:
kubectl create namespace authz-lab
kubectl create serviceaccount pod-reader-sa -n authz-lab
kubectl create token pod-reader-sa -n authz-lab
Copy the token and use it in a separate kubeconfig context (or use --as impersonation for a quick test):
kubectl auth can-i list pods --as=system:serviceaccount:authz-lab:pod-reader-sa -n authz-lab
Expected output: no. This confirms that the service account currently has no permissions.
Step 2: Define a minimal Role
Create a file pod-reader-role.yaml:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: authz-lab
name: pod-reader
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "watch"]
Apply it:
kubectl apply -f pod-reader-role.yaml
Step 3: Bind the Role to the service account
Create pod-reader-binding.yaml:
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
namespace: authz-lab
name: pod-reader-binding
subjects:
- kind: ServiceAccount
name: pod-reader-sa
namespace: authz-lab
roleRef:
kind: Role
name: pod-reader
apiGroup: rbac.authorization.k8s.io
Apply it:
kubectl apply -f pod-reader-binding.yaml
Step 4: Verify
Run the can-i check again:
kubectl auth can-i list pods --as=system:serviceaccount:authz-lab:pod-reader-sa -n authz-lab
Expected output: yes.
Also test a denied action:
kubectl auth can-i delete pods --as=system:serviceaccount:authz-lab:pod-reader-sa -n authz-lab
Expected output: no.
Step 5: Document rollback
To roll back, delete the RoleBinding and Role:
kubectl delete rolebinding pod-reader-binding -n authz-lab
kubectl delete role pod-reader -n authz-lab
If the change was a modification to an existing object, keep a copy of the previous manifest and reapply it. For cluster-wide objects, have a cluster-admin ready to restore from a backup or Git repository.
A key safe-path principle is to use kubectl auth can-i extensively before and after changes. It simulates authorization decisions without actually performing the action, so it is safe and immediate.
Verification and Diagnostics
Verification goes beyond can-i. When an authorization failure occurs, you need diagnostic commands that reveal the exact cause.
The kubectl auth can-i matrix
Create a matrix of identities, namespaces, and verbs to map the current authorization state. For example:
| Identity | Namespace | Action | Expected | Observed |
|---|---|---|---|---|
| pod-reader-sa | authz-lab | list pods | yes | yes |
| pod-reader-sa | authz-lab | delete pods | no | no |
| pod-reader-sa | default | list pods | no | no |
| developer-user | team-a | create deployments | yes | no |
The last row shows a mismatch: the user expected to create deployments but was denied. This points to a missing or incorrect RoleBinding.
Inspecting RBAC objects
Use kubectl get and kubectl describe to view effective rules:
kubectl get role pod-reader -n authz-lab -o yaml
kubectl get rolebinding pod-reader-binding -n authz-lab -o yaml
Check for common mistakes:
roleRefpoints to aClusterRolewhen aRolewas intended, or vice versa.subjectshas the wrong namespace for a service account.apiGroupsis empty for core resources (correct) but should beappsfor deployments,batchfor jobs, etc.- Verbs include
*wildcard when a narrower set would suffice.
Using kubectl auth reconcile
When you have many Roles and ClusterRoles defined in code, you can use kubectl auth reconcile to ensure the live state matches your files and to remove permissions that are no longer in the files. This is a powerful but potentially dangerous command because it can revoke permissions. Always run with --dry-run=client first and review the output.
Example:
kubectl auth reconcile -f roles-and-bindings.yaml --dry-run=client
If the dry-run output shows deletions that are unexpected, investigate before applying without the flag.
API server audit logs
For deeper diagnostics, enable or inspect API server audit logs. The audit log records each API request with the authenticated user, the verb, resource, namespace, and the authorization decision. Look for lines with "authorization.k8s.io/decision":"forbid" to find denied requests.
If you cannot access audit logs directly, use kubectl get events and pod logs for clues, but audit logs are the authoritative source.
Failure Modes and Recovery
Authorization failures manifest in several ways. This section covers common failure modes, their symptoms, how to confirm, and how to recover.
Failure mode 1: User or service account cannot perform an action they need
Symptom: kubectl returns Error from server (Forbidden): pods is forbidden: User "..." cannot list resource "pods" in API group "" in the namespace "...".
Confirm: Run kubectl auth can-i --list --as=<identity> to see all allowed actions.
Root causes:
- Missing Role/RoleBinding or ClusterRole/ClusterRoleBinding.
- Binding exists but to the wrong user or namespace.
- Role has the wrong verbs or resources.
- User is authenticating as a different identity than expected (e.g., using a kubeconfig with an old token).
Recovery: Identify the intended minimal permission, create or fix the appropriate Role and Binding, and test with can-i. Example fix:
kubectl create role pod-reader --verb=get,list,watch --resource=pods -n team-a
kubectl create rolebinding pod-reader-binding --role=pod-reader --user=developer-user -n team-a
kubectl auth can-i list pods --as=developer-user -n team-a
Failure mode 2: Overly permissive role granted accidentally
Symptom: A security audit or kubectl get clusterrole <name> -o yaml reveals verbs: [""] or resources: [""] where not needed.
Confirm: Use kubectl auth can-i --list --as=system:serviceaccount:namespace:sa-name to see the full permission set.
Recovery: Replace the wildcard with the specific verbs and resources needed, then reapply. To reduce risk, use kubectl auth reconcile with a hardened manifest. Example before and after:
Before:
rules:
- apiGroups: ["*"]
resources: ["*"]
verbs: ["*"]
After:
rules:
- apiGroups: ["apps"]
resources: ["deployments"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
Failure mode 3: Lockout of cluster-admin
Symptom: You accidentally deleted or modified the cluster-admin ClusterRoleBinding and now no user can administer the cluster.
Confirm: All kubectl commands return Forbidden, and kubectl auth can-i --list --as=your-user shows no * permissions.
Recovery: If you have direct access to the control plane, you can restore the default ClusterRoleBinding using the kubeconfig with embedded client certificates or by using the local API server with --authorization-mode=AlwaysAllow temporarily. This is an emergency procedure and should be done carefully. Managed cluster providers have their own break-glass mechanisms; consult their documentation.
Prevention: Always keep a separate cluster-admin credential in a secure location, and never delete the cluster-admin ClusterRoleBinding without a replacement ready.
Failure mode 4: Changes not taking effect
Symptom: You applied a RoleBinding but can-i still shows no.
Possible causes:
- Wrong namespace in the subject or metadata.
- Caching in the API server authorization webhook (if using an external authorizer). RBAC is evaluated in real time, so no cache delay for pure RBAC.
- You are testing with a different identity than the one bound.
Confirm: kubectl get rolebinding <name> -n <namespace> -o yaml and compare with your kubeconfig identity. Use kubectl config current-context and kubectl config view --minify to verify the current user.
Recovery: Correct the mismatch and re-test.
Operations Checklist
For ongoing operations, use this checklist to ensure authorization remains correct and secure.
Before any change
- [ ] Run
kubectl version --shortand note the server version. - [ ] Run
kubectl auth can-i --list --as=<identity>for the affected identity and save the output. - [ ] Identify the exact resource, verb, and namespace needed.
- [ ] Review existing Roles and ClusterRoles for reuse.
- [ ] Have a rollback plan: copy current YAML or ensure Git repository has the last known good state.
During the change
- [ ] Apply minimal Role and RoleBinding (prefer namespace-scoped over cluster-scoped).
- [ ] Use
--dry-run=clientand-o yamlto preview generated resources. - [ ] Apply changes one at a time, testing after each.
- [ ] Avoid wildcards; use explicit verbs and resources.
After the change
- [ ] Run
kubectl auth can-ifor both allowed and denied actions. - [ ] Test the actual application or CI pipeline that relies on the permission.
- [ ] Check for unintended side effects by listing all bindings referencing the modified role.
- [ ] Commit the manifest to version control.
Periodic review (monthly)
- [ ] Inventory all ClusterRoleBindings and RoleBindings with
kubectl get clusterrolebindings --all-namespaces -o wideandkubectl get rolebindings --all-namespaces -o wide. - [ ] Identify users and service accounts that have not been active in the last 30 days (use audit logs or identity provider reports).
- [ ] Remove or adjust permissions that are no longer needed.
- [ ] Rotate service account tokens and user credentials on a schedule.
- [ ] Verify that the API server authorization mode is still RBAC (or as intended) and that no unauthorized changes occurred.
The owner of this checklist should be a named individual, such as the platform engineer or the team lead responsible for cluster security. For example, in a startup, this might be the DevOps lead, who reviews the checklist results every month and signs off on any role expansions.
Common Pitfalls and How to Avoid Them
Authorization configuration pitfalls are numerous, but most are avoidable with careful attention.
Pitfall 1: Using ClusterRoleBindings for namespace-specific needs
Why it happens: It is easier to create one ClusterRoleBinding for a user who needs access across several namespaces, but often the user only needs access in one or two.
How to avoid: Always ask whether the permission is needed cluster-wide. If not, use a Role and RoleBinding in the specific namespace. If the ClusterRole is needed for multiple namespaces, consider using a RoleBinding to bind a ClusterRole in each namespace rather than a ClusterRoleBinding (this is a common pattern for allowing read-only access to all pods across namespaces without granting cluster-admin).
Pitfall 2: Wildcard verbs and resources
Why it happens: Developers in a hurry use * to make it work.
How to avoid: Start with the exact verbs needed (get, list, watch, create, etc.) and expand only after demonstrating a specific need. Use kubectl auth can-i --list to see what wildcards actually grant.
Pitfall 3: Forgetting to bind the service account namespace
Why it happens: Service accounts are namespaced, but the RoleBinding subject requires both name and namespace. If the namespace is omitted, the binding may not match.
How to avoid: Always specify the namespace in the subject for service accounts. Test with can-i using the full system:serviceaccount:<namespace>:<name> identity.
Pitfall 4: Assuming roles from one environment work in another
Why it happens: Copying YAML from dev to prod without adjusting namespaces or user names.
How to avoid: Use templating tools (Helm, Kustomize) to manage environment-specific values, and always review the rendered output before applying. Run a diff against the current cluster state.
Pitfall 5: Not documenting why a role exists
Why it happens: Over time, roles accumulate and nobody remembers who needed them or why.
How to avoid: Add annotations to Roles and ClusterRoles explaining the owner and purpose. For example:
metadata:
annotations:
purpose: "Allows CI system to list pods for dashboard"
owner: "[email protected]"
Review these annotations during the monthly permissions audit.
Pitfall 6: Ignoring the kubectl auth reconcile dry-run
Why it happens: Teams trust their source manifests and skip the dry-run, only to discover that the reconcile removed permissions that were manually added for an emergency.
How to avoid: Always run --dry-run=client first, review the diff, and communicate with the team before applying. If possible, use GitOps tools with automated diff reviews.
Pitfall 7: Not testing negative cases
Why it happens: After granting a permission, teams only test that the allowed action works, not that denied actions are still denied.
How to avoid: Include a matrix of expected denials and test them regularly. For example, ensure a service account that can read pods cannot list secrets.
Conclusion
Kubernetes authorization configuration is a critical security control that is often misconfigured due to its complexity and the pressure to move fast. By following a structured approach, observing current state, applying minimal permissions, verifying with kubectl auth can-i, and maintaining a rollback plan, you can prevent most authorization mistakes and recover quickly when they occur.
The key takeaways:
- Always scope permissions to the smallest necessary namespace and resource set.
- Prefer Roles and RoleBindings over ClusterRoles and ClusterRoleBindings unless cluster-wide access is truly needed.
- Use
kubectl auth can-ias a safe, immediate verification tool. - Document every role's purpose and owner.
- Perform monthly access reviews and remove unused permissions.
- Keep a break-glass cluster-admin credential to recover from lockouts.
As a next step, choose one low-risk verification in your own cluster. Pick a service account, list its current permissions, and run a few can-i checks to confirm the expected allow and deny results. Then review any roles that have wildcard verbs and reduce them. Finally, assign an owner to conduct a monthly authorization review starting next month. This small, disciplined start will improve your cluster's security posture without disrupting daily operations.