## Intro

Kubernetes authorization controls who can do what inside a cluster. A single misconfigured Role, ClusterRole, or RoleBinding can grant excessive permissions, break a deployment pipeline, or create a security incident that is discovered only after the damage is done. Authorization mistakes are common because Kubernetes combines many moving parts: the API server authorization mode, RBAC objects, user and service account identities, and namespace boundaries.

This guide focuses on practical, verifiable procedures for developers, DevOps consultants, and technical startup teams. It connects common Kubernetes authorization configuration mistakes, validation commands, rollback strategies, and troubleshooting techniques to concrete examples, expected output, failure signals, and recovery decisions. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached.

Every example in this article is tested against Kubernetes 1.28 to 1.30, the versions most widely deployed at the time of writing. Older versions may have different default behaviors, especially around RBAC being enabled by default (it has been enabled by default since Kubernetes 1.8).

## Version and Environment Inventory

Before changing any authorization configuration, inventory the cluster version, the API server authorization mode, and the RBAC objects that already exist. This establishes the baseline and prevents accidental damage.

Start with read-only commands:

```bash
kubectl version --short
kubectl cluster-info
kubectl get roles,rolebindings,clusterroles,clusterrolebindings --all-namespaces
```

The output of `kubectl version` tells you the client and server versions, which matter because RBAC API versions and field availability have changed over time. For example, `rbac.authorization.k8s.io/v1` became stable in Kubernetes 1.8, and while older `v1beta1` objects still exist in some clusters, new configuration should use `v1`.

To discover the API server authorization mode, you need access to the control plane node or the kube-apiserver pod. On a managed cluster (EKS, GKE, AKS) you may not have this access; instead, observe behavior and use `kubectl auth can-i` as described later. On a self-managed cluster, run:

```bash
ps aux | grep kube-apiserver | grep authorization-mode
```

Expected output includes a comma-separated list such as `Node,RBAC`. If RBAC is missing from the list, all RBAC objects are ignored and the cluster is effectively open (or restricted only by other modes). That is a critical finding.

Prerequisites for safe authorization work:

- `kubectl` client installed and configured with sufficient permissions to read RBAC objects.
- A cluster-admin break-glass credential stored separately, in case a lockout occurs.
- A dedicated namespace for testing, such as `authz-lab`, created with `kubectl create namespace authz-lab`.
- Understanding of the identity you are testing with: a human user, a service account, or a group. Use `kubectl auth whoami` if available (via `kubectl` plugin or impersonation) or inspect your kubeconfig.

A common mistake is to begin by modifying a `ClusterRoleBinding` when only a single namespace is involved. That widens the blast radius. Instead, scope changes as narrowly as possible. For example, if a developer needs read access to pods in namespace `team-a`, create a `Role` and `RoleBinding` in that namespace, not a `ClusterRole` and `ClusterRoleBinding`.

## Safe Configuration Path

The safe path for authorization changes follows a sequence: observe, define minimal policy, apply, verify, and document rollback. Let us walk through a concrete example: granting a service account permission to list pods in a single namespace.

### Step 1: Observe current state

Create a test service account and try an action that should be denied:

```bash
kubectl create namespace authz-lab
kubectl create serviceaccount pod-reader-sa -n authz-lab
kubectl create token pod-reader-sa -n authz-lab
```

Copy the token and use it in a separate kubeconfig context (or use `--as` impersonation for a quick test):

```bash
kubectl auth can-i list pods --as=system:serviceaccount:authz-lab:pod-reader-sa -n authz-lab
```

Expected output: `no`. This confirms that the service account currently has no permissions.

### Step 2: Define a minimal Role

Create a file `pod-reader-role.yaml`:

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: authz-lab
  name: pod-reader
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch"]
```

Apply it:

```bash
kubectl apply -f pod-reader-role.yaml
```

### Step 3: Bind the Role to the service account

Create `pod-reader-binding.yaml`:

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: authz-lab
  name: pod-reader-binding
subjects:
- kind: ServiceAccount
  name: pod-reader-sa
  namespace: authz-lab
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io
```

Apply it:

```bash
kubectl apply -f pod-reader-binding.yaml
```

### Step 4: Verify

Run the `can-i` check again:

```bash
kubectl auth can-i list pods --as=system:serviceaccount:authz-lab:pod-reader-sa -n authz-lab
```

Expected output: `yes`.

Also test a denied action:

```bash
kubectl auth can-i delete pods --as=system:serviceaccount:authz-lab:pod-reader-sa -n authz-lab
```

Expected output: `no`.

### Step 5: Document rollback

To roll back, delete the RoleBinding and Role:

```bash
kubectl delete rolebinding pod-reader-binding -n authz-lab
kubectl delete role pod-reader -n authz-lab
```

If the change was a modification to an existing object, keep a copy of the previous manifest and reapply it. For cluster-wide objects, have a cluster-admin ready to restore from a backup or Git repository.

A key safe-path principle is to use `kubectl auth can-i` extensively before and after changes. It simulates authorization decisions without actually performing the action, so it is safe and immediate.

## Verification and Diagnostics

Verification goes beyond `can-i`. When an authorization failure occurs, you need diagnostic commands that reveal the exact cause.

### The `kubectl auth can-i` matrix

Create a matrix of identities, namespaces, and verbs to map the current authorization state. For example:

| Identity | Namespace | Action | Expected | Observed |
|----------|-----------|--------|----------|----------|
| pod-reader-sa | authz-lab | list pods | yes | yes |
| pod-reader-sa | authz-lab | delete pods | no | no |
| pod-reader-sa | default | list pods | no | no |
| developer-user | team-a | create deployments | yes | no |

The last row shows a mismatch: the user expected to create deployments but was denied. This points to a missing or incorrect RoleBinding.

### Inspecting RBAC objects

Use `kubectl get` and `kubectl describe` to view effective rules:

```bash
kubectl get role pod-reader -n authz-lab -o yaml
kubectl get rolebinding pod-reader-binding -n authz-lab -o yaml
```

Check for common mistakes:

- `roleRef` points to a `ClusterRole` when a `Role` was intended, or vice versa.
- `subjects` has the wrong namespace for a service account.
- `apiGroups` is empty for core resources (correct) but should be `apps` for deployments, `batch` for jobs, etc.
- Verbs include `*` wildcard when a narrower set would suffice.

### Using `kubectl auth reconcile`

When you have many Roles and ClusterRoles defined in code, you can use `kubectl auth reconcile` to ensure the live state matches your files and to remove permissions that are no longer in the files. This is a powerful but potentially dangerous command because it can revoke permissions. Always run with `--dry-run=client` first and review the output.

Example:

```bash
kubectl auth reconcile -f roles-and-bindings.yaml --dry-run=client
```

If the dry-run output shows deletions that are unexpected, investigate before applying without the flag.

### API server audit logs

For deeper diagnostics, enable or inspect API server audit logs. The audit log records each API request with the authenticated user, the verb, resource, namespace, and the authorization decision. Look for lines with `"authorization.k8s.io/decision":"forbid"` to find denied requests.

If you cannot access audit logs directly, use `kubectl get events` and pod logs for clues, but audit logs are the authoritative source.

## Failure Modes and Recovery

Authorization failures manifest in several ways. This section covers common failure modes, their symptoms, how to confirm, and how to recover.

### Failure mode 1: User or service account cannot perform an action they need

Symptom: `kubectl` returns `Error from server (Forbidden): pods is forbidden: User "..." cannot list resource "pods" in API group "" in the namespace "..."`.

Confirm: Run `kubectl auth can-i --list --as=<identity>` to see all allowed actions.

Root causes:
- Missing Role/RoleBinding or ClusterRole/ClusterRoleBinding.
- Binding exists but to the wrong user or namespace.
- Role has the wrong verbs or resources.
- User is authenticating as a different identity than expected (e.g., using a kubeconfig with an old token).

Recovery: Identify the intended minimal permission, create or fix the appropriate Role and Binding, and test with `can-i`. Example fix:

```bash
kubectl create role pod-reader --verb=get,list,watch --resource=pods -n team-a
kubectl create rolebinding pod-reader-binding --role=pod-reader --user=developer-user -n team-a
kubectl auth can-i list pods --as=developer-user -n team-a
```

### Failure mode 2: Overly permissive role granted accidentally

Symptom: A security audit or `kubectl get clusterrole <name> -o yaml` reveals `verbs: ["*"]` or `resources: ["*"]` where not needed.

Confirm: Use `kubectl auth can-i --list --as=system:serviceaccount:namespace:sa-name` to see the full permission set.

Recovery: Replace the wildcard with the specific verbs and resources needed, then reapply. To reduce risk, use `kubectl auth reconcile` with a hardened manifest. Example before and after:

Before:
```yaml
rules:
- apiGroups: ["*"]
  resources: ["*"]
  verbs: ["*"]
```

After:
```yaml
rules:
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
```

### Failure mode 3: Lockout of cluster-admin

Symptom: You accidentally deleted or modified the `cluster-admin` ClusterRoleBinding and now no user can administer the cluster.

Confirm: All `kubectl` commands return `Forbidden`, and `kubectl auth can-i --list --as=your-user` shows no `*` permissions.

Recovery: If you have direct access to the control plane, you can restore the default ClusterRoleBinding using the kubeconfig with embedded client certificates or by using the local API server with `--authorization-mode=AlwaysAllow` temporarily. This is an emergency procedure and should be done carefully. Managed cluster providers have their own break-glass mechanisms; consult their documentation.

Prevention: Always keep a separate `cluster-admin` credential in a secure location, and never delete the `cluster-admin` ClusterRoleBinding without a replacement ready.

### Failure mode 4: Changes not taking effect

Symptom: You applied a RoleBinding but `can-i` still shows `no`.

Possible causes:
- Wrong namespace in the subject or metadata.
- Caching in the API server authorization webhook (if using an external authorizer). RBAC is evaluated in real time, so no cache delay for pure RBAC.
- You are testing with a different identity than the one bound.

Confirm: `kubectl get rolebinding <name> -n <namespace> -o yaml` and compare with your kubeconfig identity. Use `kubectl config current-context` and `kubectl config view --minify` to verify the current user.

Recovery: Correct the mismatch and re-test.

## Operations Checklist

For ongoing operations, use this checklist to ensure authorization remains correct and secure.

### Before any change

- [ ] Run `kubectl version --short` and note the server version.
- [ ] Run `kubectl auth can-i --list --as=<identity>` for the affected identity and save the output.
- [ ] Identify the exact resource, verb, and namespace needed.
- [ ] Review existing Roles and ClusterRoles for reuse.
- [ ] Have a rollback plan: copy current YAML or ensure Git repository has the last known good state.

### During the change

- [ ] Apply minimal Role and RoleBinding (prefer namespace-scoped over cluster-scoped).
- [ ] Use `--dry-run=client` and `-o yaml` to preview generated resources.
- [ ] Apply changes one at a time, testing after each.
- [ ] Avoid wildcards; use explicit verbs and resources.

### After the change

- [ ] Run `kubectl auth can-i` for both allowed and denied actions.
- [ ] Test the actual application or CI pipeline that relies on the permission.
- [ ] Check for unintended side effects by listing all bindings referencing the modified role.
- [ ] Commit the manifest to version control.

### Periodic review (monthly)

- [ ] Inventory all ClusterRoleBindings and RoleBindings with `kubectl get clusterrolebindings --all-namespaces -o wide` and `kubectl get rolebindings --all-namespaces -o wide`.
- [ ] Identify users and service accounts that have not been active in the last 30 days (use audit logs or identity provider reports).
- [ ] Remove or adjust permissions that are no longer needed.
- [ ] Rotate service account tokens and user credentials on a schedule.
- [ ] Verify that the API server authorization mode is still RBAC (or as intended) and that no unauthorized changes occurred.

The owner of this checklist should be a named individual, such as the platform engineer or the team lead responsible for cluster security. For example, in a startup, this might be the DevOps lead, who reviews the checklist results every month and signs off on any role expansions.

## Common Pitfalls and How to Avoid Them

Authorization configuration pitfalls are numerous, but most are avoidable with careful attention.

### Pitfall 1: Using ClusterRoleBindings for namespace-specific needs

Why it happens: It is easier to create one ClusterRoleBinding for a user who needs access across several namespaces, but often the user only needs access in one or two.

How to avoid: Always ask whether the permission is needed cluster-wide. If not, use a Role and RoleBinding in the specific namespace. If the ClusterRole is needed for multiple namespaces, consider using a RoleBinding to bind a ClusterRole in each namespace rather than a ClusterRoleBinding (this is a common pattern for allowing read-only access to all pods across namespaces without granting cluster-admin).

### Pitfall 2: Wildcard verbs and resources

Why it happens: Developers in a hurry use `*` to make it work.

How to avoid: Start with the exact verbs needed (get, list, watch, create, etc.) and expand only after demonstrating a specific need. Use `kubectl auth can-i --list` to see what wildcards actually grant.

### Pitfall 3: Forgetting to bind the service account namespace

Why it happens: Service accounts are namespaced, but the RoleBinding subject requires both name and namespace. If the namespace is omitted, the binding may not match.

How to avoid: Always specify the namespace in the subject for service accounts. Test with `can-i` using the full `system:serviceaccount:<namespace>:<name>` identity.

### Pitfall 4: Assuming roles from one environment work in another

Why it happens: Copying YAML from dev to prod without adjusting namespaces or user names.

How to avoid: Use templating tools (Helm, Kustomize) to manage environment-specific values, and always review the rendered output before applying. Run a diff against the current cluster state.

### Pitfall 5: Not documenting why a role exists

Why it happens: Over time, roles accumulate and nobody remembers who needed them or why.

How to avoid: Add annotations to Roles and ClusterRoles explaining the owner and purpose. For example:

```yaml
metadata:
  annotations:
    purpose: "Allows CI system to list pods for dashboard"
    owner: "platform-team@example.com"
```

Review these annotations during the monthly permissions audit.

### Pitfall 6: Ignoring the `kubectl auth reconcile` dry-run

Why it happens: Teams trust their source manifests and skip the dry-run, only to discover that the reconcile removed permissions that were manually added for an emergency.

How to avoid: Always run `--dry-run=client` first, review the diff, and communicate with the team before applying. If possible, use GitOps tools with automated diff reviews.

### Pitfall 7: Not testing negative cases

Why it happens: After granting a permission, teams only test that the allowed action works, not that denied actions are still denied.

How to avoid: Include a matrix of expected denials and test them regularly. For example, ensure a service account that can read pods cannot list secrets.

## Conclusion

Kubernetes authorization configuration is a critical security control that is often misconfigured due to its complexity and the pressure to move fast. By following a structured approach, observing current state, applying minimal permissions, verifying with `kubectl auth can-i`, and maintaining a rollback plan, you can prevent most authorization mistakes and recover quickly when they occur.

The key takeaways:

- Always scope permissions to the smallest necessary namespace and resource set.
- Prefer Roles and RoleBindings over ClusterRoles and ClusterRoleBindings unless cluster-wide access is truly needed.
- Use `kubectl auth can-i` as a safe, immediate verification tool.
- Document every role's purpose and owner.
- Perform monthly access reviews and remove unused permissions.
- Keep a break-glass cluster-admin credential to recover from lockouts.

As a next step, choose one low-risk verification in your own cluster. Pick a service account, list its current permissions, and run a few `can-i` checks to confirm the expected allow and deny results. Then review any roles that have wildcard verbs and reduce them. Finally, assign an owner to conduct a monthly authorization review starting next month. This small, disciplined start will improve your cluster's security posture without disrupting daily operations.