## Intro

Kubernetes authorization failures are usually silent until a developer cannot deploy, a service account cannot read a ConfigMap, or an attacker exploits an over-privileged binding. This article provides a production operations checklist for Kubernetes authorization with practical examples that move you from an observed problem to a verified result. It is written for developers, DevOps consultants, and technical startup teams who operate clusters and need to manage Role-Based Access Control (RBAC), admission policies, and API access safely.

The checklist connects Kubernetes authorization operations, checklists, best practices, and maintenance to concrete commands, expected output, failure signals, and recovery decisions. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify every result, and document recovery paths before an incident occurs.

Throughout this guide, we use a running example: a startup running a production cluster on Kubernetes 1.28 with a web application and a backend API. Namespaces include `web`, `api`, and `infra`. We will use a service account named `web-sa` and a developer user `alice`. All commands assume `kubectl` is configured with appropriate cluster-admin rights for inspection, but we will show how to operate with least privilege where possible.

## Version and Environment Inventory

Before changing authorization settings, you must know exactly what is running. Authorization behavior changes between Kubernetes versions, and misconfigured RBAC can be masked by outdated API servers or incompatible admission controllers.

### Prerequisites and read-only observation

The first step in any authorization operation is to record the cluster version, the API server flags related to authorization, and the current RBAC state. Run these commands and save the output with timestamps:

```bash
kubectl version --short
kubectl get --raw /metrics | grep apiserver_authorization
```

Expected output for a healthy cluster on Kubernetes 1.28:

```
Client Version: v1.28.0
Server Version: v1.28.0
apiserver_authorization_decision_total{result="allow",verb="get",resource="pods"} 42
apiserver_authorization_decision_total{result="deny",verb="list",resource="secrets"} 2
```

If the server metric shows unexpected denials for read operations on common resources, that is a signal to audit RBAC rules. Next, inspect the API server authorization mode. This is usually configured in the kube-apiserver manifest or systemd unit.

```bash
kubectl -n kube-system get pod -l component=kube-apiserver -o yaml | grep -A5 'command:'
```

Look for `--authorization-mode=RBAC,Node` or similar. If `AlwaysAllow` is present, authorization is effectively disabled for all requests except those handled by admission control. In production that is a critical finding.

### Smallest justified change

Do not change authorization modes without a maintenance window and rollback plan. If you need to enable RBAC on a cluster that was using `AlwaysAllow`, first create a comprehensive set of roles and bindings in a staging environment, verify them against a copy of production traffic, then switch modes with the API server restart.

For example, to verify that RBAC is active and enforced, run:

```bash
kubectl auth can-i list pods --as system:serviceaccount:web:web-sa -n web
```

Expected output for a properly restricted service account that lacks list permission:

```
no
```

If the output is `yes` and you expected `no`, review the bindings in the `web` namespace.

## Safe Configuration Path

Authorization configuration changes should follow a path from local testing to production with verification at each step. The key principle is to separate observation from intervention: capture current state first, protect credentials, and change one scoped item only when its blast radius and recovery path are understood.

### Local test with a single manifest

Start with a minimal Role and RoleBinding in a dedicated test namespace. Suppose you want to grant the `web-sa` service account permission to read ConfigMaps in the `web` namespace but nothing else. Create the following manifest file `web-configmap-reader.yaml`:

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: web
  name: configmap-reader
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: web
  name: web-sa-configmap-reader
subjects:
- kind: ServiceAccount
  name: web-sa
  namespace: web
roleRef:
  kind: Role
  name: configmap-reader
  apiGroup: rbac.authorization.k8s.io
```

Apply it to a local test cluster (such as kind or minikube):

```bash
kubectl apply -f web-configmap-reader.yaml --dry-run=client -o yaml
kubectl apply -f web-configmap-reader.yaml
```

Then verify the permission with `auth can-i`:

```bash
kubectl auth can-i get configmaps --as system:serviceaccount:web:web-sa -n web
```

Expected: `yes`.

```bash
kubectl auth can-i get secrets --as system:serviceaccount:web:web-sa -n web
```

Expected: `no`.

This validates that the rule is scoped correctly before moving to a shared environment.

### Version-controlled rollout

Once the local test passes, store the manifest in Git with a review process. Use a tool like Kustomize or Helm to manage the resource. For example, a Kustomize overlay for production might adjust the namespace or add labels. Apply to a staging cluster that mirrors production RBAC and run a series of positive and negative tests.

After staging verification, apply to production with a recorded command and expected output:

```bash
kubectl apply -f deploy/prod/web-configmap-reader.yaml
```

Expected output:

```
role.rbac.authorization.k8s.io/configmap-reader unchanged
rolebinding.rbac.authorization.k8s.io/web-sa-configmap-reader configured
```

Then immediately verify the service account can list ConfigMaps and still cannot list Secrets:

```bash
kubectl auth can-i list configmaps --as system:serviceaccount:web:web-sa -n web
kubectl auth can-i list secrets --as system:serviceaccount:web:web-sa -n web
```

Expected: `yes` then `no`.

If the second command returns `yes`, roll back the binding immediately:

```bash
kubectl delete rolebinding web-sa-configmap-reader -n web
```

## Verification and Diagnostics

Authorization issues often surface as confusing API errors. A pod may fail with a `Forbidden` message, or a controller may stop reconciling. The key is to collect evidence before changing anything.

### Diagnosing a denied request

The first step is to reproduce the denial from the perspective of the affected identity. If a developer reports they cannot deploy, ask them to run:

```bash
kubectl auth can-i create deployments --as alice -n api
```

If the output is `no`, inspect Alice's roles and bindings:

```bash
kubectl get rolebindings,clusterrolebindings -o wide | grep alice
kubectl describe clusterrolebinding alice-api-binding
```

This shows the roles assigned. Then inspect the role rules:

```bash
kubectl describe clusterrole alice-api-role
```

Expected output for a role that grants create on deployments:

```
Name:         alice-api-role
Labels:       <none>
Annotations:  <none>
PolicyRule:
  Resources         Non-Resource URLs  Resource Names  Verbs
  ---------         -----------------  --------------  -----
  deployments.apps  []                 []              [create]
```

If the verb `create` is missing, that is the root cause. You can also check the API audit log if enabled. Look for entries with `"verb":"create","resource":"deployments","user":"alice","stage":"ResponseComplete","responseStatus":{"code":403}`. This confirms the decision was made at the authorization stage.

### Using kubectl auth reconcile

When you have many roles and bindings, manual inspection is error-prone. Use `kubectl auth reconcile` to compare the desired state in your manifests with the live cluster state. This command is safe because it only adds or updates rules; it does not remove permissions unless you use the `--remove-extra-permissions` flag.

First, generate a baseline of the current RBAC:

```bash
kubectl get roles,rolebindings,clusterroles,clusterrolebindings -A -o yaml > rbac-backup-$(date +%Y%m%d).yaml
```

Then run reconcile with your source of truth files:

```bash
kubectl auth reconcile -f deploy/prod/rbac/
```

Expected output for a cluster with a drift:

```
clusterrole.rbac.authorization.k8s.io/alice-api-role reconciled
rolebinding.rbac.authorization.k8s.io/web-sa-configmap-reader reconciled
```

After reconcile, re-run the `auth can-i` checks to ensure the intended permissions are in effect and that no unintended permissions were added. Always review the diff output before applying with `--remove-extra-permissions`.

## Failure Modes and Recovery

Authorization misconfigurations can cause immediate outages or latent security vulnerabilities. This section covers three common failure modes and step-by-step recovery.

### Failure mode 1: Overly permissive ClusterRoleBinding

**Symptom:** A compromised pod can read all secrets across the cluster.

**Detection:** Run a permission audit using `kubectl auth can-i --list` for a suspicious service account:

```bash
kubectl auth can-i --list --as system:serviceaccount:web:web-sa -n web
```

Expected output includes `list secrets` in all namespaces if the binding is too broad.

**Recovery:** Identify the offending ClusterRoleBinding:

```bash
kubectl get clusterrolebindings -o yaml | grep -B5 -A10 web-sa
```

Remove the binding:

```bash
kubectl delete clusterrolebinding web-sa-cluster-admin
```

Then re-verify:

```bash
kubectl auth can-i list secrets --as system:serviceaccount:web:web-sa -n web
kubectl auth can-i list secrets --as system:serviceaccount:web:web-sa -n api
```

Expected: `no` for both.

Finally, apply a least-privilege RoleBinding as described in the Safe Configuration Path section.

### Failure mode 2: Missing permission causing pod startup failure

**Symptom:** A new pod fails with `Error: configmaps "app-config" is forbidden: User "system:serviceaccount:api:api-sa" cannot get resource "configmaps" in API group "" in the namespace "api".`

**Detection:** The pod events show the error:

```bash
kubectl describe pod api-deployment-7f8c9d5b6-abcde -n api
```

Look for `FailedMount` or `FailedSync` events with the forbidden message.

**Recovery:** Add the missing permission to the `api-sa` service account. Create a Role and RoleBinding exactly as needed, apply, and then restart the deployment:

```bash
kubectl apply -f api-configmap-reader.yaml
kubectl rollout restart deployment/api-deployment -n api
kubectl rollout status deployment/api-deployment -n api
```

Expected output for a successful rollout:

```
deployment "api-deployment" successfully rolled out
```

If the rollout does not succeed, check the pod logs for the previous error type. If a different forbidden error appears, repeat the permission analysis.

### Failure mode 3: Webhook admission controller blocking legitimate requests

**Symptom:** Even with correct RBAC, requests are denied with a message like `admission webhook "validation.example.com" denied the request`.

**Detection:** Check the admission webhook configuration and its associated service:

```bash
kubectl get validatingwebhookconfigurations
kubectl describe validatingwebhookconfiguration validation-example-com
```

Look for failure policies and whether the webhook service is reachable. If the webhook has `failurePolicy: Fail` and the backend service is down, all matching requests are denied.

**Recovery:** If the webhook is not needed for the specific request, you can temporarily patch its `failurePolicy` to `Ignore` after evaluating the security implications:

```bash
kubectl patch validatingwebhookconfiguration validation-example-com --type='json' -p='[{"op": "replace", "path": "/webhooks/0/failurePolicy", "value": "Ignore"}]'
```

Then retry the failing request. If it succeeds, fix the webhook backend service. Once restored, revert the failure policy:

```bash
kubectl patch validatingwebhookconfiguration validation-example-com --type='json' -p='[{"op": "replace", "path": "/webhooks/0/failurePolicy", "value": "Fail"}]'
```

Always document these temporary changes and ensure monitoring alerts on webhook failures.

## Operations Checklist

Use the following checklist before and after any authorization change. Each item includes a command or verification step with expected output.

| # | Checklist item | Command / verification | Expected output |
|---|---|---|---|
| 1 | Record cluster version and authorization mode | `kubectl version --short` and inspect API server flags | Server version v1.28.0; `--authorization-mode=RBAC,Node` |
| 2 | Backup current RBAC state | `kubectl get roles,rolebindings,clusterroles,clusterrolebindings -A -o yaml > rbac-backup-$(date +%Y%m%d).yaml` | File created without errors |
| 3 | Identify affected identity and namespace | `kubectl auth can-i --list --as system:serviceaccount:web:web-sa -n web` | Permission list for the service account |
| 4 | Test change in local cluster with dry-run first | `kubectl apply -f web-configmap-reader.yaml --dry-run=client -o yaml` | YAML output with no errors |
| 5 | Apply change with version control | `kubectl apply -f deploy/prod/web-configmap-reader.yaml` | `rolebinding... configured` |
| 6 | Verify positive permission | `kubectl auth can-i get configmaps --as system:serviceaccount:web:web-sa -n web` | `yes` |
| 7 | Verify negative permission (least privilege) | `kubectl auth can-i get secrets --as system:serviceaccount:web:web-sa -n web` | `no` |
| 8 | Check for unintended permissions drift | `kubectl auth reconcile -f deploy/prod/rbac/ --dry-run=client -o yaml` | Shows differences without applying |
| 9 | Review audit logs if available | `grep 'responseStatus":{"code":403' /var/log/kubernetes/audit/audit.log` | Relevant denied requests |
| 10 | Document the change and rollback plan | Update runbook with `kubectl delete rolebinding web-sa-configmap-reader -n web` as rollback | Runbook created |
| 11 | Monitor after change | `kubectl get events -n web --sort-by='.lastTimestamp'` | No new Forbidden events |

This checklist ensures that every authorization change is observable, reversible, and aligned with least privilege.

## Conclusion

Kubernetes authorization operations require a deliberate, version-scoped approach. The checklist in this article helps you verify the environment, test changes safely, diagnose denials, recover from failures, and maintain least privilege. Each step includes concrete commands and expected outputs so that operators can follow along and adapt to their own clusters.

Start with one low-risk improvement: audit a single service account using `kubectl auth can-i --list`, identify any over-permissions, and replace broad bindings with narrowly scoped roles. Record the current state, apply the change in a staging cluster first, verify both allowed and denied operations, and document the rollback command.

A reliable authorization workflow makes failures visible, protects sensitive credentials, limits changes to the intended resource, and defines recovery verification before an incident forces a rushed decision. By integrating these practices into your operations, you reduce the risk of both downtime and security breaches.