## Intro

Running Kubernetes RBAC in production is different from enabling it in a test cluster. Access mistakes can lock out workloads, grant more privilege than intended, or leave stale permissions that survive an incident review. This guide is a practical operations checklist for working with Kubernetes RBAC in production. It covers version and environment inventory, safe configuration changes, verification and diagnostics, failure modes and recovery, and a day-to-day operations checklist.

Each section includes concrete commands, expected output, failure signals, and recovery steps. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached.

## Version and Environment Inventory

Before touching RBAC objects, establish the cluster version, the API availability, and the current state of RBAC resources. This inventory creates a baseline and helps you notice drift later.

Start with the cluster version:

kubectl version --short 
 Expected output example:

Client Version: v1.29.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.2 
 If the client and server major versions differ by more than one minor release, some RBAC fields may be unavailable or behave differently. For example, the kubernetes.io/bootstrapping annotation was deprecated but may still appear in clusters upgraded from older versions.

Check that the RBAC API is enabled (it is by default in all supported Kubernetes versions) by listing roles in a namespace:

kubectl get roles -n kube-system 
 If you see No resources found , the namespace may have no roles, or you may lack permission to list roles. Verify your own permissions with:

kubectl auth can-i list roles -n kube-system 
 This should return yes for a cluster-admin or a user with appropriate rights.

Next, inventory existing roles, role bindings, cluster roles, and cluster role bindings. Use read-only commands and save the output to files for later comparison:

kubectl get roles,rolebindings --all-namespaces -o yaml > rbac-namespaced-backup.yaml
kubectl get clusterroles,clusterrolebindings -o yaml > rbac-cluster-backup.yaml 
 For a human-readable summary, use:

kubectl get rolebindings,clusterrolebindings --all-namespaces 
 Look for bindings that grant access to service accounts outside their own namespace, especially in kube-system , kube-public , and any default namespace. These are common sources of unexpected privilege.

Check the Kubernetes version's RBAC release notes for any changes. For instance, in Kubernetes 1.24, the legacy serviceAccountToken projection was fully replaced by BoundServiceAccountTokenVolume . If your workloads still use legacy tokens, RBAC decisions may be based on an old token subject.

Finally, record the current state with timestamps:

date -u +"Inventory run: %Y-%m-%dT%H:%M:%SZ"
kubectl get clusterroles --show-labels 
 Keep these backups in a secure location, not in the cluster itself.

<style>
.eno-quiz-widget{margin:2rem 0;padding:1.5rem;border-radius:12px;background:var(--eno-surface-lowest,#f7f7f8);border:1px solid var(--eno-border-soft,#e2e2e6);font-family:var(--eno-font-body,Inter,sans-serif)}
.eno-quiz-widget .eno-quiz-kicker{font-family:var(--eno-font-label,"JetBrains Mono",monospace);font-size:.75rem;letter-spacing:.05em;text-transform:uppercase;color:var(--eno-text-muted,#5f5f68);margin:0 0 .5rem}
.eno-quiz-widget .eno-quiz-question{font-family:var(--eno-font-heading,"Hanken Grotesk",sans-serif);font-size:1.0625rem;font-weight:600;margin:0 0 1rem;color:var(--eno-text-strong,#1a1a1f)}
.eno-quiz-widget .eno-quiz-options{list-style:none;margin:0;padding:0;display:flex;flex-direction:column;gap:.5rem}
.eno-quiz-widget .eno-quiz-option{display:block;width:100%;min-height:44px;text-align:left;padding:.625rem .875rem;border-radius:8px;border:1.5px solid var(--eno-border-soft,#e2e2e6);background:#fff;font-size:.9375rem;cursor:pointer;transition:border-color var(--eno-motion-base,180ms ease-out),background var(--eno-motion-base,180ms ease-out)}
.eno-quiz-widget .eno-quiz-option:hover{border-color:var(--eno-primary,#0059bb)}
.eno-quiz-widget .eno-quiz-option:focus-visible{outline:none;box-shadow:0 0 0 3px rgb(0 89 187 / 0.15)}
.eno-quiz-widget .eno-quiz-option[aria-pressed="true"]{border-color:var(--eno-primary,#0059bb);background:rgb(0 89 187 / 0.06)}
.eno-quiz-widget .eno-quiz-option[data-correct="true"].eno-quiz-revealed{border-color:var(--eno-success,#17803a);background:rgb(23 128 58 / 0.08)}
.eno-quiz-widget .eno-quiz-option[data-correct="false"].eno-quiz-revealed.eno-quiz-was-selected{border-color:var(--eno-error,#ba1a1a);background:var(--eno-error-soft,#ffdad6)}
.eno-quiz-widget .eno-quiz-option-icon{display:inline-block;width:1.1em;margin-right:.4em;font-weight:700}
.eno-quiz-widget .eno-quiz-submit{margin-top:1rem;min-height:44px;padding:.5rem 1.25rem;border-radius:8px;border:none;background:var(--eno-primary,#0059bb);color:#fff;font-weight:600;font-size:.9375rem;cursor:pointer;transition:opacity var(--eno-motion-base,180ms ease-out)}
.eno-quiz-widget .eno-quiz-submit:disabled{opacity:.5;cursor:not-allowed}
.eno-quiz-widget .eno-quiz-submit:focus-visible{outline:none;box-shadow:0 0 0 3px rgb(0 89 187 / 0.15)}
.eno-quiz-widget .eno-quiz-explanation{margin-top:1rem;padding:.875rem 1rem;border-radius:8px;font-size:.9375rem;line-height:1.5;display:none}
.eno-quiz-widget .eno-quiz-explanation.eno-quiz-visible{display:block}
.eno-quiz-widget .eno-quiz-explanation.eno-quiz-correct{background:rgb(23 128 58 / 0.08);color:var(--eno-success,#17803a)}
.eno-quiz-widget .eno-quiz-explanation.eno-quiz-incorrect{background:var(--eno-error-soft,#ffdad6);color:var(--eno-error,#ba1a1a)}
@media (prefers-reduced-motion: reduce){.eno-quiz-widget *{transition:none!important}}
</style><div class="eno-quiz-widget" role="group" aria-label="Quick check question">
Quick check 1 of 2

What is a potential risk when a user can create deployments indirectly?

- They might delete pods directly
- They can create pods indirectly through deployments
- They can access nodes directly
- They can modify cluster roles directly

<button type="button" class="eno-quiz-submit" disabled>Submit</button>
<div class="eno-quiz-explanation">The passage states that a user may not be able to create pods directly, but allowing them to create a deployment, which creates pods on their behalf, will let them create those pods indirectly.</div>
</div>
<script>
document.addEventListener('DOMContentLoaded', function () {
  document.querySelectorAll('.eno-quiz-widget:not([data-eno-quiz-bound])').forEach(function (widget) {
    widget.setAttribute('data-eno-quiz-bound', '1');
    var options = Array.prototype.slice.call(widget.querySelectorAll('.eno-quiz-option'));
    var submitBtn = widget.querySelector('.eno-quiz-submit');
    var explanation = widget.querySelector('.eno-quiz-explanation');
    var selected = null;
    options.forEach(function (opt) {
      opt.addEventListener('click', function () {
        if (widget.hasAttribute('data-eno-quiz-answered')) return;
        options.forEach(function (o) { o.setAttribute('aria-pressed', 'false'); });
        opt.setAttribute('aria-pressed', 'true');
        selected = opt;
        submitBtn.disabled = false;
      });
    });
    submitBtn.addEventListener('click', function () {
      if (!selected || widget.hasAttribute('data-eno-quiz-answered')) return;
      widget.setAttribute('data-eno-quiz-answered', '1');
      submitBtn.disabled = true;
      var correct = selected.getAttribute('data-correct') === 'true';
      options.forEach(function (o) {
        o.classList.add('eno-quiz-revealed');
        if (o === selected) o.classList.add('eno-quiz-was-selected');
        var icon = o.querySelector('.eno-quiz-option-icon');
        if (o.getAttribute('data-correct') === 'true') icon.textContent = '\u2713';
        else if (o === selected) icon.textContent = '\u2717';
      });
      explanation.classList.add('eno-quiz-visible', correct ? 'eno-quiz-correct' : 'eno-quiz-incorrect');
    });
  });
});
</script>

## Safe Configuration Path

RBAC changes are about granting or revoking access. A misstep can break a running workload or open a security hole. The safe configuration path is a sequence of observe, plan, apply, verify, and document.

### Step 1: Read the current configuration

Before changing a role, see what privileges it already has:

kubectl get role my-app-role -n my-namespace -o yaml 
 If the role is bound to multiple subjects, list all bindings:

kubectl get rolebinding -n my-namespace -o yaml | grep -B5 -A5 "name: my-app-role" 

### Step 2: Make the smallest change

 Suppose the role my-app-role currently grants get , list , and watch on deployments, but a new feature needs create on pods. Instead of granting * on all resources, add only the required permission.

Create a new role file or patch the existing one. Using a GitOps workflow is recommended. For a quick patch:

kubectl patch role my-app-role -n my-namespace --type='json' -p='[{"op": "add", "path": "/rules/0/verbs", "value": ["get", "list", "watch", "create"]}]' 
 Important : The above adds create to the first rule. If the first rule might not be the deployments rule, inspect the YAML first and adjust the path accordingly. A safer approach is to edit the manifest locally and apply it:

# my-app-role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
 name: my-app-role
 namespace: my-namespace
rules:
- apiGroups: ["apps"]
 resources: ["deployments"]
 verbs: ["get", "list", "watch"]
- apiGroups: [""]
 resources: ["pods"]
 verbs: ["create"] 
 Apply it:

kubectl apply -f my-app-role.yaml 

### Step 3: Use dry-run to preview changes

 Before applying any change, run a dry-run to see what would happen:

kubectl apply -f my-app-role.yaml --dry-run=client -o yaml 
 This prints the object that would be sent to the API server. For more accurate validation, use --dry-run=server if your cluster supports it:

kubectl apply -f my-app-role.yaml --dry-run=server 
 The server will validate the object against the API schema and admission webhooks without persisting it.

### Step 4: Apply and verify immediately

After applying, check the role:

kubectl get role my-app-role -n my-namespace -o yaml 
 Use kubectl auth can-i to test access as the subject. If the subject is a service account named my-app-sa in my-namespace , run:

kubectl auth can-i create pods --as=system:serviceaccount:my-namespace:my-app-sa -n my-namespace 
 This should return yes .

### Step 5: Document the change

Record the change in your team's change log or incident management system. Include the date, the reason, the exact diff, and the rollback plan.

## Verification and Diagnostics

Verification goes beyond checking that the object exists. You need to ensure that the RBAC decision matches your intent and that there are no unintended consequences.

### Verify effective permissions with auth can-i

The most direct way is to impersonate the subject:

kubectl auth can-i --list --as=system:serviceaccount:my-namespace:my-app-sa -n my-namespace 
 This lists all permissions the service account has in that namespace. Look for verbs you did not intend to grant.

For a specific permission:

kubectl auth can-i update deployments/scale --as=system:serviceaccount:my-namespace:my-app-sa -n my-namespace 
 If it returns no , check why.

### Trace role bindings

To see all subjects bound to a role:

kubectl get rolebinding -n my-namespace -o json | jq '.items[] | select(.roleRef.name=="my-app-role") | .subjects' 
 This shows the service accounts, users, or groups that receive the role's permissions.

### Check API server audit logs

If audit logging is enabled (as it should be in production for security-critical workloads), query for denied requests:

# Example audit log line for a denied request:
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"RequestResponse","auditID":"1234","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/my-namespace/pods","verb":"create","user":{"username":"system:serviceaccount:my-namespace:my-app-sa","groups":["system:serviceaccounts","system:serviceaccounts:my-namespace"]},"responseStatus":{"code":403,"reason":"Forbidden"}} 
 Use your log aggregation tool to filter for 403 responses and correlate with RBAC changes.

### Use RBAC policy linting tools

Tools like kubectl-who-can (from Aqua Security) or rakkess can help you understand who can do what:

kubectl who-can create pods -n my-namespace 
 Expected output example:

ROLE NAMESPACE SUBJECT
my-app-role my-namespace ServiceAccount/my-app-sa 
 This confirms that only the intended subject has that permission.

### Verify with a real test pod

For critical permissions, create a temporary pod using the service account and attempt the action:

kubectl run test-pod --image=bitnami/kubectl:latest --restart=Never -n my-namespace --overrides='{"spec":{"serviceAccountName":"my-app-sa"}}' --command -- sleep 3600
kubectl exec -it test-pod -n my-namespace -- kubectl auth can-i create pods -n my-namespace 
 If the output is yes , the permission is effective. Clean up the test pod:

kubectl delete pod test-pod -n my-namespace 

 Quick check 2 of 2

What is the recommended combination of authorizers and admission plugin for API authorization?

- ABAC and Node together with NodeRestriction
- RBAC and ABAC together with PodSecurity
- Node and RBAC together with NodeRestriction
- RBAC alone with no admission plugin

<button type="button" class="eno-quiz-submit" disabled>Submit</button>
<div class="eno-quiz-explanation">The passage recommends using the Node and RBAC authorizers together, in combination with the NodeRestriction admission plugin.</div>
</div>
<script>
document.addEventListener('DOMContentLoaded', function () {
  document.querySelectorAll('.eno-quiz-widget:not([data-eno-quiz-bound])').forEach(function (widget) {
    widget.setAttribute('data-eno-quiz-bound', '1');
    var options = Array.prototype.slice.call(widget.querySelectorAll('.eno-quiz-option'));
    var submitBtn = widget.querySelector('.eno-quiz-submit');
    var explanation = widget.querySelector('.eno-quiz-explanation');
    var selected = null;
    options.forEach(function (opt) {
      opt.addEventListener('click', function () {
        if (widget.hasAttribute('data-eno-quiz-answered')) return;
        options.forEach(function (o) { o.setAttribute('aria-pressed', 'false'); });
        opt.setAttribute('aria-pressed', 'true');
        selected = opt;
        submitBtn.disabled = false;
      });
    });
    submitBtn.addEventListener('click', function () {
      if (!selected || widget.hasAttribute('data-eno-quiz-answered')) return;
      widget.setAttribute('data-eno-quiz-answered', '1');
      submitBtn.disabled = true;
      var correct = selected.getAttribute('data-correct') === 'true';
      options.forEach(function (o) {
        o.classList.add('eno-quiz-revealed');
        if (o === selected) o.classList.add('eno-quiz-was-selected');
        var icon = o.querySelector('.eno-quiz-option-icon');
        if (o.getAttribute('data-correct') === 'true') icon.textContent = '\u2713';
        else if (o === selected) icon.textContent = '\u2717';
      });
      explanation.classList.add('eno-quiz-visible', correct ? 'eno-quiz-correct' : 'eno-quiz-incorrect');
    });
  });
});
</script>

## Failure Modes and Recovery

RBAC failures can cause application outages or security incidents. Here are common failure modes and recovery steps.

### Failure: Workload gets 403 Forbidden

Symptom : Pods crash or applications log 403 Forbidden errors when accessing the Kubernetes API.

Diagnosis : Check the pod logs:

kubectl logs my-app-pod -n my-namespace --tail=50 
 Look for lines like:

Error: GET https://kubernetes.default.svc:443/api/v1/namespaces/my-namespace/pods: 403 Forbidden 
 Then check the service account permissions:

kubectl auth can-i --list --as=system:serviceaccount:my-namespace:my-app-sa -n my-namespace 
 Recovery : Identify the missing verb/resource from the error and add it to the role bound to the service account. Use the safe configuration path above. After applying, restart the pod or wait for the client to retry.

### Failure: User cannot perform an action

Symptom : A developer reports kubectl get pods returns Error from server (Forbidden): pods is forbidden...

Diagnosis : Verify the user's identity and group memberships:

kubectl auth whoami 
 If this returns the user, check their permissions with:

kubectl auth can-i list pods --as=developer-sara -n dev 
 Also check the role bindings for the user:

kubectl get rolebindings,clusterrolebindings -o json | jq '.items[] | select(.subjects[]?.name=="developer-sara")' 
 Recovery : If the user should have the permission, create or update a role binding. If they should not, deny the request and explain the policy.

### Failure: Accidental privilege escalation

Symptom : A role or cluster role was modified to include * verbs or resources, or a role binding was created for an unintended subject.

Diagnosis : Use your GitOps audit trail or cluster audit logs to find the change. Compare with the backup from the inventory step:

diff -u rbac-cluster-backup.yaml <(kubectl get clusterroles,clusterrolebindings -o yaml) 
 Recovery : Revert the change by applying the previous manifest from version control:

kubectl apply -f known-good-rolebinding.yaml 
 If the cluster was compromised, you may need to revoke all tokens and rotate credentials. Follow your incident response plan.

### Failure: Stale role bindings after namespace deletion

Symptom : Namespace was deleted, but cluster role bindings referencing service accounts in that namespace persist.

Diagnosis : List cluster role bindings and look for subjects with non-existent namespaces:

kubectl get clusterrolebindings -o json | jq '.items[] | select(.subjects[]?.namespace != null and (.subjects[]?.namespace | IN( ($kubectl get namespaces -o json | jq -r '.items[].metadata.name') ) | not))' 
 This is a complex jq query; a simpler approach is to manually review suspicious bindings.

Recovery : Delete or update the stale binding:

kubectl delete clusterrolebinding stale-binding 

### Failure: Missing RBAC permission breaks upgrade

 Symptom : After upgrading a chart or operator, it fails with permission errors.

Diagnosis : Check the operator's logs and compare required permissions with the installed role. Many operators have a documented set of cluster roles; see their installation manifest.

Recovery : Update the operator's cluster role to include the new permissions. If using a chart, re-run helm upgrade with the updated values that specify the required RBAC.

## Operations Checklist

Use this checklist regularly (weekly or after any change) to maintain RBAC hygiene.

### 1. Inventory roles and bindings

Run:

kubectl get roles,rolebindings,clusterroles,clusterrolebindings --all-namespaces -o yaml > rbac-full-backup-$(date +%Y%m%d).yaml 
 Store the backup securely and diff against the previous week to spot changes.

### 2. Check for overly permissive roles

Look for any role or cluster role with * in verbs or resources:

kubectl get clusterroles -o json | jq '.items[] | select(.rules[]?.verbs[]? == "*" or .rules[]?.resources[]? == "*") | .metadata.name' 
 Review each hit and justify the need. Replace * with explicit lists where possible.

### 3. Validate service account usage

Ensure every service account is bound to at least one role and has a clear purpose:

kubectl get serviceaccounts --all-namespaces 
 Check for unused service accounts:

kubectl get pods --all-namespaces -o json | jq -r '.items[] | .spec.serviceAccountName' | sort -u 
 Compare the two lists to find service accounts not used by any pod.

### 4. Test critical permissions

For key service accounts (CI/CD, monitoring, gitops), run:

kubectl auth can-i --list --as=system:serviceaccount:ci-namespace:ci-sa 
 Verify that the permissions match the intended workload.

### 5. Audit bindings for external users

If you use OIDC or certificate-based users, list all cluster role bindings with user subjects:

kubectl get clusterrolebindings -o json | jq '.items[] | select(.subjects[]?.kind == "User")' 
 Ensure each binding is still necessary.

### 6. Check for default and bootstrapping roles

Some clusters have system:basic-user or system:discovery bindings that may grant more than desired. Review:

kubectl get clusterrolebindings | grep -E 'system:basic-user|system:discovery' 

### 7. Verify RBAC with policy tools

 Run a policy tool like kube-rbac-proxy , kubescape , or OPA Gatekeeper policies against RBAC configurations. For example, with kubescape :

kubescape scan framework nsa --include-namespaces kube-system 

### 8. Document exceptions and approvals

 Maintain a register of RBAC exceptions. In the register, include:

<div class="my-stack-md overflow-x-auto">
<table class="min-w-[42rem] border-collapse text-left">
<thead><tr><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Item</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Approver</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Date</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Expiry</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Link to approval</th></tr></thead>
<tbody><tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant"><code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">developer-sara</code> added to <code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">view</code> cluster role for on-call debugging</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Priya Shah, Engineering Lead</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">2025-05-14</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">2025-05-21</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">https://tickets.example.com/INC-1234</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">CI service account granted <code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">create</code> on <code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">deployments</code> in <code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">staging</code></td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Marcus Chen, DevOps Manager</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">2025-05-10</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Permanent with quarterly review</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">https://wiki.example.com/rbac/ci-staging</td></tr></tbody>
</table>
</div>

## Conclusion

Kubernetes RBAC in production requires ongoing attention. The checklist in this article gives you a structured way to inventory permissions, make safe changes, verify outcomes, and recover from failures. The key principles are:

- Always observe and back up before changing RBAC objects.

- Make the smallest change that meets the requirement.

- Use dry-run and impersonation to verify before and after.

- Keep an audit trail and schedule regular reviews.

- Have a rollback plan for each change.

By following these practices, you can maintain a secure and operational RBAC posture. Start with a low-risk verification: run kubectl auth can-i --list --as=system:serviceaccount:default:default in a non-production namespace to see what the default service account can do. Then expand your inventory and fix any surprises.

A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision.