## Intro Kubernetes multi-tenancy is a powerful model, but it introduces a specific class of operational problems: tenant isolation failures, noisy neighbor resource contention, RBAC misconfigurations that lock out legitimate users, and silent quota exhaustion that blocks workloads without a clear signal. Troubleshooting these issues requires more than running generic `kubectl get pods`. You need a systematic approach that connects observed symptoms to concrete evidence, scoped changes, and verified recovery. This guide is written for developers, DevOps consultants, and technical startup teams who operate shared Kubernetes clusters. It focuses on practical, copy-pasteable commands and decision paths. Every section starts with a read-only observation, moves to the smallest justified change, and ends with a verification command that confirms the result. You will learn how to inspect namespaces, resource quotas, RBAC rules, and admission control failures; how to diagnose the most common multi-tenancy failures; and how to recover without affecting other tenants. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached. ## Version and Environment Inventory Before touching anything, you need a precise picture of the cluster, the control plane version, and the components that implement multi-tenancy. A mismatch between Kubernetes API versions and your manifests is a frequent root cause of silent failures. **Identify the cluster version and API resources:** ```bash kubectl version --short kubectl api-resources | grep -E 'namespace|resourcequota|limitrange|role|rolebinding|networkpolicy' ``` Expected output includes the server version, for example `Server Version: v1.28.1`, and a list of API resources. If `resourcequota` or `networkpolicy` is missing, your cluster may not have the required controllers, or you are using a managed distribution that exposes them differently. **Check which namespaces exist and their labels.** Multi-tenancy often relies on namespace labels for policy enforcement. A missing label can silently exclude a tenant from a quota or network policy. ```bash kubectl get namespaces --show-labels ``` Example output: ```text NAME STATUS AGE LABELS team-a Active 21d tenant=acme,environment=prod team-b Active 21d tenant=globex,environment=prod default Active 90d ``` If a namespace expected to have `tenant=acme` is missing that label, a `ResourceQuota` selector based on that label will not apply. This is a common cause of "quota not enforced" or "quota applied to wrong namespace" issues. **Verify the API server feature gates for multi-tenancy features.** Some advanced isolation features, such as Pod Security Admission (PSA) or hierarchical namespaces, depend on feature gates. PSA replaces PodSecurityPolicy and is enabled by default from Kubernetes 1.25 onward. ```bash kubectl get --raw /metrics | grep apiserver_feature_enabled | grep -E 'PodSecurity|HierarchicalNamespace' ``` This returns metrics like `apiserver_feature_enabled{name="PodSecurity",stage="stable"} 1`. If the feature gate is not present or returns 0, PSA is not active and you cannot rely on it for tenant isolation. **Inventory admission controllers.** Many multi-tenancy policies are enforced by admission webhooks (e.g., OPA Gatekeeper, Kyverno, custom webhooks). A misconfigured webhook can reject all tenant workloads. ```bash kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations ``` Look for webhooks with `failurePolicy: Fail`. If the webhook service is unavailable, every matching request fails. You can check the webhook's effect by describing it: ```bash kubectl describe validatingwebhookconfiguration ``` The output shows the rules and the service. If the service has no endpoints, requests time out. **Keep a local test small.** Before applying any change to a shared cluster, reproduce the issue in a dedicated test namespace. For example: ```bash kubectl create namespace test-tenant-a kubectl label namespace test-tenant-a tenant=acme kubectl apply -f test-manifest.yaml -n test-tenant-a kubectl get pods -n test-tenant-a -o wide ``` Inspect the generated resources and verify traffic with `kubectl port-forward` or a `ClusterIP` service before moving to a cloud load balancer or ingress controller. ## Safe Configuration Path Changing configuration in a multi-tenant cluster can trigger cascading failures. Follow a path that minimizes risk: retrieve current state, back it up, make one scoped change, and verify. **Example: adjusting a ResourceQuota for tenant "acme".** Step 1: Observe current quota usage. ```bash kubectl get resourcequota -n team-a kubectl describe resourcequota compute-quota -n team-a ``` Sample output: ```text Name: compute-quota Namespace: team-a Resource Used Hard -------- ---- ---- requests.cpu 3 4 requests.memory 6Gi 8Gi limits.cpu 5 6 ``` Step 2: Export the current quota for backup. ```bash kubectl get resourcequota compute-quota -n team-a -o yaml > team-a-quota-backup.yaml ``` Step 3: Make a scoped change. Suppose you need to increase the CPU limit from 6 to 8 because a tenant's deployment is being throttled. Edit the quota with `kubectl edit` or apply a modified manifest. Here is a minimal patch: ```bash kubectl patch resourcequota compute-quota -n team-a --type='json' -p='[{"op": "replace", "path": "/spec/hard/limits.cpu", "value": "8"}]' ``` Step 4: Verify that the change took effect and that existing pods are not evicted. ```bash kubectl get resourcequota compute-quota -n team-a -o yaml | grep limits.cpu kubectl get pods -n team-a ``` If you see pods in `Evicted` or `Pending` state, the quota increase may have triggered a cascade of pending pods that now exceed memory limits. Roll back using the backup: ```bash kubectl apply -f team-a-quota-backup.yaml ``` **Safe change principles:** - Never delete a quota or limit range without first exporting its current definition. - Change one field at a time. If you must change multiple fields, apply them in separate patches and verify after each. - Use `kubectl diff -f new-manifest.yaml` before applying to see exactly what will change. - For Role-Based Access Control (RBAC), test changes with `kubectl auth can-i --as system:serviceaccount:team-a:default` to simulate the tenant's identity. **Example: grant a tenant service account read-only access to its namespace.** First, check current permissions: ```bash kubectl auth can-i list pods --as system:serviceaccount:team-a:default -n team-a ``` If the output is `no`, create a Role and RoleBinding: ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: namespace: team-a name: pod-reader rules: - apiGroups: [""] resources: ["pods"] verbs: ["get", "list", "watch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: namespace: team-a name: read-pods subjects: - kind: ServiceAccount name: default namespace: team-a roleRef: kind: Role name: pod-reader apiGroup: rbac.authorization.k8s.io ``` Apply and verify: ```bash kubectl apply -f rbac-readonly.yaml kubectl auth can-i list pods --as system:serviceaccount:team-a:default -n team-a ``` Expected output: `yes`. If the output remains `no`, check that the RoleBinding exists and the ServiceAccount name matches exactly. A common mistake is a typo in the namespace or service account name. ## Verification and Diagnostics Once you suspect a problem, you need to collect evidence that points to the root cause. The following commands create a diagnostic trail for multi-tenancy issues. **Check pod scheduling failures.** Resource contention between tenants is a frequent cause. ```bash kubectl get pods -n team-a -o wide kubectl describe pod -n team-a ``` In the `Events` section of `describe`, look for messages like: ```text Warning FailedScheduling 2m (x5 over 10m) default-scheduler 0/3 nodes are available: 3 Insufficient cpu, 2 Insufficient memory. ``` If you see `Insufficient cpu` or `Insufficient memory`, the tenant's resource requests exceed what the cluster can provide given other tenants' usage. Check the node allocatable resources: ```bash kubectl describe nodes | grep -A 5 'Allocated resources' ``` This shows the total resource requests and limits per node. If CPU requests are at 95% of allocatable, no new tenant pods can schedule even if actual usage is low. **Examine quota enforcement.** If a pod is rejected at admission, `kubectl apply` returns an error like: ```text Error from server (Forbidden): error when creating "deployment.yaml": pods "web-7d9f8c" is forbidden: exceeded quota: compute-quota, requested: requests.cpu=2, used: requests.cpu=7, limited: requests.cpu=8 ``` This tells you exactly which quota is exhausted. You can verify current usage with `kubectl describe resourcequota` as above. **Inspect RBAC denial logs.** Kubernetes API server audit logs record forbidden requests. If you have access to the audit log (often in `/var/log/kubernetes/audit.log` on control plane nodes or via a managed service), search for the tenant's service account: ```bash grep 'system:serviceaccount:team-a:default' /var/log/kubernetes/audit.log | grep '"code":403' ``` The log line includes the verb, resource, and namespace. For example, a denied `create deployments` in `team-b` namespace indicates the service account is trying to act outside its namespace, which is a tenant isolation boundary violation. **Test network policy enforcement.** Multi-tenancy often uses NetworkPolicies to isolate namespaces. If a tenant reports that they cannot reach a service in another namespace, check the policies: ```bash kubectl get networkpolicies -A kubectl describe networkpolicy deny-from-other-namespaces -n team-a ``` The describe output shows the pod selectors and ingress/egress rules. To confirm whether a policy is blocking traffic, you can run a connectivity test from a pod: ```bash kubectl run test-pod --rm -it --image=busybox -n team-b -- sh # Inside the pod: wget -qO- --timeout=2 http://service-in-team-a.team-a.svc.cluster.local ``` If the connection times out, and the NetworkPolicy in `team-a` denies ingress from all namespaces except those with a specific label, then the policy is working. If the connection succeeds unexpectedly, the policy may be misconfigured or not applied due to a missing pod selector. **Check admission webhook failures.** If all requests to a namespace suddenly fail with a message like: ```text Error from server (InternalError): Internal error occurred: failed calling webhook "validate.kyverno.svc" ``` Then the webhook service is unreachable. Check its endpoints: ```bash kubectl get endpoints -n kyverno ``` If there are no endpoints, the webhook pods are down. Restart them and verify before retrying the tenant workload. To temporarily unblock a namespace, you can adjust the webhook's `failurePolicy` to `Ignore` (not recommended for production security) or fix the underlying service. ## Failure Modes and Recovery Understanding common multi-tenancy failure modes prepares you to recover quickly and with minimal impact. ### 1. Quota exhaustion causing pod creation failure **Symptom:** `kubectl apply` returns a forbidden error mentioning `exceeded quota`. **Root cause:** The tenant's resource usage has hit the namespace's ResourceQuota hard limit. **Recovery:** - Identify the quota: `kubectl describe resourcequota -n `. - Determine if the tenant legitimately needs more resources or if they have orphaned resources consuming quota (e.g., completed Jobs, evicted pods, unused PVCs). - Clean up orphaned resources: ```bash kubectl delete pods --field-selector=status.phase=Failed -n kubectl delete jobs --field-selector=status.successful=1 -n ``` - If a quota increase is justified, follow the Safe Configuration Path to raise the limit in small increments and monitor node capacity. - Verify by attempting to create a test pod. ### 2. RBAC misconfiguration locking out a tenant **Symptom:** A developer reports `forbidden` errors when running `kubectl` commands in their namespace, even though they previously worked. **Root cause:** A RoleBinding was accidentally deleted or modified, or a cluster-wide policy changed. **Recovery:** - Check the tenant's current permissions: ```bash kubectl auth can-i --list --as system:serviceaccount:team-a:default -n team-a ``` - Look for missing `get`, `list`, `create` on expected resources. - Compare with an audit trail or GitOps repository to find what changed. - Restore the correct Role and RoleBinding from version control: ```bash kubectl apply -f team-a-rbac.yaml ``` - Verify with `kubectl auth can-i create deployments --as system:serviceaccount:team-a:default -n team-a` returning `yes`. ### 3. Noisy neighbor causing performance degradation **Symptom:** Tenant A's application responds slowly, and pods are being throttled or OOMKilled. **Root cause:** Another tenant on the same nodes is consuming excessive CPU or memory, leaving less for Tenant A. This happens when requests and limits are not set, allowing a tenant to use unbounded resources. **Recovery:** - Identify top resource-consuming pods cluster-wide: ```bash kubectl top pods -A --sort-by=cpu kubectl top pods -A --sort-by=memory ``` - For the offending namespace, check if limits are set: ```bash kubectl get pods -n team-b -o json | jq '.items[] | select(.spec.containers[].resources.limits == null) | .metadata.name' ``` - If limits are missing, the tenant must add them. As a temporary mitigation, you can set a LimitRange to enforce default limits: ```yaml apiVersion: v1 kind: LimitRange metadata: name: default-limits namespace: team-b spec: limits: - default: cpu: "1" memory: "1Gi" defaultRequest: cpu: "0.5" memory: "512Mi" type: Container ``` Apply it and then restart the pods. Existing pods are not updated automatically, so you may need to delete them to pick up the new defaults. - Long-term, use a combination of ResourceQuotas at the namespace level and LimitRanges at the pod level, and consider node affinity or taints to separate tenants if strict isolation is needed. ### 4. Network policy misconfiguration breaking connectivity **Symptom:** Tenant A suddenly cannot reach a service in Tenant B, or external ingress traffic is blocked. **Root cause:** A NetworkPolicy was added or modified and now denies traffic unintentionally. Common mistake: a policy with an empty pod selector, which selects all pods in the namespace and blocks all traffic not explicitly allowed. **Recovery:** - List all NetworkPolicies in the affected namespaces: ```bash kubectl get networkpolicies -A ``` - For the policy you suspect, view its details: ```bash kubectl describe networkpolicy -n ``` - If the policy is too broad, edit it to add the correct pod selector or allow necessary ingress from specific namespaces. Example of a deny-all-ingress but allow from same namespace and from a monitoring namespace: ```yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny-ingress namespace: team-a spec: podSelector: {} policyTypes: - Ingress ingress: - from: - podSelector: {} - namespaceSelector: matchLabels: purpose: monitoring ``` - After applying, test connectivity again. ## Operations Checklist Use this checklist as a daily or weekly operational review for multi-tenant clusters. Assign each item to a single accountable owner and revisit at least monthly to ensure it remains valid. | Item | Check | Accountable Owner | Frequency of Review | |------|-------|-------------------|---------------------| | 1. Cluster version and API resources | `kubectl version` and `kubectl api-resources` show expected multi-tenancy resources | Platform Lead (e.g., Priya Shah) | Monthly or after any cluster upgrade | | 2. Namespace labels | Every tenant namespace has required labels (`tenant=`, `environment=`) | DevOps Engineer (e.g., Marcus Chen) | Weekly | | 3. Resource quota usage | No namespace is above 80% of its hard limits | Platform Lead (Priya Shah) | Weekly | | 4. LimitRange defaults | All namespaces have LimitRange defaults for CPU and memory | DevOps Engineer (Marcus Chen) | Monthly | | 5. RBAC least privilege | `kubectl auth can-i --list` shows no overly permissive roles; no tenant service account has cluster-admin | Security Owner (e.g., Elena Rodriguez) | Monthly | | 6. NetworkPolicy isolation | Default-deny policies exist for all tenant namespaces | Security Owner (Elena Rodriguez) | Monthly | | 7. Admission webhook health | Webhook endpoints are healthy; no webhook has `failurePolicy: Fail` with unreachable service | Platform Lead (Priya Shah) | Weekly | | 8. Pod Security Admission | Namespaces have appropriate PSA levels (e.g., `baseline` or `restricted`) | Security Owner (Elena Rodriguez) | Monthly | | 9. Resource requests and limits on all pods | No pod is missing requests or limits (except explicitly allowed system pods) | DevOps Engineer (Marcus Chen) | Weekly | | 10. Audit log review for denied requests | No unexpected 403 errors from tenant service accounts | Security Owner (Elena Rodriguez) | Weekly | | 11. Backup of tenant configurations | All ResourceQuotas, LimitRanges, RBAC, and NetworkPolicies are version-controlled in Git | DevOps Engineer (Marcus Chen) | After any change | | 12. Recovery tested | Once a quarter, run a tabletop exercise to recover from a simulated quota exhaustion or RBAC lockout | Platform Lead (Priya Shah) | Quarterly | ## Common Pitfalls Avoid these mistakes that frequently appear in Kubernetes multi-tenancy operations. ### 1. Applying changes without exporting current state **Why it happens:** Operators under pressure to fix an issue skip the backup step. **Consequence:** If the change is wrong, there is no easy way to revert; the problem worsens and affects more tenants. **How to avoid and recover:** Always run `kubectl get -o yaml > backup.yaml` before modifying. If you forget, check if the resource is in a GitOps repository or if the cluster has an audit trail. For immediate recovery, if you know the previous values, re-apply them manually. ### 2. Using cluster-admin for tenant operations **Why it happens:** It's convenient to use a cluster-admin kubeconfig for all tasks. **Consequence:** A mistaken command can affect all tenants; security boundaries are violated. **How to avoid:** Use per-namespace service accounts with limited roles for routine operations. For example, create a `tenant-admin` Role that allows all actions within a namespace but not across namespaces. Use `kubectl --as` to simulate tenant identity before granting permissions. ### 3. Ignoring LimitRange and relying only on ResourceQuota **Why it happens:** Teams think a namespace quota is enough to prevent overconsumption. **Consequence:** A single pod without limits can consume all CPU within the namespace quota, starving other pods in the same namespace. **How to avoid:** Always define both ResourceQuota (aggregate limits) and LimitRange (default and max per pod). Example: ```yaml apiVersion: v1 kind: LimitRange metadata: name: pod-limits namespace: team-a spec: limits: - max: cpu: "2" memory: "4Gi" min: cpu: "100m" memory: "128Mi" default: cpu: "500m" memory: "512Mi" defaultRequest: cpu: "250m" memory: "256Mi" type: Pod ``` ### 4. Overly broad NetworkPolicies **Why it happens:** Operators write a policy that allows all ingress from any namespace because they are unsure of the exact traffic requirements. **Consequence:** Tenant isolation is broken; any pod in the cluster can access services in the namespace. **How to avoid:** Start with a default-deny policy and explicitly allow only required sources. Use `kubectl describe networkpolicy` to verify the selectors. Test with a probe pod from another namespace to confirm denial. ### 5. Not monitoring quota usage **Why it happens:** Quotas are set once and forgotten. **Consequence:** Tenants hit hard limits unexpectedly, causing production outages. **How to avoid:** Set up alerts on quota usage. For example, use Prometheus and alert when `kube_resourcequota{type="used"} / kube_resourcequota{type="hard"} > 0.8`. Review quota usage weekly and adjust based on trends. ## Conclusion Troubleshooting Kubernetes multi-tenancy effectively requires a disciplined approach: know your environment, make safe incremental changes, collect hard evidence, and prepare for common failures. The commands and playbooks in this guide give you a starting point, but you must adapt them to your cluster's specific configuration and policies. As a next step, choose one low-risk verification from the Operations Checklist, record the current state, run the documented check, compare the result with the expected signal, and review dependencies such as Namespace, Resource Quota, and RBAC. Assign an owner and set a review cadence. A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. Build these habits now, and your multi-tenant cluster will remain stable, secure, and fair to all tenants.