Kubernetes Multi-Tenancy Troubleshooting: A Practical Guide

Intro

Kubernetes multi-tenancy is a powerful model, but it introduces a specific class of operational problems: tenant isolation failures, noisy neighbor resource contention, RBAC misconfigurations that lock out legitimate users, and silent quota exhaustion that blocks workloads without a clear signal. Troubleshooting these issues requires more than running generic kubectl get pods. You need a systematic approach that connects observed symptoms to concrete evidence, scoped changes, and verified recovery.

This guide is written for developers, DevOps consultants, and technical startup teams who operate shared Kubernetes clusters. It focuses on practical, copy-pasteable commands and decision paths. Every section starts with a read-only observation, moves to the smallest justified change, and ends with a verification command that confirms the result. You will learn how to inspect namespaces, resource quotas, RBAC rules, and admission control failures; how to diagnose the most common multi-tenancy failures; and how to recover without affecting other tenants.

The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached.

Version and Environment Inventory

Before touching anything, you need a precise picture of the cluster, the control plane version, and the components that implement multi-tenancy. A mismatch between Kubernetes API versions and your manifests is a frequent root cause of silent failures.

Identify the cluster version and API resources:

kubectl version --short
kubectl api-resources | grep -E 'namespace|resourcequota|limitrange|role|rolebinding|networkpolicy'

Expected output includes the server version, for example Server Version: v1.28.1, and a list of API resources. If resourcequota or networkpolicy is missing, your cluster may not have the required controllers, or you are using a managed distribution that exposes them differently.

Check which namespaces exist and their labels. Multi-tenancy often relies on namespace labels for policy enforcement. A missing label can silently exclude a tenant from a quota or network policy.

kubectl get namespaces --show-labels

Example output:

NAME              STATUS   AGE   LABELS
team-a            Active   21d   tenant=acme,environment=prod
team-b            Active   21d   tenant=globex,environment=prod
default           Active   90d   <none>

If a namespace expected to have tenant=acme is missing that label, a ResourceQuota selector based on that label will not apply. This is a common cause of "quota not enforced" or "quota applied to wrong namespace" issues.

Verify the API server feature gates for multi-tenancy features. Some advanced isolation features, such as Pod Security Admission (PSA) or hierarchical namespaces, depend on feature gates. PSA replaces PodSecurityPolicy and is enabled by default from Kubernetes 1.25 onward.

kubectl get --raw /metrics | grep apiserver_feature_enabled | grep -E 'PodSecurity|HierarchicalNamespace'

This returns metrics like apiserver_feature_enabled{name="PodSecurity",stage="stable"} 1. If the feature gate is not present or returns 0, PSA is not active and you cannot rely on it for tenant isolation.

Inventory admission controllers. Many multi-tenancy policies are enforced by admission webhooks (e.g., OPA Gatekeeper, Kyverno, custom webhooks). A misconfigured webhook can reject all tenant workloads.

kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations

Look for webhooks with failurePolicy: Fail. If the webhook service is unavailable, every matching request fails. You can check the webhook's effect by describing it:

kubectl describe validatingwebhookconfiguration <webhook-name>

The output shows the rules and the service. If the service has no endpoints, requests time out.

Keep a local test small. Before applying any change to a shared cluster, reproduce the issue in a dedicated test namespace. For example:

kubectl create namespace test-tenant-a
kubectl label namespace test-tenant-a tenant=acme
kubectl apply -f test-manifest.yaml -n test-tenant-a
kubectl get pods -n test-tenant-a -o wide

Inspect the generated resources and verify traffic with kubectl port-forward or a ClusterIP service before moving to a cloud load balancer or ingress controller.

Safe Configuration Path

Changing configuration in a multi-tenant cluster can trigger cascading failures. Follow a path that minimizes risk: retrieve current state, back it up, make one scoped change, and verify.

Example: adjusting a ResourceQuota for tenant "acme".

Step 1: Observe current quota usage.

kubectl get resourcequota -n team-a
kubectl describe resourcequota compute-quota -n team-a

Sample output:

Name:            compute-quota
Namespace:       team-a
Resource         Used   Hard
--------         ----   ----
requests.cpu     3      4
requests.memory  6Gi    8Gi
limits.cpu       5      6

Step 2: Export the current quota for backup.

kubectl get resourcequota compute-quota -n team-a -o yaml > team-a-quota-backup.yaml

Step 3: Make a scoped change. Suppose you need to increase the CPU limit from 6 to 8 because a tenant's deployment is being throttled. Edit the quota with kubectl edit or apply a modified manifest. Here is a minimal patch:

kubectl patch resourcequota compute-quota -n team-a --type='json' -p='[{"op": "replace", "path": "/spec/hard/limits.cpu", "value": "8"}]'

Step 4: Verify that the change took effect and that existing pods are not evicted.

kubectl get resourcequota compute-quota -n team-a -o yaml | grep limits.cpu
kubectl get pods -n team-a

If you see pods in Evicted or Pending state, the quota increase may have triggered a cascade of pending pods that now exceed memory limits. Roll back using the backup:

kubectl apply -f team-a-quota-backup.yaml

Safe change principles:

• Never delete a quota or limit range without first exporting its current definition.
• Change one field at a time. If you must change multiple fields, apply them in separate patches and verify after each.
• Use kubectl diff -f new-manifest.yaml before applying to see exactly what will change.
• For Role-Based Access Control (RBAC), test changes with kubectl auth can-i --as system:serviceaccount:team-a:default to simulate the tenant's identity.

Example: grant a tenant service account read-only access to its namespace.

First, check current permissions:

kubectl auth can-i list pods --as system:serviceaccount:team-a:default -n team-a

If the output is no, create a Role and RoleBinding:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: team-a
  name: pod-reader
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: team-a
  name: read-pods
subjects:
- kind: ServiceAccount
  name: default
  namespace: team-a
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

Apply and verify:

kubectl apply -f rbac-readonly.yaml
kubectl auth can-i list pods --as system:serviceaccount:team-a:default -n team-a

Expected output: yes.

If the output remains no, check that the RoleBinding exists and the ServiceAccount name matches exactly. A common mistake is a typo in the namespace or service account name.

Verification and Diagnostics

Once you suspect a problem, you need to collect evidence that points to the root cause. The following commands create a diagnostic trail for multi-tenancy issues.

Check pod scheduling failures. Resource contention between tenants is a frequent cause.

kubectl get pods -n team-a -o wide
kubectl describe pod <pod-name> -n team-a

In the Events section of describe, look for messages like:

Warning  FailedScheduling  2m (x5 over 10m)  default-scheduler  0/3 nodes are available: 3 Insufficient cpu, 2 Insufficient memory.

If you see Insufficient cpu or Insufficient memory, the tenant's resource requests exceed what the cluster can provide given other tenants' usage. Check the node allocatable resources:

kubectl describe nodes | grep -A 5 'Allocated resources'

This shows the total resource requests and limits per node. If CPU requests are at 95% of allocatable, no new tenant pods can schedule even if actual usage is low.

Examine quota enforcement. If a pod is rejected at admission, kubectl apply returns an error like:

Error from server (Forbidden): error when creating "deployment.yaml": pods "web-7d9f8c" is forbidden: exceeded quota: compute-quota, requested: requests.cpu=2, used: requests.cpu=7, limited: requests.cpu=8

This tells you exactly which quota is exhausted. You can verify current usage with kubectl describe resourcequota as above.

Inspect RBAC denial logs. Kubernetes API server audit logs record forbidden requests. If you have access to the audit log (often in /var/log/kubernetes/audit.log on control plane nodes or via a managed service), search for the tenant's service account:

grep 'system:serviceaccount:team-a:default' /var/log/kubernetes/audit.log | grep '"code":403'

The log line includes the verb, resource, and namespace. For example, a denied create deployments in team-b namespace indicates the service account is trying to act outside its namespace, which is a tenant isolation boundary violation.

Test network policy enforcement. Multi-tenancy often uses NetworkPolicies to isolate namespaces. If a tenant reports that they cannot reach a service in another namespace, check the policies:

kubectl get networkpolicies -A
kubectl describe networkpolicy deny-from-other-namespaces -n team-a

The describe output shows the pod selectors and ingress/egress rules. To confirm whether a policy is blocking traffic, you can run a connectivity test from a pod:

kubectl run test-pod --rm -it --image=busybox -n team-b -- sh
# Inside the pod:
wget -qO- --timeout=2 http://service-in-team-a.team-a.svc.cluster.local

If the connection times out, and the NetworkPolicy in team-a denies ingress from all namespaces except those with a specific label, then the policy is working. If the connection succeeds unexpectedly, the policy may be misconfigured or not applied due to a missing pod selector.

Check admission webhook failures. If all requests to a namespace suddenly fail with a message like:

Error from server (InternalError): Internal error occurred: failed calling webhook "validate.kyverno.svc"

Then the webhook service is unreachable. Check its endpoints:

kubectl get endpoints -n kyverno

If there are no endpoints, the webhook pods are down. Restart them and verify before retrying the tenant workload. To temporarily unblock a namespace, you can adjust the webhook's failurePolicy to Ignore (not recommended for production security) or fix the underlying service.

Failure Modes and Recovery

Understanding common multi-tenancy failure modes prepares you to recover quickly and with minimal impact.

1. Quota exhaustion causing pod creation failure

Symptom: kubectl apply returns a forbidden error mentioning exceeded quota.

Root cause: The tenant's resource usage has hit the namespace's ResourceQuota hard limit.

Recovery:

• Identify the quota: kubectl describe resourcequota -n <namespace>.
• Determine if the tenant legitimately needs more resources or if they have orphaned resources consuming quota (e.g., completed Jobs, evicted pods, unused PVCs).
• Clean up orphaned resources:

kubectl delete pods --field-selector=status.phase=Failed -n <namespace>
kubectl delete jobs --field-selector=status.successful=1 -n <namespace>

• If a quota increase is justified, follow the Safe Configuration Path to raise the limit in small increments and monitor node capacity.
• Verify by attempting to create a test pod.

2. RBAC misconfiguration locking out a tenant

Symptom: A developer reports forbidden errors when running kubectl commands in their namespace, even though they previously worked.

Root cause: A RoleBinding was accidentally deleted or modified, or a cluster-wide policy changed.

Recovery:

• Check the tenant's current permissions:

kubectl auth can-i --list --as system:serviceaccount:team-a:default -n team-a

• Look for missing get, list, create on expected resources.
• Compare with an audit trail or GitOps repository to find what changed.
• Restore the correct Role and RoleBinding from version control:

kubectl apply -f team-a-rbac.yaml

• Verify with kubectl auth can-i create deployments --as system:serviceaccount:team-a:default -n team-a returning yes.

3. Noisy neighbor causing performance degradation

Symptom: Tenant A's application responds slowly, and pods are being throttled or OOMKilled.

Root cause: Another tenant on the same nodes is consuming excessive CPU or memory, leaving less for Tenant A. This happens when requests and limits are not set, allowing a tenant to use unbounded resources.

Recovery:

• Identify top resource-consuming pods cluster-wide:

kubectl top pods -A --sort-by=cpu
kubectl top pods -A --sort-by=memory

• For the offending namespace, check if limits are set:

kubectl get pods -n team-b -o json | jq '.items[] | select(.spec.containers[].resources.limits == null) | .metadata.name'

• If limits are missing, the tenant must add them. As a temporary mitigation, you can set a LimitRange to enforce default limits:

apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
  namespace: team-b
spec:
  limits:
  - default:
      cpu: "1"
      memory: "1Gi"
    defaultRequest:
      cpu: "0.5"
      memory: "512Mi"
    type: Container

Apply it and then restart the pods. Existing pods are not updated automatically, so you may need to delete them to pick up the new defaults.

• Long-term, use a combination of ResourceQuotas at the namespace level and LimitRanges at the pod level, and consider node affinity or taints to separate tenants if strict isolation is needed.

4. Network policy misconfiguration breaking connectivity

Symptom: Tenant A suddenly cannot reach a service in Tenant B, or external ingress traffic is blocked.

Root cause: A NetworkPolicy was added or modified and now denies traffic unintentionally. Common mistake: a policy with an empty pod selector, which selects all pods in the namespace and blocks all traffic not explicitly allowed.

Recovery:

• List all NetworkPolicies in the affected namespaces:

kubectl get networkpolicies -A

• For the policy you suspect, view its details:

kubectl describe networkpolicy <policy-name> -n <namespace>

• If the policy is too broad, edit it to add the correct pod selector or allow necessary ingress from specific namespaces. Example of a deny-all-ingress but allow from same namespace and from a monitoring namespace:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: team-a
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector: {}
    - namespaceSelector:
        matchLabels:
          purpose: monitoring

• After applying, test connectivity again.

Operations Checklist

Use this checklist as a daily or weekly operational review for multi-tenant clusters. Assign each item to a single accountable owner and revisit at least monthly to ensure it remains valid.

Common Pitfalls

Avoid these mistakes that frequently appear in Kubernetes multi-tenancy operations.

1. Applying changes without exporting current state

Why it happens: Operators under pressure to fix an issue skip the backup step.

Consequence: If the change is wrong, there is no easy way to revert; the problem worsens and affects more tenants.

How to avoid and recover: Always run kubectl get <resource> -o yaml > backup.yaml before modifying. If you forget, check if the resource is in a GitOps repository or if the cluster has an audit trail. For immediate recovery, if you know the previous values, re-apply them manually.

2. Using cluster-admin for tenant operations

Why it happens: It's convenient to use a cluster-admin kubeconfig for all tasks.

Consequence: A mistaken command can affect all tenants; security boundaries are violated.

How to avoid: Use per-namespace service accounts with limited roles for routine operations. For example, create a tenant-admin Role that allows all actions within a namespace but not across namespaces. Use kubectl --as to simulate tenant identity before granting permissions.

3. Ignoring LimitRange and relying only on ResourceQuota

Why it happens: Teams think a namespace quota is enough to prevent overconsumption.

Consequence: A single pod without limits can consume all CPU within the namespace quota, starving other pods in the same namespace.

How to avoid: Always define both ResourceQuota (aggregate limits) and LimitRange (default and max per pod). Example:

apiVersion: v1
kind: LimitRange
metadata:
  name: pod-limits
  namespace: team-a
spec:
  limits:
  - max:
      cpu: "2"
      memory: "4Gi"
    min:
      cpu: "100m"
      memory: "128Mi"
    default:
      cpu: "500m"
      memory: "512Mi"
    defaultRequest:
      cpu: "250m"
      memory: "256Mi"
    type: Pod

4. Overly broad NetworkPolicies

Why it happens: Operators write a policy that allows all ingress from any namespace because they are unsure of the exact traffic requirements.

Consequence: Tenant isolation is broken; any pod in the cluster can access services in the namespace.

How to avoid: Start with a default-deny policy and explicitly allow only required sources. Use kubectl describe networkpolicy to verify the selectors. Test with a probe pod from another namespace to confirm denial.

5. Not monitoring quota usage

Why it happens: Quotas are set once and forgotten.

Consequence: Tenants hit hard limits unexpectedly, causing production outages.

How to avoid: Set up alerts on quota usage. For example, use Prometheus and alert when kube_resourcequota{type="used"} / kube_resourcequota{type="hard"} > 0.8. Review quota usage weekly and adjust based on trends.

Conclusion

Troubleshooting Kubernetes multi-tenancy effectively requires a disciplined approach: know your environment, make safe incremental changes, collect hard evidence, and prepare for common failures. The commands and playbooks in this guide give you a starting point, but you must adapt them to your cluster's specific configuration and policies.

As a next step, choose one low-risk verification from the Operations Checklist, record the current state, run the documented check, compare the result with the expected signal, and review dependencies such as Namespace, Resource Quota, and RBAC. Assign an owner and set a review cadence.

A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. Build these habits now, and your multi-tenant cluster will remain stable, secure, and fair to all tenants.