Kubernetes Multi-Tenancy Local Lab: Setup, Testing, and Examples

Intro

Multi-tenancy in Kubernetes means running workloads from different teams, projects, or customers on a shared cluster while keeping them isolated from each other. In a local lab, you can safely experiment with the building blocks of multi-tenancy: namespaces, resource quotas, role-based access control (RBAC), and network policies. This article shows how to set up such a lab, test isolation, diagnose common failures, and recover from them. It is written for developers, DevOps engineers, and technical startup teams who want hands-on experience with Kubernetes multi-tenancy before moving to production.

The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify each result, and document how to recover if the expected state is not reached. Every command includes the expected output or signal, and every change is scoped and reversible where possible.

Version and Environment Inventory

Before applying any manifests, confirm that your local Kubernetes cluster is running and that you have the necessary tools. The examples in this article were tested with Kubernetes v1.27, kind v0.20, kubectl v1.27, and a Linux host with 4 CPUs and 8 GB of RAM. You can use Minikube, Docker Desktop, or any other local cluster, but commands may differ slightly.

First, check the cluster version and node status:

kubectl version --short
kubectl get nodes -o wide

Expected output shows a single node with status Ready and the Kubernetes version matching your setup. If the node is NotReady, wait a moment and re-run the command. For kind, you can check container status with docker ps.

Next, verify that the core add-ons you need for multi-tenancy are available. Most distributions include RBAC and resource quotas by default. Network policies require a CNI plugin that supports them, such as Calico, Cilium, or Antrea. In a kind cluster, the default CNI does not enforce network policies; you must install one. For example, to install Calico on kind:

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/kind/main/site/content/docs/user/calico.yaml

Wait for the Calico pods to be ready:

kubectl get pods -n kube-system | grep calico

Expected output shows calico-node pods with status Running and 1/1 ready. If they are not ready, check logs with kubectl logs -n kube-system <calico-pod>.

Finally, record the exact versions in a lab notebook or a file, because behavior can change between releases. This inventory will help you compare results with teammates or when filing bugs.

Safe Configuration Path

A multi-tenant cluster starts with namespaces. Create two namespaces to represent two tenants: tenant-a and tenant-b.

kubectl create namespace tenant-a
kubectl create namespace tenant-b

Expected output: namespace/tenant-a created and namespace/tenant-b created.

Next, apply resource quotas to limit resource consumption per namespace. Save the following YAML as quota-tenant-a.yaml:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: tenant-a-quota
  namespace: tenant-a
spec:
  hard:
    requests.cpu: "2"
    requests.memory: "4Gi"
    limits.cpu: "4"
    limits.memory: "8Gi"
    persistentvolumeclaims: "5"

Apply it:

kubectl apply -f quota-tenant-a.yaml

Verify the quota is active:

kubectl describe resourcequota tenant-a-quota -n tenant-a

The output should show the hard limits and current usage (all zero). Create a similar quota for tenant-b with different values, for example CPU limit 2, to test isolation.

Now restrict access with RBAC. Create a role that allows managing only pods in the namespace. Save as role-pod-manager.yaml:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: tenant-a
  name: pod-manager
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch", "create", "update", "delete"]

Apply it and then create a role binding for a test user. In a real cluster, you would integrate with an identity provider, but for a lab, you can use a service account to simulate a tenant user. Create a service account:

kubectl create serviceaccount tenant-a-user -n tenant-a

Create a role binding:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: tenant-a-user-pod-manager
  namespace: tenant-a
subjects:
- kind: ServiceAccount
  name: tenant-a-user
  namespace: tenant-a
roleRef:
  kind: Role
  name: pod-manager
  apiGroup: rbac.authorization.k8s.io

Apply it. Then test that the service account can create pods but cannot list services. Get the service account token:

TOKEN=$(kubectl create token tenant-a-user -n tenant-a)

Use the token with kubectl:

kubectl --token=$TOKEN --namespace=tenant-a get pods
kubectl --token=$TOKEN --namespace=tenant-a get services

The first command should succeed (empty list), and the second should return Error from server (Forbidden): services is forbidden: User "system:serviceaccount:tenant-a:tenant-a-user" cannot list resource "services" in API group "" in the namespace "tenant-a".

Finally, enforce network isolation between tenants. The following NetworkPolicy denies all ingress to pods in tenant-a except from pods in the same namespace. Save as default-deny-ingress.yaml:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: tenant-a
spec:
  podSelector: {}
  policyTypes:
  - Ingress

Apply it and test connectivity later after deploying pods.

These steps follow the principle of smallest justified change: each manifest is applied independently, verified, and is easy to revert by deleting the object.

Verification and Diagnostics

Now deploy a simple workload in each namespace to test resource limits, RBAC, and network policies. Create a deployment with an nginx pod in tenant-a:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: tenant-a
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: nginx
        image: nginx:1.23
        resources:
          requests:
            cpu: "100m"
            memory: "128Mi"
          limits:
            cpu: "500m"
            memory: "256Mi"

Apply and check rollout status:

kubectl apply -f web-deployment.yaml
kubectl rollout status deployment/web -n tenant-a

Expected: deployment "web" successfully rolled out.

Check that the resource quota is being consumed:

kubectl describe resourcequota tenant-a-quota -n tenant-a

You should see requests.cpu: 100m/2 and requests.memory: 128Mi/4Gi under Used.

Test network connectivity. Since we applied a default-deny ingress policy in tenant-a, a pod in tenant-b should not be able to reach the nginx pod. First, create a test pod in tenant-b:

kubectl run test-pod --image=busybox:1.35 --restart=Never --namespace=tenant-b -- sleep 3600

Wait for it to be running:

kubectl get pods test-pod -n tenant-b

Then, get the IP address of the nginx pod in tenant-a:

NGINX_IP=$(kubectl get pod -n tenant-a -l app=web -o jsonpath='{.items[0].status.podIP}')
echo $NGINX_IP

From the test pod, attempt to curl that IP:

kubectl exec -it test-pod -n tenant-b -- wget -qO- --timeout=5 http://$NGINX_IP

Expected: the command times out because the network policy blocks the ingress. If you get a response, the network policy is not enforced, likely due to missing CNI as noted earlier.

Now test cross-namespace access with the service account. The service account tenant-a-user has pod management rights only in tenant-a, so attempting to list pods in tenant-b should fail:

kubectl --token=$TOKEN --namespace=tenant-b get pods

Expected: Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:tenant-a:tenant-a-user" cannot list resource "pods" in API group "" in the namespace "tenant-b".

These verification steps show that namespaces, quotas, RBAC, and network policies are working as intended.

Failure Modes and Recovery

In a multi-tenant setup, several failure modes are common. Here are three with diagnosis and recovery steps.

1. Resource Quota Exceeded

If a tenant tries to create a pod that exceeds the quota, the request is denied. For example, attempt to create a pod that requests 3 CPUs in tenant-a, but the quota is 2 CPUs. You will see:

Error from server (Forbidden): error when creating "./big-pod.yaml": pods "big-pod" is forbidden: exceeded quota: tenant-a-quota, requested: requests.cpu=3, used: requests.cpu=100m, limited: requests.cpu=2

Recovery: Reduce the resource requests, or update the quota if the tenant legitimately needs more. Always coordinate with the tenant owner before changing quotas.

2. Network Policy Overblocking

If tenants report connectivity issues, use the kubectl describe command to inspect network policies:

kubectl describe networkpolicy -n tenant-a

You can see which pods are selected and what traffic is allowed. A common mistake is a pod selector that matches too many pods or a missing egress rule. To recover, temporarily allow all ingress for debugging:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-all-ingress-debug
  namespace: tenant-a
spec:
  podSelector: {}
  ingress:
  - {}

Apply this policy to restore connectivity, then remove it and apply the correct selective policy.

3. RBAC Misconfiguration

If a user cannot perform an action they should be allowed to do, check the role and role binding. Use kubectl auth can-i to test permissions:

kubectl auth can-i create pods --as=system:serviceaccount:tenant-a:tenant-a-user -n tenant-a

Expected for our setup: yes.

If you get no, inspect the role and binding with kubectl get role pod-manager -n tenant-a -o yaml and kubectl get rolebinding -n tenant-a. Ensure the subject is correct and the role rules include the needed API groups and resources.

Operations Checklist

Use this checklist after each change or troubleshooting session. Each item has an owner and a review frequency.

• [ ] Cluster health: kubectl get nodes shows all nodes Ready. Useful when: any change. Owner: cluster admin. Check: before and after each change.
• [ ] Namespace isolation: verify that each tenant namespace exists and its labels are correct: kubectl get namespaces --show-labels. Owner: platform engineer. Frequency: weekly.
• [ ] Resource quotas: confirm quotas are applied and not near limit: kubectl describe resourcequota -A. Owner: platform engineer. Frequency: after each deployment or weekly.
• [ ] RBAC: review role bindings for least privilege: kubectl get rolebindings -A. Owner: security engineer. Frequency: monthly.
• [ ] Network policies: test cross-tenant connectivity is blocked while intra-tenant works. Owner: network engineer. Frequency: after any network policy change or monthly.
• [ ] Audit logs: if enabled, check for forbidden requests that indicate misconfiguration or attack. Owner: security engineer. Frequency: weekly.
• [ ] Backup and recovery: verify that etcd backups can restore the cluster, including namespaces and policies. Owner: cluster admin. Frequency: quarterly.

Common Pitfalls

Pitfall 1: Using the Default Namespace for Everything

Many tutorials use the default namespace. In multi-tenancy, always create explicit namespaces for each tenant and avoid using default. This prevents accidental sharing of resources.

Recovery: Migrate resources by recreating them in new namespaces. Use kubectl get all -n default to inventory, then export and reapply in the correct namespace.

Pitfall 2: Ignoring Network Policy CNI Requirements

Network policies are only enforced if a CNI plugin that supports them is installed. Without it, policies are silently ignored, giving false confidence. Always test connectivity after applying policies, as shown in the Verification section.

Pitfall 3: Overly Broad RBAC Permissions

Granting cluster-admin to tenant users is common in labs but dangerous in production. Use namespace-scoped roles and bindings. Audit with kubectl get clusterrolebindings and remove suspicious subjects.

Pitfall 4: Not Setting Resource Requests and Limits

If tenants do not set resource requests, they can starve other tenants. Enforce via limit ranges or admission policies. In your lab, always include resource requests in pod specs to see quota behavior.

Pitfall 5: Forgetting to Delete Test Resources

Test pods, policies, and service accounts left behind can confuse later experiments. Clean up with kubectl delete and verify with kubectl get all -n <namespace>.

Conclusion

A local Kubernetes multi-tenancy lab provides a safe environment to learn and verify the core concepts: namespaces, resource quotas, RBAC, and network policies. This article walked through setting up a minimal two-tenant cluster, applying configurations, testing isolation, diagnosing common failures, and recovering from them. The operations checklist helps maintain control, and the pitfalls section warns against common mistakes.

As a next step, choose one low-risk verification from this article, such as enforcing a network policy and testing cross-tenant connectivity, and practice until you are comfortable. Then expand to more advanced topics like admission control, pod security standards, and hierarchical namespace controllers. Remember that true multi-tenancy is a combination of many controls working together, and regular review is essential to keep your cluster secure and efficient.