## Intro

Multi-tenancy in Kubernetes means running workloads from different teams, projects, or customers on a shared cluster while keeping them isolated from each other. In a local lab, you can safely experiment with the building blocks of multi-tenancy: namespaces, resource quotas, role-based access control (RBAC), and network policies. This article shows how to set up such a lab, test isolation, diagnose common failures, and recover from them. It is written for developers, DevOps engineers, and technical startup teams who want hands-on experience with Kubernetes multi-tenancy before moving to production.

The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify each result, and document how to recover if the expected state is not reached. Every command includes the expected output or signal, and every change is scoped and reversible where possible.

## Version and Environment Inventory

Before applying any manifests, confirm that your local Kubernetes cluster is running and that you have the necessary tools. The examples in this article were tested with Kubernetes v1.27, kind v0.20, kubectl v1.27, and a Linux host with 4 CPUs and 8 GB of RAM. You can use Minikube, Docker Desktop, or any other local cluster, but commands may differ slightly.

First, check the cluster version and node status:

```bash
kubectl version --short
kubectl get nodes -o wide
```

Expected output shows a single node with status `Ready` and the Kubernetes version matching your setup. If the node is `NotReady`, wait a moment and re-run the command. For kind, you can check container status with `docker ps`.

Next, verify that the core add-ons you need for multi-tenancy are available. Most distributions include RBAC and resource quotas by default. Network policies require a CNI plugin that supports them, such as Calico, Cilium, or Antrea. In a kind cluster, the default CNI does not enforce network policies; you must install one. For example, to install Calico on kind:

```bash
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/kind/main/site/content/docs/user/calico.yaml
```

Wait for the Calico pods to be ready:

```bash
kubectl get pods -n kube-system | grep calico
```

Expected output shows calico-node pods with status `Running` and `1/1` ready. If they are not ready, check logs with `kubectl logs -n kube-system <calico-pod>`.

Finally, record the exact versions in a lab notebook or a file, because behavior can change between releases. This inventory will help you compare results with teammates or when filing bugs.

## Safe Configuration Path

A multi-tenant cluster starts with namespaces. Create two namespaces to represent two tenants: `tenant-a` and `tenant-b`.

```bash
kubectl create namespace tenant-a
kubectl create namespace tenant-b
```

Expected output: `namespace/tenant-a created` and `namespace/tenant-b created`.

Next, apply resource quotas to limit resource consumption per namespace. Save the following YAML as `quota-tenant-a.yaml`:

```yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: tenant-a-quota
  namespace: tenant-a
spec:
  hard:
    requests.cpu: "2"
    requests.memory: "4Gi"
    limits.cpu: "4"
    limits.memory: "8Gi"
    persistentvolumeclaims: "5"
```

Apply it:

```bash
kubectl apply -f quota-tenant-a.yaml
```

Verify the quota is active:

```bash
kubectl describe resourcequota tenant-a-quota -n tenant-a
```

The output should show the hard limits and current usage (all zero). Create a similar quota for `tenant-b` with different values, for example CPU limit 2, to test isolation.

Now restrict access with RBAC. Create a role that allows managing only pods in the namespace. Save as `role-pod-manager.yaml`:

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: tenant-a
  name: pod-manager
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch", "create", "update", "delete"]
```

Apply it and then create a role binding for a test user. In a real cluster, you would integrate with an identity provider, but for a lab, you can use a service account to simulate a tenant user. Create a service account:

```bash
kubectl create serviceaccount tenant-a-user -n tenant-a
```

Create a role binding:

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: tenant-a-user-pod-manager
  namespace: tenant-a
subjects:
- kind: ServiceAccount
  name: tenant-a-user
  namespace: tenant-a
roleRef:
  kind: Role
  name: pod-manager
  apiGroup: rbac.authorization.k8s.io
```

Apply it. Then test that the service account can create pods but cannot list services. Get the service account token:

```bash
TOKEN=$(kubectl create token tenant-a-user -n tenant-a)
```

Use the token with kubectl:

```bash
kubectl --token=$TOKEN --namespace=tenant-a get pods
kubectl --token=$TOKEN --namespace=tenant-a get services
```

The first command should succeed (empty list), and the second should return `Error from server (Forbidden): services is forbidden: User "system:serviceaccount:tenant-a:tenant-a-user" cannot list resource "services" in API group "" in the namespace "tenant-a"`.

Finally, enforce network isolation between tenants. The following NetworkPolicy denies all ingress to pods in `tenant-a` except from pods in the same namespace. Save as `default-deny-ingress.yaml`:

```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: tenant-a
spec:
  podSelector: {}
  policyTypes:
  - Ingress
```

Apply it and test connectivity later after deploying pods.

These steps follow the principle of smallest justified change: each manifest is applied independently, verified, and is easy to revert by deleting the object.

## Verification and Diagnostics

Now deploy a simple workload in each namespace to test resource limits, RBAC, and network policies. Create a deployment with an nginx pod in `tenant-a`:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: tenant-a
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: nginx
        image: nginx:1.23
        resources:
          requests:
            cpu: "100m"
            memory: "128Mi"
          limits:
            cpu: "500m"
            memory: "256Mi"
```

Apply and check rollout status:

```bash
kubectl apply -f web-deployment.yaml
kubectl rollout status deployment/web -n tenant-a
```

Expected: `deployment "web" successfully rolled out`.

Check that the resource quota is being consumed:

```bash
kubectl describe resourcequota tenant-a-quota -n tenant-a
```

You should see `requests.cpu: 100m/2` and `requests.memory: 128Mi/4Gi` under Used.

Test network connectivity. Since we applied a default-deny ingress policy in `tenant-a`, a pod in `tenant-b` should not be able to reach the nginx pod. First, create a test pod in `tenant-b`:

```bash
kubectl run test-pod --image=busybox:1.35 --restart=Never --namespace=tenant-b -- sleep 3600
```

Wait for it to be running:

```bash
kubectl get pods test-pod -n tenant-b
```

Then, get the IP address of the nginx pod in `tenant-a`:

```bash
NGINX_IP=$(kubectl get pod -n tenant-a -l app=web -o jsonpath='{.items[0].status.podIP}')
echo $NGINX_IP
```

From the test pod, attempt to curl that IP:

```bash
kubectl exec -it test-pod -n tenant-b -- wget -qO- --timeout=5 http://$NGINX_IP
```

Expected: the command times out because the network policy blocks the ingress. If you get a response, the network policy is not enforced, likely due to missing CNI as noted earlier.

Now test cross-namespace access with the service account. The service account `tenant-a-user` has pod management rights only in `tenant-a`, so attempting to list pods in `tenant-b` should fail:

```bash
kubectl --token=$TOKEN --namespace=tenant-b get pods
```

Expected: `Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:tenant-a:tenant-a-user" cannot list resource "pods" in API group "" in the namespace "tenant-b"`.

These verification steps show that namespaces, quotas, RBAC, and network policies are working as intended.

## Failure Modes and Recovery

In a multi-tenant setup, several failure modes are common. Here are three with diagnosis and recovery steps.

### 1. Resource Quota Exceeded

If a tenant tries to create a pod that exceeds the quota, the request is denied. For example, attempt to create a pod that requests 3 CPUs in `tenant-a`, but the quota is 2 CPUs. You will see:

```
Error from server (Forbidden): error when creating "./big-pod.yaml": pods "big-pod" is forbidden: exceeded quota: tenant-a-quota, requested: requests.cpu=3, used: requests.cpu=100m, limited: requests.cpu=2
```

**Recovery**: Reduce the resource requests, or update the quota if the tenant legitimately needs more. Always coordinate with the tenant owner before changing quotas.

### 2. Network Policy Overblocking

If tenants report connectivity issues, use the `kubectl describe` command to inspect network policies:

```bash
kubectl describe networkpolicy -n tenant-a
```

You can see which pods are selected and what traffic is allowed. A common mistake is a pod selector that matches too many pods or a missing egress rule. To recover, temporarily allow all ingress for debugging:

```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-all-ingress-debug
  namespace: tenant-a
spec:
  podSelector: {}
  ingress:
  - {}
```

Apply this policy to restore connectivity, then remove it and apply the correct selective policy.

### 3. RBAC Misconfiguration

If a user cannot perform an action they should be allowed to do, check the role and role binding. Use `kubectl auth can-i` to test permissions:

```bash
kubectl auth can-i create pods --as=system:serviceaccount:tenant-a:tenant-a-user -n tenant-a
```

Expected for our setup: `yes`.

If you get `no`, inspect the role and binding with `kubectl get role pod-manager -n tenant-a -o yaml` and `kubectl get rolebinding -n tenant-a`. Ensure the subject is correct and the role rules include the needed API groups and resources.

## Operations Checklist

Use this checklist after each change or troubleshooting session. Each item has an owner and a review frequency.

- [ ] Cluster health: `kubectl get nodes` shows all nodes `Ready`. Useful when: any change. Owner: cluster admin. Check: before and after each change.
- [ ] Namespace isolation: verify that each tenant namespace exists and its labels are correct: `kubectl get namespaces --show-labels`. Owner: platform engineer. Frequency: weekly.
- [ ] Resource quotas: confirm quotas are applied and not near limit: `kubectl describe resourcequota -A`. Owner: platform engineer. Frequency: after each deployment or weekly.
- [ ] RBAC: review role bindings for least privilege: `kubectl get rolebindings -A`. Owner: security engineer. Frequency: monthly.
- [ ] Network policies: test cross-tenant connectivity is blocked while intra-tenant works. Owner: network engineer. Frequency: after any network policy change or monthly.
- [ ] Audit logs: if enabled, check for forbidden requests that indicate misconfiguration or attack. Owner: security engineer. Frequency: weekly.
- [ ] Backup and recovery: verify that etcd backups can restore the cluster, including namespaces and policies. Owner: cluster admin. Frequency: quarterly.

## Common Pitfalls

### Pitfall 1: Using the Default Namespace for Everything

Many tutorials use the `default` namespace. In multi-tenancy, always create explicit namespaces for each tenant and avoid using `default`. This prevents accidental sharing of resources.

**Recovery**: Migrate resources by recreating them in new namespaces. Use `kubectl get all -n default` to inventory, then export and reapply in the correct namespace.

### Pitfall 2: Ignoring Network Policy CNI Requirements

Network policies are only enforced if a CNI plugin that supports them is installed. Without it, policies are silently ignored, giving false confidence. Always test connectivity after applying policies, as shown in the Verification section.

### Pitfall 3: Overly Broad RBAC Permissions

Granting `cluster-admin` to tenant users is common in labs but dangerous in production. Use namespace-scoped roles and bindings. Audit with `kubectl get clusterrolebindings` and remove suspicious subjects.

### Pitfall 4: Not Setting Resource Requests and Limits

If tenants do not set resource requests, they can starve other tenants. Enforce via limit ranges or admission policies. In your lab, always include resource requests in pod specs to see quota behavior.

### Pitfall 5: Forgetting to Delete Test Resources

Test pods, policies, and service accounts left behind can confuse later experiments. Clean up with `kubectl delete` and verify with `kubectl get all -n <namespace>`.

## Conclusion

A local Kubernetes multi-tenancy lab provides a safe environment to learn and verify the core concepts: namespaces, resource quotas, RBAC, and network policies. This article walked through setting up a minimal two-tenant cluster, applying configurations, testing isolation, diagnosing common failures, and recovering from them. The operations checklist helps maintain control, and the pitfalls section warns against common mistakes.

As a next step, choose one low-risk verification from this article, such as enforcing a network policy and testing cross-tenant connectivity, and practice until you are comfortable. Then expand to more advanced topics like admission control, pod security standards, and hierarchical namespace controllers. Remember that true multi-tenancy is a combination of many controls working together, and regular review is essential to keep your cluster secure and efficient.