## Intro Kubernetes multi-tenancy lets multiple teams, applications, or customers share a single cluster while staying isolated from each other. Getting the basics right requires more than copying commands. You need to know what each command does, what output to expect, and what to do when things go wrong. This article is for developers, DevOps consultants, and technical startup teams who need practical command-line skills for managing multi-tenant clusters. It covers essential commands with real examples, expected outputs, failure signals, and recovery steps. You will learn how to inspect your environment, apply safe configuration changes, verify tenant isolation, and diagnose common problems. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and know how to recover if the expected state is not reached. ## Version and Environment Inventory Before running any multi-tenancy command, know what you are working with. Check your Kubernetes version, the current context, and the namespaces that already exist. This helps you avoid running commands against the wrong cluster or accidentally impacting other tenants. ### Check cluster version and API availability Run: ```bash kubectl version --short ``` Expected output looks like: ``` Client Version: v1.27.3 Kustomize Version: v5.0.1 Server Version: v1.27.3 ``` The server version tells you which API resources are available. Multi-tenancy features like ResourceQuota and NetworkPolicy have been stable since early versions, but newer features like hierarchical namespaces (HNC) require v1.20+ and are not enabled by default. ### Verify current context Always confirm which cluster and namespace you are operating in: ```bash kubectl config current-context ``` Expected output example: ``` gke_my-project_us-central1-a_production-cluster ``` If it shows a development or test cluster when you expect production, stop. Switching to the wrong context is a common cause of accidental tenant impact. ### List existing namespaces Multi-tenancy often starts with namespace-level isolation. See what namespaces already exist: ```bash kubectl get namespaces ``` Expected output example: ``` NAME STATUS AGE default Active 210d kube-system Active 210d kube-public Active 210d team-a Active 45d team-b Active 45d ``` Each tenant should have at least one dedicated namespace. Avoid placing tenant workloads in `default` or `kube-system`. ### Inspect current pod and service distribution See a cluster-wide view of pods: ```bash kubectl get pods --all-namespaces -o wide ``` This shows which pods belong to which namespace and node. It helps you spot tenants that are co-located or consuming unusual resources. For a deeper look at a specific pod: ```bash kubectl describe pod -n ``` Look for `Node-Selectors`, `Tolerations`, and `Resource Requests/Limits`. In a multi-tenant environment, these should be set to prevent noisy neighbor problems. ### Check resource quotas Resource quotas are critical for fair sharing. List them across all namespaces: ```bash kubectl get resourcequota --all-namespaces ``` Expected output example: ``` NAMESPACE NAME AGE REQUEST LIMIT team-a team-a-quota 10d requests.cpu: 0/4, requests.memory: 0/8Gi limits.cpu: 0/8, limits.memory: 0/16Gi team-b team-b-quota 10d requests.cpu: 0/2, requests.memory: 0/4Gi limits.cpu: 0/4, limits.memory: 0/8Gi ``` If no quotas exist, tenants can consume unlimited resources and starve others. ### Inspect RBAC bindings Role-Based Access Control (RBAC) enforces who can do what inside each tenant namespace. List role bindings for a namespace: ```bash kubectl get rolebindings -n team-a ``` Expected output: ``` NAME ROLE AGE team-a-admins ClusterRole/admin 30d team-a-devs Role/developer 30d ``` This tells you which users or groups have access. Too many bindings to highly privileged roles is a security risk. ## Safe Configuration Path Making changes in a shared cluster requires care. Follow a path that minimizes risk: define the desired state in a manifest, validate it, apply it to a test namespace, and verify the result before rolling out to production tenants. ### Use YAML manifests, never imperative one-offs Imperative commands like `kubectl create namespace team-c` are fast but not reproducible. Use declarative manifests stored in version control. Example namespace manifest `team-c-namespace.yaml`: ```yaml apiVersion: v1 kind: Namespace metadata: name: team-c labels: tenant: team-c environment: production ``` Apply it: ```bash kubectl apply -f team-c-namespace.yaml ``` Expected output: ``` namespace/team-c created ``` Now `kubectl get namespace team-c` should show `Active`. ### Set resource quotas for a tenant Create a ResourceQuota to limit CPU and memory for pods in `team-c`: ```yaml apiVersion: v1 kind: ResourceQuota metadata: name: team-c-quota namespace: team-c spec: hard: requests.cpu: "4" requests.memory: 8Gi limits.cpu: "8" limits.memory: 16Gi pods: "20" ``` Apply and verify: ```bash kubectl apply -f team-c-quota.yaml kubectl get resourcequota -n team-c ``` Expected output: ``` NAME AGE REQUEST LIMIT team-c-quota 5s requests.cpu: 0/4, requests.memory: 0/8Gi limits.cpu: 0/8, limits.memory: 0/16Gi, pods: 0/20 ``` ### Restrict access with RBAC Create a Role that allows read-only access to pods in `team-c`: ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: namespace: team-c name: pod-reader rules: - apiGroups: [""] resources: ["pods"] verbs: ["get", "list", "watch"] ``` Bind it to a development team group: ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: team-c-devs-read-pods namespace: team-c subjects: - kind: Group name: team-c-devs apiGroup: rbac.authorization.k8s.io roleRef: kind: Role name: pod-reader apiGroup: rbac.authorization.k8s.io ``` Apply both: ```bash kubectl apply -f role.yaml -f rolebinding.yaml ``` Verify: ```bash kubectl auth can-i get pods -n team-c --as system:serviceaccount:default:test-user ``` Expected output if allowed: ``` yes ``` ### Isolate network traffic with NetworkPolicy By default, all pods in a cluster can talk to each other. In multi-tenant environments, you should deny all traffic between namespaces unless explicitly allowed. Example default-deny for `team-c`: ```yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny-all namespace: team-c spec: podSelector: {} policyTypes: - Ingress - Egress ``` This blocks all ingress and egress traffic for pods in `team-c`. You then add specific allow rules. ### Test in a staging namespace first Never apply multi-tenancy changes directly to production namespaces. Use a copy of the production namespace with a suffix like `-staging`. After applying and verifying in staging, roll forward. Example flow: ```bash kubectl create namespace team-c-staging kubectl apply -f team-c-namespace.yaml -n team-c-staging kubectl apply -f team-c-quota.yaml -n team-c-staging kubectl apply -f role.yaml -n team-c-staging kubectl apply -f rolebinding.yaml -n team-c-staging kubectl apply -f networkpolicy.yaml -n team-c-staging ``` Then run tests to confirm isolation works as expected. ## Verification and Diagnostics After applying configuration, verify that each isolation mechanism works. Do not assume that because a manifest applied without errors, the policy is enforced correctly. Actively test boundaries. ### Verify namespace isolation Check that pods in one namespace cannot be listed from another namespace using a service account that lacks cross-namespace permissions. Create a service account in `team-a` and try to list pods in `team-b`: ```bash kubectl create serviceaccount sa-test -n team-a kubectl auth can-i list pods -n team-b --as system:serviceaccount:team-a:sa-test ``` Expected output: ``` no ``` If `yes`, RBAC permissions are too broad. ### Test resource quota enforcement Try to create a pod in `team-c` that exceeds the memory quota. Example pod with 20Gi memory request when the quota limit is 16Gi: ```yaml apiVersion: v1 kind: Pod metadata: name: test-memory-exceed namespace: team-c spec: containers: - name: nginx image: nginx resources: requests: memory: 20Gi limits: memory: 20Gi ``` Attempt to create: ```bash kubectl apply -f test-pod.yaml ``` Expected error: ``` Error from server (Forbidden): error when creating "test-pod.yaml": pods "test-memory-exceed" is forbidden: exceeded quota: team-c-quota, requested: limits.memory=20Gi, used: limits.memory=0, limited: limits.memory=16Gi ``` This proves the quota is working. ### Test network policy enforcement Deploy two pods in different namespaces and check connectivity. In `team-c`, run a simple web server pod: ```bash kubectl run web --image=nginx -n team-c kubectl expose pod web --port=80 -n team-c ``` From `team-b`, try to reach it: ```bash kubectl run test-pod --image=busybox -n team-b --restart=Never -- sleep 3600 kubectl exec -n team-b test-pod -- wget -qO- --timeout=5 http://web.team-c.svc.cluster.local ``` Expected result: the command times out or fails because the default-deny policy in `team-c` blocks ingress from `team-b`. ### Diagnostic commands for common issues When something fails, gather information systematically: 1. Check pod status: ```bash kubectl get pods -n team-c ``` If a pod is in `Pending`, it might be due to resource quota exhaustion or node pressure. 2. Describe the pod: ```bash kubectl describe pod -n team-c ``` Look at the `Events` section for messages like `FailedScheduling` or `Insufficient memory`. 3. Check pod logs: ```bash kubectl logs -n team-c --previous ``` This shows logs from a previous crashed container, useful for diagnosing crash loops. 4. Check deployment rollout status: ```bash kubectl rollout status deployment/ -n team-c ``` If the rollout is stuck, inspect replicas and events. 5. View resource quota usage: ```bash kubectl describe resourcequota team-c-quota -n team-c ``` This shows current usage versus limits, helping you see if a tenant is at capacity. ## Failure Modes and Recovery Even with careful planning, things go wrong. Here are common failure modes in Kubernetes multi-tenancy and how to recover. ### Quota exhaustion Symptom: New pods in a tenant namespace stay in `Pending` state with events like: ``` FailedScheduling: 0/3 nodes are available: 3 Insufficient cpu. ``` Or you get an error when creating a pod: ``` Error from server (Forbidden): pods "my-pod" is forbidden: exceeded quota: team-c-quota ``` Cause: The tenant has reached its CPU or memory limits. Recovery: - Identify which resource is exhausted: `kubectl describe resourcequota team-c-quota -n team-c`. - Check actual usage: `kubectl top pods -n team-c` (if metrics server installed). - Decide whether to increase the quota or scale down workloads. - To temporarily increase quota, edit the ResourceQuota: ```bash kubectl edit resourcequota team-c-quota -n team-c ``` Increase `requests.cpu` from `4` to `6` and save. But this should be a deliberate action, not a default. ### Cross-tenant network access Symptom: A pod in `team-b` can reach services in `team-a` even though you intended isolation. Cause: Missing or permissive NetworkPolicy, or a policy that allows all ingress. Recovery: - List NetworkPolicies in both namespaces: ```bash kubectl get networkpolicy -n team-a kubectl get networkpolicy -n team-b ``` - If none exist, apply default-deny policies. - If policies exist, inspect them: ```bash kubectl describe networkpolicy -n team-a ``` - Adjust the policy to deny cross-namespace traffic while allowing necessary communication (e.g., from ingress controller). ### Overly permissive RBAC Symptom: A user or service account can list secrets or delete pods in a namespace they should not have access to. Cause: RoleBinding assigned a ClusterRole like `admin` or `cluster-admin` to a broad group. Recovery: - Review role bindings: ```bash kubectl get rolebindings -n team-a kubectl get clusterrolebindings | grep team-a ``` - Find the subject that should not have access and remove the binding: ```bash kubectl delete rolebinding -n team-a ``` - Replace with a more restrictive Role or RoleBinding. ### Namespace deletion stuck Symptom: `kubectl delete namespace team-c` hangs and the namespace remains in `Terminating` state. Cause: Finalizers or resources in the namespace that cannot be deleted. Recovery: - Check namespace status: ```bash kubectl get namespace team-c -o yaml ``` - Look for `finalizers` field. If there is a custom finalizer, you may need to remove it after ensuring cleanup tasks are done. - List remaining resources: ```bash kubectl get all -n team-c ``` - Force deletion by patching the namespace to remove finalizers (use with extreme caution): ```bash kubectl patch namespace team-c -p '{"spec":{"finalizers":[]}}' --type=merge ``` This should only be done when you are sure no external resources depend on it. ## Operations Checklist Use this checklist before, during, and after making multi-tenancy changes. Assign an accountable owner to each item and review the checklist at least monthly, or after any incident. ### Pre-change checklist - [ ] Confirm correct cluster context: `kubectl config current-context`. Owner: Priya Shah, Engineering Lead. Review: monthly. - [ ] Backup current RBAC, quotas, and network policies: `kubectl get rolebindings,resourcequota,networkpolicy -n -o yaml > backup.yaml`. Owner: DevOps on-call. Review: weekly. - [ ] Identify blast radius: which other namespaces or services could be affected? Owner: DevOps on-call. Review: per change. - [ ] Prepare rollback plan: save original manifests in git. Owner: DevOps on-call. Review: per change. - [ ] Get approval from tenant owner if change affects their namespace. Owner: Tenant lead (e.g., Team A Lead). Review: per change. ### During change - [ ] Apply to staging namespace first: `kubectl apply -f manifest.yaml -n team-c-staging`. Owner: DevOps on-call. Review: immediate. - [ ] Verify expected output: run verification commands listed above. Owner: DevOps on-call. Review: immediate. - [ ] Monitor cluster-wide metrics for unexpected changes: `kubectl top pods --all-namespaces`. Owner: SRE on-call. Review: continuous. ### Post-change - [ ] Confirm resource quotas are enforced: try to create a pod that exceeds quota and expect failure. Owner: DevOps on-call. Review: after each change. - [ ] Test RBAC permissions: use `kubectl auth can-i` as a test service account. Owner: Security engineer (e.g., Marcus Lee). Review: quarterly. - [ ] Test network isolation: attempt cross-namespace connection from a test pod. Owner: Security engineer. Review: quarterly. - [ ] Document any deviations from expected behavior. Owner: DevOps on-call. Review: after each change. - [ ] Update runbooks with new recovery steps. Owner: Tech writer (e.g., Alex Kim). Review: monthly. ## Common Pitfalls and How to Avoid Them Avoid these frequent mistakes in Kubernetes multi-tenancy. ### Pitfall 1: Using a single namespace for all tenants Why it happens: Easier to manage at first, but leads to resource contention and security issues. How to avoid: Create a namespace per tenant from day one. Enforce with an admission controller or policy engine. Recovery: If already in a single namespace, plan a migration. Create new namespaces, set quotas and RBAC, then move workloads gradually. ### Pitfall 2: Ignoring resource requests and limits Why it happens: Teams omit them to simplify YAML, but this allows a noisy neighbor to consume all resources. How to avoid: Require resource requests and limits via admission policies (e.g., `LimitRange`). Example LimitRange: ```yaml apiVersion: v1 kind: LimitRange metadata: name: default-limits namespace: team-c spec: limits: - default: cpu: 500m memory: 512Mi defaultRequest: cpu: 250m memory: 256Mi type: Container ``` Apply and verify: `kubectl get limitrange -n team-c`. Recovery: Add LimitRange to existing namespaces and update deployments to include resources. ### Pitfall 3: Overly broad RBAC roles Why it happens: Reusing `cluster-admin` for convenience. How to avoid: Follow least privilege. Use `Role` for namespace-scoped permissions; never bind `cluster-admin` to a tenant's group. Recovery: Audit with `kubectl get clusterrolebindings -o yaml` and remove risky bindings. ### Pitfall 4: No network policies Why it happens: Default Kubernetes allows all pod-to-pod traffic, and many assume isolation is automatic. How to avoid: Define default-deny policies for each tenant namespace and explicitly allow required traffic. Recovery: Apply NetworkPolicies and test connectivity as described earlier. ### Pitfall 5: Not testing in staging Why it happens: Pressure to ship quickly leads to direct production changes. How to avoid: Make staging a requirement in your change process and use CI/CD to enforce it. Recovery: If a change breaks production, roll back to the previous manifest from version control: `kubectl apply -f previous-manifest.yaml` after reverting in git. ## Conclusion Kubernetes multi-tenancy commands are only useful when they are version-scoped, observable, and reversible. Copying commands without understanding prerequisites and expected output is not a reliable operational practice. Start with one low-risk verification: choose a tenant namespace, record its current state, run the documented checks, compare the result with the expected signal, and review dependencies like Namespace, Resource Quota, and RBAC. A reliable workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. By applying the commands and checklists in this article, you can maintain a stable and secure multi-tenant cluster.