## Introduction
Kubernetes resource requests and limits are essential for cluster stability, but misconfigured resources can lead to elusive networking problems. A pod starved of CPU may experience slow DNS lookups, while a memory-capped container may drop connections or fail to establish new ones. This guide provides a practical, command-driven approach to diagnose and resolve networking issues caused by improper resource requests and limits. We'll cover DNS resolution, port allocation, routing, firewall checks, connectivity tests, and safe diagnostic commands, all illustrated with concrete examples.
The article follows a systematic workflow: first, inventory your environment and versions; then, safely adjust resource configurations; verify with diagnostic commands; understand common failure modes; and adopt an operations checklist for ongoing management. By the end, you'll be equipped to identify and resolve networking problems arising from CPU throttling, memory limits, or misallocated resources in your Kubernetes cluster.
## Environment and Version Inventory
Before making any changes, document your Kubernetes environment. This helps reproduce issues and ensures compatibility. Use the following commands to gather essential information.
### Check Kubernetes Version
Run:
kubectl version --short
Expected output example:
Client Version: v1.24.0
Server Version: v1.24.0
Knowing both client and server versions is crucial because resource management features evolve between releases.
### Identify CNI Plugin and Version
The Container Network Interface (CNI) plugin handles pod networking. To find which CNI is installed and its version, inspect the relevant pod images. For Calico:
kubectl get pods -n kube-system -l k8s-app=calico-node -o jsonpath='{.items[0].spec.containers[0].image}'
Example output:
calico/node:v3.22.0
For other CNI plugins (Flannel, Weave, Cilium), adjust the label selector accordingly. Common labels:
- Flannel: app=flannel
- Weave: name=weave-net
- Cilium: k8s-app=cilium
### List Nodes and Resource Capacity
Check node capacity to understand resource availability:
kubectl get nodes -o custom-columns='NAME:.metadata.name,CPU:.status.capacity.cpu,MEMORY:.status.capacity.memory'
Example output:
NAME CPU MEMORY
node-1 4 16Gi
node-2 8 32Gi
This shows the total allocatable resources per node, which matters when pods are evicted due to resource pressure.
### List All Pods with Resource Requests and Limits
To see the current resource configurations of all pods:
kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,POD:.metadata.name,CPU_REQ:.spec.containers[*].resources.requests.cpu,CPU_LIM:.spec.containers[*].resources.limits.cpu,MEM_REQ:.spec.containers[*].resources.requests.memory,MEM_LIM:.spec.containers[*].resources.limits.memory'
This produces a wide table. For readability, filter with awk or use a tool like kubectl-neat . Example filtered output:
NAMESPACE POD CPU_REQ CPU_LIM MEM_REQ MEM_LIM
kube-system coredns-64897985 100m - 70Mi 170Mi
kube-system kube-proxy-abcde - - - -
default myapp-7c9d8b 200m 500m 128Mi 256Mi
Notice that some pods have no requests or limits; these are unrestricted and can consume unlimited resources, potentially starving others.
### Document Network Policies and Resource Quotas
Network policies can block traffic even when resources are adequate. List them:
kubectl get networkpolicies --all-namespaces
Check for resource quotas that may limit total resource usage in a namespace:
kubectl get resourcequota --all-namespaces
If a quota is set, it may prevent pods from being scheduled or cause evictions, indirectly affecting networking.
Prerequisites for this guide:
- kubectl configured with cluster access and appropriate permissions.
- Basic understanding of Kubernetes resources (pods, deployments, services).
- Permission to view and modify pod specs (or ability to delete/recreate pods).
This inventory forms the baseline for correlating resource constraints with network behavior.
## Safe Configuration Path
When adjusting resource requests and limits to resolve networking issues, proceed carefully to avoid service disruption. A staged approach minimizes risk.
### Step 1: Identify the Problematic Pod or Deployment
Use labels or names to narrow down. For example, if DNS lookups are slow, the issue may be with CoreDNS pods. Run:
kubectl get pods -n kube-system -l k8s-app=kube-dns
Note the pod names and current resource settings.
### Step 2: Capture Current Configuration
Before changing anything, back up the existing deployment:
kubectl get deployment myapp -o yaml > myapp-deployment-backup.yaml
This backup enables quick rollback if the changes cause new problems.
### Step 3: Use a Staged Approach
For critical workloads, first test changes on a canary deployment or with a temporary label. For example, if a pod is being CPU throttled causing slow DNS lookups and currently has no CPU limit, set a generous limit first, then gradually reduce if needed.
Patch the deployment with a JSON patch:
kubectl patch deployment myapp --type='json' -p='[{"op": "add", "path": "/spec/template/spec/containers/0/resources", "value": {"requests": {"cpu": "100m", "memory": "128Mi"}, "limits": {"cpu": "500m", "memory": "256Mi"}}}]'
This adds a resource block to the first container. If the deployment has multiple containers, adjust the index.
Alternatively, apply a modified YAML file with kubectl apply -f myapp-resources.yaml after editing.
### Step 4: Monitor the Rollout
Check the rollout status:
kubectl rollout status deployment/myapp
Expected output:
deployment "myapp" successfully rolled out
If the rollout fails, investigate with kubectl describe deployment myapp and kubectl get events .
### Step 5: Dry-Run Validation
Before applying changes to production, use dry-run to validate YAML:
kubectl apply -f myapp-resources.yaml --dry-run=client
This checks syntax without making changes.
### Step 6: Direct Pod Changes
For standalone pods (not managed by a controller), you cannot update resources in place. Delete and recreate the pod with updated spec. Example:
kubectl delete pod mypod
# Then apply a new pod spec with desired resources.
Always keep backups and use version control for manifests to enable quick rollback.
## Verification and Diagnostics
After adjusting resources, verify network functionality and diagnose if issues persist. Use these commands systematically.
### Check Pod Status and Resource Usage
Start with pod status:
kubectl get pods
Then check actual resource usage:
kubectl top pod myapp-pod
Example output:
NAME CPU(cores) MEMORY(bytes)
myapp-pod 50m 100Mi
Compare usage against limits. If usage is near limits, consider increasing them. For example, if CPU limit is 100m and usage is 90m, the pod is likely being throttled.
### Test DNS Resolution from Within a Pod
Create a temporary busybox pod for diagnostics:
kubectl run -it --rm debug --image=busybox --restart=Never -- sh
Inside the pod, run:
nslookup kubernetes.default
Expected output:
Server: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
Name: kubernetes.default
Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local
If DNS fails or times out, check the CoreDNS pods' CPU and memory. They may be evicted or throttled. Use kubectl top pods -n kube-system to see their usage.
### Test Connectivity to a Service
From the same busybox pod, test HTTP connectivity:
wget -qO- http://myservice.default.svc.cluster.local:8080
Expected: response from the service. If no response, check service endpoints:
kubectl get endpoints myservice
If endpoints are empty, the service selector may not match any healthy pods.
### Check Network Policies
Network policies might block traffic even if resources are fine. List policies in the namespace:
kubectl get networkpolicies -n default
Describe a specific policy to see its rules:
kubectl describe networkpolicy allow-web -n default
Look for ingress/egress rules that may be too restrictive.
### Inspect Pod Logs for Network Errors
Check recent logs:
kubectl logs myapp-pod --tail=50
Look for connection refused, timeout, or DNS resolution errors.
### Use Port Forwarding for Local Testing
To test connectivity from your local machine:
kubectl port-forward pod/myapp-pod 8080:80
Then open http://localhost:8080 in a browser or use curl . This bypasses service routing and helps isolate issues.
### Detect CPU Throttling
CPU throttling can be inferred from kubectl top if usage consistently hits the limit. For more detail, if you have node access, inspect cgroup stats:
cat /sys/fs/cgroup/cpu/cpu.stat
Look for nr_throttled and throttled_time . High values indicate throttling.
### Check Kube-Proxy Logs for Routing Issues
Routing issues may be due to kube-proxy. Inspect its logs:
kubectl logs -n kube-system kube-proxy-xxxxx
Replace with the actual pod name. Look for errors related to iptables or IPVS.
### Check for OOM Kills and Evictions
If pods are frequently evicted due to memory limits, check events:
kubectl describe pod myapp-pod | grep -A5 Events
Look for OOMKilled or Evicted messages. OOM kills can cause connection resets and failed requests.
These diagnostics help pinpoint whether resource constraints are causing network failures.
## Common Failure Modes and Recovery
Understanding typical failure patterns speeds up troubleshooting. Here are common scenarios with symptoms and recovery steps.
### CPU Throttling Leading to Slow DNS Lookups
Symptom: Intermittent DNS resolution failures or high latency. Applications report timeouts when connecting to services.
Cause: CoreDNS pods may be CPU throttled under load, especially if limits are set too low.
Recovery: Increase CPU limit for CoreDNS. Edit the deployment:
kubectl edit deployment coredns -n kube-system
Change the container's resources, for example from:
resources:
limits:
cpu: 200m
memory: 170Mi
requests:
cpu: 100m
memory: 70Mi
to:
resources:
limits:
cpu: 500m
memory: 300Mi
requests:
cpu: 200m
memory: 150Mi
Save and exit. Monitor CoreDNS CPU usage with kubectl top pods -n kube-system .
### Memory Limit Causing OOM Kills
Symptom: Pod restarts frequently, connections dropped, service disruption. kubectl get pods shows RESTARTS count increasing.
Cause: The container's memory limit is too low for its workload, leading to out-of-memory (OOM) kills by the kernel.
Recovery: Increase memory limit or fix memory leak in the application. Check the pod's last state:
kubectl describe pod myapp-pod | grep -A10 'Last State'
If it shows OOMKilled , increase the memory limit in the deployment spec.
### Pod Eviction Due to Node Pressure
Symptom: Pods are evicted from nodes, network endpoints disappear, services become unreachable.
Cause: Node is under memory or disk pressure, and Kubernetes evicts pods to reclaim resources.
Recovery: Adjust resource requests to match actual usage to improve scheduling, or add node capacity. Check node conditions:
kubectl describe node node-1 | grep -A5 Conditions
If MemoryPressure or DiskPressure is True , investigate and address.
### Network Policy Misconfiguration
Symptom: Traffic blocked between pods even when resources are fine. Connections time out or are refused.
Cause: Network policies may be too restrictive.
Recovery: Review and modify network policies. Use kubectl describe networkpolicy to see rules, and adjust selectors and ingress/egress rules.
### DNS Resolution Failure Due to CoreDNS Resource Starvation
Symptom: DNS queries time out consistently. nslookup from a pod hangs.
Cause: CoreDNS pods may have insufficient resources or have been evicted, leaving no DNS service.
Recovery: Check CoreDNS pod status:
kubectl get pods -n kube-system -l k8s-app=kube-dns
If pods are missing or crash-looping, scale up or adjust resources. Consider using Horizontal Pod Autoscaler (HPA) for CoreDNS:
kubectl autoscale deployment coredns -n kube-system --cpu-percent=80 --min=2 --max=5
### Rollback Procedures
If resource changes cause issues, revert to the previous configuration promptly.
- For deployments: kubectl rollout undo deployment/myapp
- For manually created pods: delete and recreate with original spec from backup.
- For CoreDNS: kubectl rollout undo deployment coredns -n kube-system
After rollback, verify networking is restored with the tests from the previous section. Continue monitoring for recurring issues.
## Operations Checklist
Prevent networking issues caused by resource misconfiguration with this ongoing checklist.
### Monitoring and Alerting
- Run kubectl top pods --all-namespaces regularly, at least daily.
- Set alerts for pods approaching their limits (e.g., >80% CPU or memory for sustained periods). Use Prometheus and Alertmanager if available.
- Monitor node resource usage: kubectl top nodes .
### Review Resource Settings Periodically
- Review resource requests and limits every quarter or after significant application changes.
- Compare actual usage ( kubectl top ) with configured requests/limits; adjust to avoid over- or under-provisioning.
### DNS Reliability
- Ensure CoreDNS has at least 2 replicas for high availability. Scale up with cluster size: kubectl scale deployment coredns -n kube-system --replicas=3 for larger clusters.
- Test DNS resolution after any cluster upgrade or change:
kubectl run -it --rm dns-test --image=busybox --restart=Never -- nslookup kubernetes.default
### Network Policy Validation
- Maintain a connectivity test matrix. For each service-to-service communication, run periodic tests from a debug pod.
- After modifying network policies, execute the test matrix to ensure no unintended blocks.
### Version Control and Documentation
- Keep all manifests in version control (Git). Review changes before merging.
- Document standard resource values for common application types. For example:
Application Type CPU Request CPU Limit Memory Request Memory Limit
Web frontend 100m 500m 128Mi 256Mi
API backend 250m 1 256Mi 512Mi
Database 500m 2 1Gi 2Gi
### Enforce Defaults with LimitRange and ResourceQuota
Use LimitRange to enforce default requests and limits for containers that don't specify them. Example:
apiVersion: v1
kind: LimitRange
metadata:
name: default-limits
namespace: default
spec:
limits:
- default:
cpu: 500m
memory: 256Mi
defaultRequest:
cpu: 100m
memory: 128Mi
type: Container
Apply with:
kubectl apply -f limitrange.yaml
Use ResourceQuota to limit total resource usage in a namespace and prevent overcommitment:
apiVersion: v1
kind: ResourceQuota
metadata:
name: namespace-quota
namespace: default
spec:
hard:
requests.cpu: "4"
requests.memory: "8Gi"
limits.cpu: "8"
limits.memory: "16Gi"
Apply with kubectl apply -f resourcequota.yaml .
### Team Training and Communication
- Train developers on setting appropriate requests/limits. Emphasize the impact on networking.
- Share this troubleshooting guide with the operations team.
Use this checklist to proactively manage resources and prevent networking issues.
## Conclusion
Kubernetes resource requests and limits directly impact network performance and reliability. CPU throttling can slow DNS lookups, memory limits can cause OOM kills and dropped connections, and pod evictions can disrupt service endpoints. By following a systematic approach—starting with a thorough environment inventory, making safe configuration changes, verifying with concrete diagnostic commands, and preparing rollback procedures—you can effectively diagnose and resolve these issues.
Adopt the operations checklist to maintain a healthy networking environment. Regular monitoring, periodic resource reviews, and proper defaults with LimitRange and ResourceQuota help prevent problems before they affect your applications. With these practices, your Kubernetes cluster will deliver reliable and efficient networking.