Troubleshooting Kubernetes Resource Requests and Limits Networking Issues: A Practical Guide

Introduction

Kubernetes resource requests and limits are essential for cluster stability, but misconfigured resources can lead to elusive networking problems. A pod starved of CPU may experience slow DNS lookups, while a memory-capped container may drop connections or fail to establish new ones. This guide provides a practical, command-driven approach to diagnose and resolve networking issues caused by improper resource requests and limits. We'll cover DNS resolution, port allocation, routing, firewall checks, connectivity tests, and safe diagnostic commands, all illustrated with concrete examples.

The article follows a systematic workflow: first, inventory your environment and versions; then, safely adjust resource configurations; verify with diagnostic commands; understand common failure modes; and adopt an operations checklist for ongoing management. By the end, you'll be equipped to identify and resolve networking problems arising from CPU throttling, memory limits, or misallocated resources in your Kubernetes cluster.

Environment and Version Inventory

Before making any changes, document your Kubernetes environment. This helps reproduce issues and ensures compatibility. Use the following commands to gather essential information.

Check Kubernetes Version

Run:

kubectl version --short

Expected output example:

Client Version: v1.24.0
Server Version: v1.24.0

Knowing both client and server versions is crucial because resource management features evolve between releases.

Identify CNI Plugin and Version

The Container Network Interface (CNI) plugin handles pod networking. To find which CNI is installed and its version, inspect the relevant pod images. For Calico:

kubectl get pods -n kube-system -l k8s-app=calico-node -o jsonpath='{.items[0].spec.containers[0].image}'

Example output:

calico/node:v3.22.0

For other CNI plugins (Flannel, Weave, Cilium), adjust the label selector accordingly. Common labels:

• Flannel: app=flannel
• Weave: name=weave-net
• Cilium: k8s-app=cilium

List Nodes and Resource Capacity

Check node capacity to understand resource availability:

kubectl get nodes -o custom-columns='NAME:.metadata.name,CPU:.status.capacity.cpu,MEMORY:.status.capacity.memory'

Example output:

NAME    CPU   MEMORY
node-1  4     16Gi
node-2  8     32Gi

This shows the total allocatable resources per node, which matters when pods are evicted due to resource pressure.

List All Pods with Resource Requests and Limits

To see the current resource configurations of all pods:

kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,POD:.metadata.name,CPU_REQ:.spec.containers[*].resources.requests.cpu,CPU_LIM:.spec.containers[*].resources.limits.cpu,MEM_REQ:.spec.containers[*].resources.requests.memory,MEM_LIM:.spec.containers[*].resources.limits.memory'

This produces a wide table. For readability, filter with awk or use a tool like kubectl-neat. Example filtered output:

NAMESPACE   POD                CPU_REQ   CPU_LIM   MEM_REQ   MEM_LIM
kube-system coredns-64897985  100m      -         70Mi      170Mi
kube-system kube-proxy-abcde  -         -         -         -
default     myapp-7c9d8b    200m      500m      128Mi     256Mi

Notice that some pods have no requests or limits; these are unrestricted and can consume unlimited resources, potentially starving others.

Document Network Policies and Resource Quotas

Network policies can block traffic even when resources are adequate. List them:

kubectl get networkpolicies --all-namespaces

Check for resource quotas that may limit total resource usage in a namespace:

kubectl get resourcequota --all-namespaces

If a quota is set, it may prevent pods from being scheduled or cause evictions, indirectly affecting networking.

Prerequisites for this guide:

• kubectl configured with cluster access and appropriate permissions.
• Basic understanding of Kubernetes resources (pods, deployments, services).
• Permission to view and modify pod specs (or ability to delete/recreate pods).

This inventory forms the baseline for correlating resource constraints with network behavior.

Safe Configuration Path

When adjusting resource requests and limits to resolve networking issues, proceed carefully to avoid service disruption. A staged approach minimizes risk.

Step 1: Identify the Problematic Pod or Deployment

Use labels or names to narrow down. For example, if DNS lookups are slow, the issue may be with CoreDNS pods. Run:

kubectl get pods -n kube-system -l k8s-app=kube-dns

Note the pod names and current resource settings.

Step 2: Capture Current Configuration

Before changing anything, back up the existing deployment:

kubectl get deployment myapp -o yaml > myapp-deployment-backup.yaml

This backup enables quick rollback if the changes cause new problems.

Step 3: Use a Staged Approach

For critical workloads, first test changes on a canary deployment or with a temporary label. For example, if a pod is being CPU throttled causing slow DNS lookups and currently has no CPU limit, set a generous limit first, then gradually reduce if needed.

Patch the deployment with a JSON patch:

kubectl patch deployment myapp --type='json' -p='[{"op": "add", "path": "/spec/template/spec/containers/0/resources", "value": {"requests": {"cpu": "100m", "memory": "128Mi"}, "limits": {"cpu": "500m", "memory": "256Mi"}}}]'

This adds a resource block to the first container. If the deployment has multiple containers, adjust the index.

Alternatively, apply a modified YAML file with kubectl apply -f myapp-resources.yaml after editing.

Step 4: Monitor the Rollout

Check the rollout status:

kubectl rollout status deployment/myapp

Expected output:

deployment "myapp" successfully rolled out

If the rollout fails, investigate with kubectl describe deployment myapp and kubectl get events.

Step 5: Dry-Run Validation

Before applying changes to production, use dry-run to validate YAML:

kubectl apply -f myapp-resources.yaml --dry-run=client

This checks syntax without making changes.

Step 6: Direct Pod Changes

For standalone pods (not managed by a controller), you cannot update resources in place. Delete and recreate the pod with updated spec. Example:

kubectl delete pod mypod
# Then apply a new pod spec with desired resources.

Always keep backups and use version control for manifests to enable quick rollback.

Verification and Diagnostics

After adjusting resources, verify network functionality and diagnose if issues persist. Use these commands systematically.

Check Pod Status and Resource Usage

Start with pod status:

kubectl get pods

Then check actual resource usage:

kubectl top pod myapp-pod

Example output:

NAME        CPU(cores)   MEMORY(bytes)
myapp-pod   50m          100Mi

Compare usage against limits. If usage is near limits, consider increasing them. For example, if CPU limit is 100m and usage is 90m, the pod is likely being throttled.

Test DNS Resolution from Within a Pod

Create a temporary busybox pod for diagnostics:

kubectl run -it --rm debug --image=busybox --restart=Never -- sh

Inside the pod, run:

nslookup kubernetes.default

Expected output:

Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes.default
Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local

If DNS fails or times out, check the CoreDNS pods' CPU and memory. They may be evicted or throttled. Use kubectl top pods -n kube-system to see their usage.

Test Connectivity to a Service

From the same busybox pod, test HTTP connectivity:

wget -qO- http://myservice.default.svc.cluster.local:8080

Expected: response from the service. If no response, check service endpoints:

kubectl get endpoints myservice

If endpoints are empty, the service selector may not match any healthy pods.

Check Network Policies

Network policies might block traffic even if resources are fine. List policies in the namespace:

kubectl get networkpolicies -n default

Describe a specific policy to see its rules:

kubectl describe networkpolicy allow-web -n default

Look for ingress/egress rules that may be too restrictive.

Inspect Pod Logs for Network Errors

Check recent logs:

kubectl logs myapp-pod --tail=50

Look for connection refused, timeout, or DNS resolution errors.

Use Port Forwarding for Local Testing

To test connectivity from your local machine:

kubectl port-forward pod/myapp-pod 8080:80

Then open http://localhost:8080 in a browser or use curl. This bypasses service routing and helps isolate issues.

Detect CPU Throttling

CPU throttling can be inferred from kubectl top if usage consistently hits the limit. For more detail, if you have node access, inspect cgroup stats:

cat /sys/fs/cgroup/cpu/cpu.stat

Look for nr_throttled and throttled_time. High values indicate throttling.

Check Kube-Proxy Logs for Routing Issues

Routing issues may be due to kube-proxy. Inspect its logs:

kubectl logs -n kube-system kube-proxy-xxxxx

Replace with the actual pod name. Look for errors related to iptables or IPVS.

Check for OOM Kills and Evictions

If pods are frequently evicted due to memory limits, check events:

kubectl describe pod myapp-pod | grep -A5 Events

Look for OOMKilled or Evicted messages. OOM kills can cause connection resets and failed requests.

These diagnostics help pinpoint whether resource constraints are causing network failures.

Common Failure Modes and Recovery

Understanding typical failure patterns speeds up troubleshooting. Here are common scenarios with symptoms and recovery steps.

CPU Throttling Leading to Slow DNS Lookups

Symptom: Intermittent DNS resolution failures or high latency. Applications report timeouts when connecting to services.

Cause: CoreDNS pods may be CPU throttled under load, especially if limits are set too low.

Recovery: Increase CPU limit for CoreDNS. Edit the deployment:

kubectl edit deployment coredns -n kube-system

Change the container's resources, for example from:

resources:
  limits:
    cpu: 200m
    memory: 170Mi
  requests:
    cpu: 100m
    memory: 70Mi

to:

resources:
  limits:
    cpu: 500m
    memory: 300Mi
  requests:
    cpu: 200m
    memory: 150Mi

Save and exit. Monitor CoreDNS CPU usage with kubectl top pods -n kube-system.

Memory Limit Causing OOM Kills

Symptom: Pod restarts frequently, connections dropped, service disruption. kubectl get pods shows RESTARTS count increasing.

Cause: The container's memory limit is too low for its workload, leading to out-of-memory (OOM) kills by the kernel.

Recovery: Increase memory limit or fix memory leak in the application. Check the pod's last state:

kubectl describe pod myapp-pod | grep -A10 'Last State'

If it shows OOMKilled, increase the memory limit in the deployment spec.

Pod Eviction Due to Node Pressure

Symptom: Pods are evicted from nodes, network endpoints disappear, services become unreachable.

Cause: Node is under memory or disk pressure, and Kubernetes evicts pods to reclaim resources.

Recovery: Adjust resource requests to match actual usage to improve scheduling, or add node capacity. Check node conditions:

kubectl describe node node-1 | grep -A5 Conditions

If MemoryPressure or DiskPressure is True, investigate and address.

Network Policy Misconfiguration

Symptom: Traffic blocked between pods even when resources are fine. Connections time out or are refused.

Cause: Network policies may be too restrictive.

Recovery: Review and modify network policies. Use kubectl describe networkpolicy to see rules, and adjust selectors and ingress/egress rules.

DNS Resolution Failure Due to CoreDNS Resource Starvation

Symptom: DNS queries time out consistently. nslookup from a pod hangs.

Cause: CoreDNS pods may have insufficient resources or have been evicted, leaving no DNS service.

Recovery: Check CoreDNS pod status:

kubectl get pods -n kube-system -l k8s-app=kube-dns

If pods are missing or crash-looping, scale up or adjust resources. Consider using Horizontal Pod Autoscaler (HPA) for CoreDNS:

kubectl autoscale deployment coredns -n kube-system --cpu-percent=80 --min=2 --max=5

Rollback Procedures

If resource changes cause issues, revert to the previous configuration promptly.

• For deployments: kubectl rollout undo deployment/myapp
• For manually created pods: delete and recreate with original spec from backup.
• For CoreDNS: kubectl rollout undo deployment coredns -n kube-system

After rollback, verify networking is restored with the tests from the previous section. Continue monitoring for recurring issues.

Operations Checklist

Prevent networking issues caused by resource misconfiguration with this ongoing checklist.

Monitoring and Alerting

• Run kubectl top pods --all-namespaces regularly, at least daily.
• Set alerts for pods approaching their limits (e.g., >80% CPU or memory for sustained periods). Use Prometheus and Alertmanager if available.
• Monitor node resource usage: kubectl top nodes.

Review Resource Settings Periodically

• Review resource requests and limits every quarter or after significant application changes.
• Compare actual usage (kubectl top) with configured requests/limits; adjust to avoid over- or under-provisioning.

DNS Reliability

• Ensure CoreDNS has at least 2 replicas for high availability. Scale up with cluster size: kubectl scale deployment coredns -n kube-system --replicas=3 for larger clusters.
• Test DNS resolution after any cluster upgrade or change:

kubectl run -it --rm dns-test --image=busybox --restart=Never -- nslookup kubernetes.default

Network Policy Validation

• Maintain a connectivity test matrix. For each service-to-service communication, run periodic tests from a debug pod.
• After modifying network policies, execute the test matrix to ensure no unintended blocks.

Version Control and Documentation

• Keep all manifests in version control (Git). Review changes before merging.
• Document standard resource values for common application types. For example:

Enforce Defaults with LimitRange and ResourceQuota

Use LimitRange to enforce default requests and limits for containers that don't specify them. Example:

apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
  namespace: default
spec:
  limits:
  - default:
      cpu: 500m
      memory: 256Mi
    defaultRequest:
      cpu: 100m
      memory: 128Mi
    type: Container

Apply with:

kubectl apply -f limitrange.yaml

Use ResourceQuota to limit total resource usage in a namespace and prevent overcommitment:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: namespace-quota
  namespace: default
spec:
  hard:
    requests.cpu: "4"
    requests.memory: "8Gi"
    limits.cpu: "8"
    limits.memory: "16Gi"

Apply with kubectl apply -f resourcequota.yaml.

Team Training and Communication

• Train developers on setting appropriate requests/limits. Emphasize the impact on networking.
• Share this troubleshooting guide with the operations team.

Use this checklist to proactively manage resources and prevent networking issues.

Conclusion

Kubernetes resource requests and limits directly impact network performance and reliability. CPU throttling can slow DNS lookups, memory limits can cause OOM kills and dropped connections, and pod evictions can disrupt service endpoints. By following a systematic approach—starting with a thorough environment inventory, making safe configuration changes, verifying with concrete diagnostic commands, and preparing rollback procedures—you can effectively diagnose and resolve these issues.

Adopt the operations checklist to maintain a healthy networking environment. Regular monitoring, periodic resource reviews, and proper defaults with LimitRange and ResourceQuota help prevent problems before they affect your applications. With these practices, your Kubernetes cluster will deliver reliable and efficient networking.