## Intro Kubernetes Ingress is a powerful resource for routing external HTTP and HTTPS traffic to services inside a cluster. But when an Ingress misbehaves, the symptoms can be confusing: 404s, timeouts, TLS errors, or traffic going to the wrong backend. Troubleshooting Ingress requires a systematic approach: understand the components involved, check configuration, inspect logs, and verify connectivity step by step. This guide provides practical examples for diagnosing and fixing common Ingress failures. We will cover environment inventory, safe configuration changes, verification commands, failure modes, and recovery steps. Every example includes concrete kubectl commands and expected outputs. By the end, you will have a repeatable troubleshooting workflow. ## Version and Environment Inventory Before troubleshooting, collect information about your cluster, Ingress controller, and related resources. This establishes a baseline and helps identify mismatches. A common pitfall is assuming the Ingress controller is healthy when it is not, or using an outdated kubectl client that misreports resource versions. Start with a clean inventory. Prerequisites: - Access to a Kubernetes cluster with kubectl configured. - Permission to get and describe resources in the relevant namespaces. - Basic understanding of Kubernetes Services, Deployments, and Ingress. Commands to run: ```bash kubectl version --short ``` Expected output (example): ``` Client Version: v1.28.2 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.28.3 ``` If client and server versions differ significantly (e.g., client is 1.22 and server is 1.28), some kubectl output fields may be missing or deprecated. Upgrade your client to match the server minor version whenever possible. ```bash kubectl get nodes -o wide ``` This shows node status, Kubernetes version, and internal/external IPs. Look for any node in NotReady state, which can affect Ingress controller scheduling. ```bash kubectl get pods -n ingress-nginx ``` Replace `ingress-nginx` with your Ingress controller namespace. Common namespaces include `ingress-nginx`, `kube-system`, `traefik`, `istio-system`, or `contour`. Expected output shows running pods, e.g.: ``` NAME READY STATUS RESTARTS AGE ingress-nginx-controller-6d5c7b8f9b-abcde 1/1 Running 0 5d ``` If the controller pod is not running, describe it to see why: ```bash kubectl describe pod -n ingress-nginx ``` Check the IngressClass: ```bash kubectl get ingressclass ``` Example output: ``` NAME CONTROLLER PARAMETERS AGE nginx k8s.io/ingress-nginx 10d ``` Identify the Ingress resource and its configuration: ```bash kubectl get ingress -A kubectl describe ingress -n ``` The describe output shows rules, backend services, TLS settings, and annotations. Verify that the IngressClass matches the controller's class (or that the Ingress uses the default class if `ingressClassName` is omitted). A mismatch here is a frequent cause of Ingress being ignored. Topology checklist: - Ingress controller is running and healthy. - Ingress resource exists in the expected namespace. - Backend Service exists and selects correct pods. - Service has endpoints (pods ready). - TLS secret exists if HTTPS is used. - Network policies allow traffic from Ingress controller to backend pods. ## Safe Configuration Path When making changes, proceed incrementally. First, test with a simple Ingress before modifying complex routing. This mirrors the practice of starting with a narrow, measurable change and inspecting locally before broader rollout. Avoid editing production Ingress resources directly via `kubectl edit`; instead, export the current manifest, modify it in a file, and apply with a diff check. Example minimal Ingress: Create a test deployment and service: ```bash kubectl create deployment web --image=nginx:1.25 --port=80 -n test kubectl expose deployment web --port=80 --target-port=80 -n test ``` Create an Ingress for it (save as `ingress.yaml`): ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: web-ingress namespace: test annotations: nginx.ingress.kubernetes.io/rewrite-target: / spec: ingressClassName: nginx rules: - host: test.example.com http: paths: - path: / pathType: Prefix backend: service: name: web port: number: 80 ``` Apply and test: ```bash kubectl apply -f ingress.yaml curl -H "Host: test.example.com" http:/// ``` If the Ingress controller is exposed via a LoadBalancer, get its IP: ```bash kubectl get svc -n ingress-nginx ingress-nginx-controller ``` Expected output shows an external IP. Use that IP in curl. If you don't have an external IP, you can test using port-forward: ```bash kubectl port-forward -n ingress-nginx svc/ingress-nginx-controller 8080:80 ``` Then curl `http://localhost:8080/` with the Host header. If you need to change an existing Ingress, first back up the current manifest: ```bash kubectl get ingress -n -o yaml > ingress-backup.yaml ``` Then edit, apply, and verify. Use `kubectl diff` to preview changes: ```bash kubectl diff -f updated-ingress.yaml ``` This shows what would change without applying. Always review the diff before applying to catch unintended modifications. ## Verification and Diagnostics After configuration, verify that traffic flows correctly. Use multiple commands to check each layer. Treat the Ingress controller as a black box initially: confirm that requests reach it and that it can reach backends, then narrow down. ### 1. Check Ingress resource status ```bash kubectl describe ingress web-ingress -n test ``` Look for events: successful sync or errors. Example of a healthy Ingress: ``` Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Sync 2m nginx-ingress-controller Scheduled for sync ``` If there are errors, they appear here. Common errors include "endpoint not found" or "service not found". ### 2. Check Ingress controller logs ```bash kubectl logs -n ingress-nginx deployment/ingress-nginx-controller --tail=50 ``` Adjust namespace and deployment name. Look for lines containing your Ingress name, host, or error. Example error: ``` E0301 12:00:00.123456 1 controller.go:123] Unexpected error ingressing: service "test/web" is type "ClusterIP", expected "NodePort" or "LoadBalancer" ``` With most modern Ingress controllers, ClusterIP services are valid; this error may indicate a misconfigured controller or an outdated version. Check controller documentation. To filter logs for a specific host: ```bash kubectl logs -n ingress-nginx deployment/ingress-nginx-controller --tail=200 | grep "test.example.com" ``` ### 3. Verify backend service and endpoints ```bash kubectl get svc web -n test kubectl get endpoints web -n test ``` Expected: ``` NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE web ClusterIP 10.96.123.45 80/TCP 10m NAME ENDPOINTS AGE web 10.244.0.10:80 10m ``` If endpoints are empty, the service selector doesn't match pods or pods are not ready. Check pod labels: ```bash kubectl get pods -n test --show-labels ``` Compare with service selector: ```bash kubectl describe svc web -n test ``` ### 4. Test from inside the cluster Run a temporary pod with curl: ```bash kubectl run curl-test --image=curlimages/curl -i --rm --restart=Never -- curl -v -H "Host: test.example.com" http://ingress-nginx-controller.ingress-nginx.svc.cluster.local/ ``` This tests internal routing to the controller's ClusterIP service. Check response status and headers. If this fails, the problem is inside the cluster (controller, service, or backend). If this succeeds but external access fails, focus on external networking (LoadBalancer, DNS, firewall). ### 5. Test TLS (if applicable) ```bash kubectl get secret tls-secret -n test -o yaml ``` Verify secret contains `tls.crt` and `tls.key`. Then test with curl using `--resolve` to map host to ingress IP: ```bash curl -v --resolve test.example.com:443: https://test.example.com/ ``` Check for certificate errors. Use `openssl` to examine the certificate: ```bash kubectl get secret tls-secret -n test -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -dates -subject ``` ### Expected results for a working setup - Ingress describe shows a Sync event. - Ingress controller logs show no errors for the host. - Endpoints contain pod IPs. - Curl returns HTTP 200 and the expected content. - TLS handshake succeeds (no certificate warnings). If any of these fail, proceed to the failure modes section below. ## Common Failure Modes and Recovery This section covers the most frequent Ingress failures, with symptoms, diagnosis, and recovery steps. Each failure mode includes concrete commands and expected outputs. ### Failure Mode 1: 404 Not Found **Symptoms:** Curl returns 404 even though the path seems correct. **Possible causes:** - Ingress path does not match request path (e.g., path is `/api` but request is `/api/` due to `pathType` mismatch). - `rewrite-target` annotation misconfigured, causing backend to receive unexpected path. - Backend service itself returns 404 for the path. **Diagnosis:** Check Ingress rules: ```bash kubectl get ingress web-ingress -n test -o yaml ``` Look at `path` and `pathType`. Ingress-nginx default `pathType` is `Prefix`. With `Prefix`, `/api` matches `/api`, `/api/`, `/api/v1`, etc. With `Exact`, only `/api` matches. If you need exact matching but used `Prefix`, change it. Check backend service directly from a pod: ```bash kubectl exec -it -n test -- curl http://web.test.svc.cluster.local/ ``` If direct curl works but Ingress returns 404, Ingress routing is wrong. **Recovery:** - Adjust Ingress path or `pathType` to match the requested URL. - If using `rewrite-target`, test without it first. Misconfigured rewrite can strip or add path segments unexpectedly. - Apply changes and re-test: ```bash kubectl apply -f updated-ingress.yaml curl -H "Host: test.example.com" http:/// ``` ### Failure Mode 2: 503 Service Unavailable **Symptoms:** Ingress returns 503. **Possible causes:** - Backend service has no endpoints (pods not ready). - Service name or port mismatch in Ingress. - Network policy blocking traffic from Ingress controller to pods. - Readiness probe failing, so pods are not marked ready. **Diagnosis:** Check endpoints: ```bash kubectl get endpoints -n ``` If empty, check pods and their labels: ```bash kubectl get pods -n test --show-labels kubectl describe svc web -n test ``` Ensure service selector matches pod labels exactly. Note that a service selector with `app: web` will not match pods with `app: web-backend`. Check Ingress backend definition: ```bash kubectl describe ingress web-ingress -n test ``` Verify service name and port. In the example, the backend should be `serviceName: web` and `servicePort: 80`. Check Ingress controller logs for error: ```bash kubectl logs -n ingress-nginx deployment/ingress-nginx-controller | grep -i "test/web" ``` Example: ``` W0301 12:00:00.123456 1 controller.go:456] Service "test/web" does not have any active Endpoint. ``` **Recovery:** - Fix service selector to match pods. - Ensure pods are ready: check `kubectl get pods -n test` and look at READY column. If 0/1, describe pod and check readiness probe logs. - Correct service name/port in Ingress. - If network policy is suspected, temporarily allow all traffic in the namespace for testing, then tighten. ### Failure Mode 3: TLS Certificate Errors **Symptoms:** Browser shows certificate warning or curl reports SSL certificate problem. **Possible causes:** - TLS secret missing or malformed. - Certificate hostname mismatch. - Certificate expired. - Secret not in the same namespace as Ingress. **Diagnosis:** Check secret: ```bash kubectl get secret tls-secret -n test ``` If not found: ``` Error from server (NotFound): secrets "tls-secret" not found ``` Check certificate details: ```bash kubectl get secret tls-secret -n test -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -dates -subject ``` Expected output shows validity dates and subject CN/SAN. Verify that the SAN includes the hostname used in Ingress. **Recovery:** - Create a valid secret: ```bash kubectl create secret tls tls-secret --cert=path/to/tls.crt --key=path/to/tls.key -n test ``` - Ensure the certificate includes the hostname in SAN (not just CN). - Renew if expired. - Ensure the secret is in the same namespace as the Ingress resource. ### Failure Mode 4: Connection Refused or Timeout **Symptoms:** Curl cannot connect; connection refused or times out. **Possible causes:** - Ingress controller service not reachable externally (e.g., LoadBalancer IP not assigned, firewall blocks). - Ingress controller pod not running. - Network policy blocking traffic to controller. - DNS resolution failure for the host. **Diagnosis:** Check Ingress controller service: ```bash kubectl get svc -n ingress-nginx ``` Look for EXTERNAL-IP. If ``, the LoadBalancer may still be provisioning, or your cluster may not support LoadBalancer (e.g., bare metal). Check service events: ```bash kubectl describe svc ingress-nginx-controller -n ingress-nginx ``` Check controller pods: ```bash kubectl get pods -n ingress-nginx ``` If pods are not running, describe them: ```bash kubectl describe pod -n ingress-nginx ``` Look for CrashLoopBackOff, ImagePullBackOff, or resource issues. **Recovery:** - Wait for LoadBalancer IP or use NodePort / port-forward for testing. - Fix pod issues (e.g., resource limits, config errors). - Check cloud provider firewall or security groups. Allow inbound traffic on ports 80/443 from your IP. - Verify DNS record points to the correct IP or use `--resolve` to bypass DNS. ### Failure Mode 5: Routing to Wrong Backend **Symptoms:** Request is served by unexpected service. **Possible causes:** - Multiple Ingress resources with same host but different paths, and path matching order leads to wrong backend. - Default backend configuration. - Annotation misconfiguration (e.g., canary weights). **Diagnosis:** List all Ingress resources with the host: ```bash kubectl get ingress -A | grep ``` Inspect each Ingress's rules. Ingress controllers merge rules from all Ingress resources in the cluster, so conflicts can occur. Ingress-nginx uses a priority system: longest path match wins, and if equal, creation timestamp decides. This can lead to unexpected routing. Check for default backend: ```bash kubectl get ingress -n test web-ingress -o yaml ``` If `spec.defaultBackend` is set, any request not matching a rule goes there. **Recovery:** - Consolidate rules into a single Ingress for the host to avoid ambiguity. - Or adjust path specificity so the longest intended path wins. - Use different hosts if possible. - Explicitly set `ingressClassName` to ensure the correct controller processes the Ingress. ### Rollback and Recovery Checks Always keep backups of Ingress manifests. If a change causes issues, rollback: ```bash kubectl apply -f ingress-backup.yaml ``` Then verify with curl and describe. Also, consider using `kubectl rollout undo` for Deployments if pod changes caused the issue: ```bash kubectl rollout undo deployment/web -n test ``` ## Common Pitfalls and How to Avoid Them Beyond the specific failure modes, several recurring mistakes can make Ingress troubleshooting harder. Here are the most common pitfalls and their solutions. ### Pitfall 1: Ignoring IngressClass Many users omit `ingressClassName` or set it incorrectly, assuming the default controller will handle the Ingress. If multiple controllers exist or the default class is not set, the Ingress may be ignored entirely. **How to avoid:** Always set `ingressClassName` explicitly in your Ingress manifest. Verify the class exists with `kubectl get ingressclass`. ### Pitfall 2: Misunderstanding pathType The difference between `Prefix` and `Exact` path types causes many 404s. For example, with `pathType: Exact` and `path: /foo`, a request to `/foo/` will not match, returning 404. With `Prefix`, `/foo` matches `/foo`, `/foo/`, and `/foo/bar`. **How to avoid:** Choose `Prefix` for most applications unless you need exact path matching. Test with and without trailing slashes. ### Pitfall 3: Rewrite-target annotation misuse The `rewrite-target` annotation is powerful but frequently misconfigured. Setting `rewrite-target: /` for an Ingress with path `/api` will strip `/api` and send the remaining path to the backend. If not intended, the backend receives unexpected paths and may return 404 or 500. **How to avoid:** Test the rewrite behavior by examining backend access logs or using a simple echo server. Only use rewrite when you understand the target path semantics. ### Pitfall 4: Not checking endpoints When a service has no endpoints, the Ingress controller will return 503. Users often spend time checking Ingress rules when the real issue is that pods are not ready or labels don't match. **How to avoid:** Make endpoint verification a standard step in your troubleshooting checklist. Use `kubectl get endpoints` immediately after checking Ingress. ### Pitfall 5: Overlooking network policies Network policies can silently block traffic between the Ingress controller and backend pods. The symptom is usually a timeout or 503, but logs may show connection refused. **How to avoid:** When diagnosing connectivity issues, temporarily create a permissive network policy in the namespace to isolate the problem. Once confirmed, adjust the policy to allow necessary traffic. ## Operations Checklist Use this checklist for routine operations and during incidents. Review it monthly to ensure all components are healthy. Assign an owner to each check to maintain accountability. | Step | Command / Check | Expected Result | Owner | Review Frequency | |------|-----------------|-----------------|-------|------------------| | 1. Check cluster health | `kubectl get nodes` | All nodes Ready | Priya Shah, Platform Lead | Monthly | | 2. Check Ingress controller | `kubectl get pods -n ingress-nginx` | All pods Running | DevOps on-call | Weekly | | 3. Check Ingress resource | `kubectl get ingress -A` | Desired Ingress present, no errors in events | Application team | Monthly | | 4. Verify endpoints | `kubectl get endpoints -n ` | Non-empty endpoint list | Application team | Monthly | | 5. Test HTTP route | `curl -H "Host: " http:///` | HTTP 200 | QA engineer | Per release | | 6. Test HTTPS route | `curl --resolve :443: https:///` | TLS handshake OK, HTTP 200 | Security team | Quarterly | | 7. Check logs | `kubectl logs -n ingress-nginx deploy/ingress-nginx-controller --tail=100` | No errors for our host | DevOps on-call | Weekly | | 8. Backup before changes | `kubectl get ingress -n -o yaml > backup.yaml` | Backup file created | Change manager | Every change | | 9. Validate certificate expiry | `kubectl get secret tls-secret -n test -o jsonpath='{.data.tls\.crt}' \| base64 -d \| openssl x509 -noout -enddate` | Expiry > 30 days away | Security team | Monthly | ## Conclusion Troubleshooting Kubernetes Ingress requires a methodical approach: collect environment information, make small safe changes, verify each layer, and know common failure modes. By using the commands and examples in this guide, you can quickly identify and fix issues such as 404s, 503s, TLS errors, and connectivity problems. Next steps: - Practice the diagnostic commands on a test Ingress. - Create a runbook based on the Operations Checklist. - Automate backup of Ingress manifests using a CronJob or GitOps. - Set up monitoring for Ingress controller logs and metrics (e.g., Prometheus alerts for 5xx rates). With these tools, you can keep your Ingress routing reliable and reduce downtime.