Kubernetes Ingress Troubleshooting: Practical Examples for Common Failures

Intro

Kubernetes Ingress is a powerful resource for routing external HTTP and HTTPS traffic to services inside a cluster. But when an Ingress misbehaves, the symptoms can be confusing: 404s, timeouts, TLS errors, or traffic going to the wrong backend. Troubleshooting Ingress requires a systematic approach: understand the components involved, check configuration, inspect logs, and verify connectivity step by step.

This guide provides practical examples for diagnosing and fixing common Ingress failures. We will cover environment inventory, safe configuration changes, verification commands, failure modes, and recovery steps. Every example includes concrete kubectl commands and expected outputs. By the end, you will have a repeatable troubleshooting workflow.

Version and Environment Inventory

Before troubleshooting, collect information about your cluster, Ingress controller, and related resources. This establishes a baseline and helps identify mismatches. A common pitfall is assuming the Ingress controller is healthy when it is not, or using an outdated kubectl client that misreports resource versions. Start with a clean inventory.

Prerequisites:

• Access to a Kubernetes cluster with kubectl configured.
• Permission to get and describe resources in the relevant namespaces.
• Basic understanding of Kubernetes Services, Deployments, and Ingress.

Commands to run:

kubectl version --short

Expected output (example):

Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.3

If client and server versions differ significantly (e.g., client is 1.22 and server is 1.28), some kubectl output fields may be missing or deprecated. Upgrade your client to match the server minor version whenever possible.

kubectl get nodes -o wide

This shows node status, Kubernetes version, and internal/external IPs. Look for any node in NotReady state, which can affect Ingress controller scheduling.

kubectl get pods -n ingress-nginx

Replace ingress-nginx with your Ingress controller namespace. Common namespaces include ingress-nginx, kube-system, traefik, istio-system, or contour. Expected output shows running pods, e.g.:

NAME                                        READY   STATUS    RESTARTS   AGE
ingress-nginx-controller-6d5c7b8f9b-abcde   1/1     Running   0          5d

If the controller pod is not running, describe it to see why:

kubectl describe pod <pod-name> -n ingress-nginx

Check the IngressClass:

kubectl get ingressclass

Example output:

NAME      CONTROLLER                 PARAMETERS   AGE
nginx     k8s.io/ingress-nginx      <none>       10d

Identify the Ingress resource and its configuration:

kubectl get ingress -A
kubectl describe ingress <ingress-name> -n <namespace>

The describe output shows rules, backend services, TLS settings, and annotations. Verify that the IngressClass matches the controller's class (or that the Ingress uses the default class if ingressClassName is omitted). A mismatch here is a frequent cause of Ingress being ignored.

Topology checklist:

• Ingress controller is running and healthy.
• Ingress resource exists in the expected namespace.
• Backend Service exists and selects correct pods.
• Service has endpoints (pods ready).
• TLS secret exists if HTTPS is used.
• Network policies allow traffic from Ingress controller to backend pods.

Safe Configuration Path

When making changes, proceed incrementally. First, test with a simple Ingress before modifying complex routing. This mirrors the practice of starting with a narrow, measurable change and inspecting locally before broader rollout. Avoid editing production Ingress resources directly via kubectl edit; instead, export the current manifest, modify it in a file, and apply with a diff check.

Example minimal Ingress:

Create a test deployment and service:

kubectl create deployment web --image=nginx:1.25 --port=80 -n test
kubectl expose deployment web --port=80 --target-port=80 -n test

Create an Ingress for it (save as ingress.yaml):

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web-ingress
  namespace: test
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx
  rules:
  - host: test.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: web
            port:
              number: 80

Apply and test:

kubectl apply -f ingress.yaml
curl -H "Host: test.example.com" http://<ingress-controller-ip>/

If the Ingress controller is exposed via a LoadBalancer, get its IP:

kubectl get svc -n ingress-nginx ingress-nginx-controller

Expected output shows an external IP. Use that IP in curl. If you don't have an external IP, you can test using port-forward:

kubectl port-forward -n ingress-nginx svc/ingress-nginx-controller 8080:80

Then curl http://localhost:8080/ with the Host header.

If you need to change an existing Ingress, first back up the current manifest:

kubectl get ingress <name> -n <ns> -o yaml > ingress-backup.yaml

Then edit, apply, and verify. Use kubectl diff to preview changes:

kubectl diff -f updated-ingress.yaml

This shows what would change without applying. Always review the diff before applying to catch unintended modifications.

Verification and Diagnostics

After configuration, verify that traffic flows correctly. Use multiple commands to check each layer. Treat the Ingress controller as a black box initially: confirm that requests reach it and that it can reach backends, then narrow down.

1. Check Ingress resource status

kubectl describe ingress web-ingress -n test

Look for events: successful sync or errors. Example of a healthy Ingress:

Events:
  Type    Reason  Age   From                      Message
  ----    ------  ----  ----                      -------
  Normal  Sync    2m    nginx-ingress-controller  Scheduled for sync

If there are errors, they appear here. Common errors include "endpoint not found" or "service not found".

2. Check Ingress controller logs

kubectl logs -n ingress-nginx deployment/ingress-nginx-controller --tail=50

Adjust namespace and deployment name. Look for lines containing your Ingress name, host, or error. Example error:

E0301 12:00:00.123456       1 controller.go:123] Unexpected error ingressing: service "test/web" is type "ClusterIP", expected "NodePort" or "LoadBalancer"

With most modern Ingress controllers, ClusterIP services are valid; this error may indicate a misconfigured controller or an outdated version. Check controller documentation.

To filter logs for a specific host:

kubectl logs -n ingress-nginx deployment/ingress-nginx-controller --tail=200 | grep "test.example.com"

3. Verify backend service and endpoints

kubectl get svc web -n test
kubectl get endpoints web -n test

Expected:

NAME   TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
web    ClusterIP   10.96.123.45   <none>        80/TCP    10m

NAME   ENDPOINTS           AGE
web    10.244.0.10:80      10m

If endpoints are empty, the service selector doesn't match pods or pods are not ready. Check pod labels:

kubectl get pods -n test --show-labels

Compare with service selector:

kubectl describe svc web -n test

4. Test from inside the cluster

Run a temporary pod with curl:

kubectl run curl-test --image=curlimages/curl -i --rm --restart=Never -- curl -v -H "Host: test.example.com" http://ingress-nginx-controller.ingress-nginx.svc.cluster.local/

This tests internal routing to the controller's ClusterIP service. Check response status and headers. If this fails, the problem is inside the cluster (controller, service, or backend). If this succeeds but external access fails, focus on external networking (LoadBalancer, DNS, firewall).

5. Test TLS (if applicable)

kubectl get secret tls-secret -n test -o yaml

Verify secret contains tls.crt and tls.key. Then test with curl using --resolve to map host to ingress IP:

curl -v --resolve test.example.com:443:<ingress-ip> https://test.example.com/

Check for certificate errors. Use openssl to examine the certificate:

kubectl get secret tls-secret -n test -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -dates -subject

Expected results for a working setup

• Ingress describe shows a Sync event.
• Ingress controller logs show no errors for the host.
• Endpoints contain pod IPs.
• Curl returns HTTP 200 and the expected content.
• TLS handshake succeeds (no certificate warnings).

If any of these fail, proceed to the failure modes section below.

Common Failure Modes and Recovery

This section covers the most frequent Ingress failures, with symptoms, diagnosis, and recovery steps. Each failure mode includes concrete commands and expected outputs.

Failure Mode 1: 404 Not Found

Symptoms: Curl returns 404 even though the path seems correct.

Possible causes:

• Ingress path does not match request path (e.g., path is /api but request is /api/ due to pathType mismatch).
• rewrite-target annotation misconfigured, causing backend to receive unexpected path.
• Backend service itself returns 404 for the path.

Diagnosis:

Check Ingress rules:

kubectl get ingress web-ingress -n test -o yaml

Look at path and pathType. Ingress-nginx default pathType is Prefix. With Prefix, /api matches /api, /api/, /api/v1, etc. With Exact, only /api matches. If you need exact matching but used Prefix, change it.

Check backend service directly from a pod:

kubectl exec -it <some-pod-in-same-namespace> -n test -- curl http://web.test.svc.cluster.local/<path>

If direct curl works but Ingress returns 404, Ingress routing is wrong.

Recovery:

• Adjust Ingress path or pathType to match the requested URL.
• If using rewrite-target, test without it first. Misconfigured rewrite can strip or add path segments unexpectedly.
• Apply changes and re-test:

kubectl apply -f updated-ingress.yaml
curl -H "Host: test.example.com" http://<ingress-ip>/<path>

Failure Mode 2: 503 Service Unavailable

Symptoms: Ingress returns 503.

Possible causes:

• Backend service has no endpoints (pods not ready).
• Service name or port mismatch in Ingress.
• Network policy blocking traffic from Ingress controller to pods.
• Readiness probe failing, so pods are not marked ready.

Diagnosis:

Check endpoints:

kubectl get endpoints <service> -n <namespace>

If empty, check pods and their labels:

kubectl get pods -n test --show-labels
kubectl describe svc web -n test

Ensure service selector matches pod labels exactly. Note that a service selector with app: web will not match pods with app: web-backend.

Check Ingress backend definition:

kubectl describe ingress web-ingress -n test

Verify service name and port. In the example, the backend should be serviceName: web and servicePort: 80.

Check Ingress controller logs for error:

kubectl logs -n ingress-nginx deployment/ingress-nginx-controller | grep -i "test/web"

Example:

W0301 12:00:00.123456       1 controller.go:456] Service "test/web" does not have any active Endpoint.

Recovery:

• Fix service selector to match pods.
• Ensure pods are ready: check kubectl get pods -n test and look at READY column. If 0/1, describe pod and check readiness probe logs.
• Correct service name/port in Ingress.
• If network policy is suspected, temporarily allow all traffic in the namespace for testing, then tighten.

Failure Mode 3: TLS Certificate Errors

Symptoms: Browser shows certificate warning or curl reports SSL certificate problem.

Possible causes:

• TLS secret missing or malformed.
• Certificate hostname mismatch.
• Certificate expired.
• Secret not in the same namespace as Ingress.

Diagnosis:

Check secret:

kubectl get secret tls-secret -n test

If not found:

Error from server (NotFound): secrets "tls-secret" not found

Check certificate details:

kubectl get secret tls-secret -n test -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -dates -subject

Expected output shows validity dates and subject CN/SAN. Verify that the SAN includes the hostname used in Ingress.

Recovery:

• Create a valid secret:

kubectl create secret tls tls-secret --cert=path/to/tls.crt --key=path/to/tls.key -n test

• Ensure the certificate includes the hostname in SAN (not just CN).
• Renew if expired.
• Ensure the secret is in the same namespace as the Ingress resource.

Failure Mode 4: Connection Refused or Timeout

Symptoms: Curl cannot connect; connection refused or times out.

Possible causes:

• Ingress controller service not reachable externally (e.g., LoadBalancer IP not assigned, firewall blocks).
• Ingress controller pod not running.
• Network policy blocking traffic to controller.
• DNS resolution failure for the host.

Diagnosis:

Check Ingress controller service:

kubectl get svc -n ingress-nginx

Look for EXTERNAL-IP. If <pending>, the LoadBalancer may still be provisioning, or your cluster may not support LoadBalancer (e.g., bare metal). Check service events:

kubectl describe svc ingress-nginx-controller -n ingress-nginx

Check controller pods:

kubectl get pods -n ingress-nginx

If pods are not running, describe them:

kubectl describe pod <pod-name> -n ingress-nginx

Look for CrashLoopBackOff, ImagePullBackOff, or resource issues.

Recovery:

• Wait for LoadBalancer IP or use NodePort / port-forward for testing.
• Fix pod issues (e.g., resource limits, config errors).
• Check cloud provider firewall or security groups. Allow inbound traffic on ports 80/443 from your IP.
• Verify DNS record points to the correct IP or use --resolve to bypass DNS.

Failure Mode 5: Routing to Wrong Backend

Symptoms: Request is served by unexpected service.

Possible causes:

• Multiple Ingress resources with same host but different paths, and path matching order leads to wrong backend.
• Default backend configuration.
• Annotation misconfiguration (e.g., canary weights).

Diagnosis:

List all Ingress resources with the host:

kubectl get ingress -A | grep <host>

Inspect each Ingress's rules. Ingress controllers merge rules from all Ingress resources in the cluster, so conflicts can occur. Ingress-nginx uses a priority system: longest path match wins, and if equal, creation timestamp decides. This can lead to unexpected routing.

Check for default backend:

kubectl get ingress -n test web-ingress -o yaml

If spec.defaultBackend is set, any request not matching a rule goes there.

Recovery:

• Consolidate rules into a single Ingress for the host to avoid ambiguity.
• Or adjust path specificity so the longest intended path wins.
• Use different hosts if possible.
• Explicitly set ingressClassName to ensure the correct controller processes the Ingress.

Rollback and Recovery Checks

Always keep backups of Ingress manifests. If a change causes issues, rollback:

kubectl apply -f ingress-backup.yaml

Then verify with curl and describe. Also, consider using kubectl rollout undo for Deployments if pod changes caused the issue:

kubectl rollout undo deployment/web -n test

Common Pitfalls and How to Avoid Them

Beyond the specific failure modes, several recurring mistakes can make Ingress troubleshooting harder. Here are the most common pitfalls and their solutions.

Pitfall 1: Ignoring IngressClass

Many users omit ingressClassName or set it incorrectly, assuming the default controller will handle the Ingress. If multiple controllers exist or the default class is not set, the Ingress may be ignored entirely.

How to avoid: Always set ingressClassName explicitly in your Ingress manifest. Verify the class exists with kubectl get ingressclass.

Pitfall 2: Misunderstanding pathType

The difference between Prefix and Exact path types causes many 404s. For example, with pathType: Exact and path: /foo, a request to /foo/ will not match, returning 404. With Prefix, /foo matches /foo, /foo/, and /foo/bar.

How to avoid: Choose Prefix for most applications unless you need exact path matching. Test with and without trailing slashes.

Pitfall 3: Rewrite-target annotation misuse

The rewrite-target annotation is powerful but frequently misconfigured. Setting rewrite-target: / for an Ingress with path /api will strip /api and send the remaining path to the backend. If not intended, the backend receives unexpected paths and may return 404 or 500.

How to avoid: Test the rewrite behavior by examining backend access logs or using a simple echo server. Only use rewrite when you understand the target path semantics.

Pitfall 4: Not checking endpoints

When a service has no endpoints, the Ingress controller will return 503. Users often spend time checking Ingress rules when the real issue is that pods are not ready or labels don't match.

How to avoid: Make endpoint verification a standard step in your troubleshooting checklist. Use kubectl get endpoints immediately after checking Ingress.

Pitfall 5: Overlooking network policies

Network policies can silently block traffic between the Ingress controller and backend pods. The symptom is usually a timeout or 503, but logs may show connection refused.

How to avoid: When diagnosing connectivity issues, temporarily create a permissive network policy in the namespace to isolate the problem. Once confirmed, adjust the policy to allow necessary traffic.

Operations Checklist

Use this checklist for routine operations and during incidents. Review it monthly to ensure all components are healthy. Assign an owner to each check to maintain accountability.

Conclusion

Troubleshooting Kubernetes Ingress requires a methodical approach: collect environment information, make small safe changes, verify each layer, and know common failure modes. By using the commands and examples in this guide, you can quickly identify and fix issues such as 404s, 503s, TLS errors, and connectivity problems.

Next steps:

• Practice the diagnostic commands on a test Ingress.
• Create a runbook based on the Operations Checklist.
• Automate backup of Ingress manifests using a CronJob or GitOps.
• Set up monitoring for Ingress controller logs and metrics (e.g., Prometheus alerts for 5xx rates).

With these tools, you can keep your Ingress routing reliable and reduce downtime.