## Intro

NGINX Ingress is the most common entry point into Kubernetes clusters. When traffic does not reach your app or fails with HTTP 404 or 502, you need a reliable way to isolate the fault: is it DNS and host routing, the Ingress rule, TLS, the Service, or the Pods behind it?

This guide gives you a practical, step-by-step workflow with safe commands and minimal manifests. You will learn how to verify host rules and path matching, validate TLS, interpret 404 vs 502 errors, and confirm backends and endpoints. Each step is designed to be low-risk and observable so you can apply it in production-like environments with confidence.

## Workflow Overview

Use this sequence to troubleshoot NGINX Ingress issues safely. Each step narrows the blast radius and avoids disruptive changes.

- Identify the exact failing request

- Note the full URL scheme, host, and path, for example: https://api.example.com/v1/ping

- Capture the status code and response headers if available

- Confirm DNS and host routing

- From a client, resolve the host:

nslookup api.example.com 

- From a node or diagnostic pod inside the cluster, test HTTP routing with the Host header:

 kubectl -n default run curl --rm -it --image=curlimages/curl:8.7.1 --restart=Never --command -- sh -lc \
 "curl -sS -o /dev/null -w '%{http_code} %{remote_ip}\n' -H 'Host: api.example.com' http://<ingress-external-ip>/v1/ping" 

- If the Ingress is behind a LoadBalancer, replace with the public IP or DNS of the load balancer.

- Verify the Ingress object and class

- List and inspect the Ingress in the target namespace:

 kubectl get ingress
 kubectl describe ingress api-ing 

- Check that spec.ingressClassName matches your controller (often nginx or ingress-nginx ). If using the legacy annotation, it should be kubernetes.io/ingress.class: nginx .

- Confirm host rules and paths exactly match your request, including case and trailing slashes.

- Validate path matching and rewrites

- Understand pathType :

- Prefix : matches by path prefix. Example: /api matches /api , /api/v1 .

- Exact : matches the exact path only.

- If your backend expects paths without the prefix, use a rewrite. For example, to strip /api from the request path:

 metadata:
 annotations:
 nginx.ingress.kubernetes.io/rewrite-target: /$1
 spec:
 rules:
 - host: api.example.com
 http:
 paths:
 - path: /api/(.*)
 pathType: Prefix
 backend:
 service:
 name: api-svc
 port:
 number: 80 

- After changes, re-run curl tests with and without trailing slashes to confirm behavior.

- Differentiate 404 vs 502 at the Ingress layer

- 404 Not Found from NGINX usually indicates no matching host or path rule for the request. Check:

- Host in the request matches spec.rules[].host

- Path and pathType match the incoming URI

- IngressClass is correct

- 502 Bad Gateway usually indicates the rule matched but the upstream failed. Common causes:

- Service has no endpoints (zero Pods Ready)

- Service port does not match containerPort

- NetworkPolicy or firewall blocking pod traffic

- Backend app closed connection or timed out

- Check Service, Endpoints, and Pods

- Confirm selector and ports:

 kubectl get svc api-svc -o wide
 kubectl describe svc api-svc
 kubectl get endpoints api-svc -o wide 

- If endpoints are empty, verify labels on Pods match the Service selector:

 kubectl get pods -l app=api -o wide
 kubectl get pod <pod> -o jsonpath='{.metadata.labels}' 

- Check container ports and readiness:

 kubectl describe pod <pod> 

- If readiness probes fail, the Pod will not appear as a ready endpoint and NGINX will return 502.

- Inspect the NGINX Ingress Controller

- Ensure the controller is running and healthy:

 kubectl -n ingress-nginx get pods -o wide
 kubectl -n ingress-nginx logs deploy/ingress-nginx-controller | tail -n 100 

- Look for configuration reloads, upstream errors, or TLS errors.

- Validate TLS configuration

- Check the TLS block and secret reference:

 kubectl get ingress api-ing -o yaml | sed -n '/tls:/,/rules:/p'
 kubectl get secret api-tls -o yaml 

- The secret type should be kubernetes.io/tls with tls.crt and tls.key .

- From a client, inspect the certificate chain:

 openssl s_client -connect api.example.com:443 -servername api.example.com -showcerts < /dev/null | openssl x509 -noout -issuer -subject -dates 

- If you see a certificate for the wrong host, either SNI is not matching or the Ingress host does not match the request host.

- Timeouts and large headers or bodies

- Long backend responses may require tuning. Apply annotations on specific paths or Ingresses rather than globally to minimize risk:

 annotations:
 nginx.ingress.kubernetes.io/proxy-read-timeout: "60"
 nginx.ingress.kubernetes.io/proxy-send-timeout: "60"
 nginx.ingress.kubernetes.io/proxy-body-size: "10m" 

- Safe validation before rollout

- Use kubectl diff to preview changes:

 kubectl apply -f ingress.yaml --server-side --dry-run=server
 kubectl diff -f ingress.yaml 

- Validate by curling the load balancer with Host headers and by curling ClusterIP Services from a diagnostic pod to isolate Service vs Ingress issues:

 # Test Service directly inside the cluster
 kubectl -n default run diag --rm -it --image=curlimages/curl:8.7.1 --restart=Never --command -- sh -lc \
 "curl -sS -i http://api-svc.default.svc.cluster.local:80/healthz" 

## Local Pilot Plan

 Create a narrow, measurable pilot in an isolated namespace so you can confirm routing, TLS, and backends end to end.

- Namespace and sample app

apiVersion: v1
 kind: Namespace
 metadata:
 name: ingress-pilot
 ---
 apiVersion: apps/v1
 kind: Deployment
 metadata:
 name: echo
 namespace: ingress-pilot
 spec:
 replicas: 2
 selector:
 matchLabels:
 app: echo
 template:
 metadata:
 labels:
 app: echo
 spec:
 containers:
 - name: echo
 image: hashicorp/http-echo:0.2.3
 args: ["-text=hello", "-listen=:8080" ]
 ports:
 - containerPort: 8080
 ---
 apiVersion: v1
 kind: Service
 metadata:
 name: echo-svc
 namespace: ingress-pilot
 spec:
 selector:
 app: echo
 ports:
 - port: 80
 targetPort: 8080 

- Basic HTTP Ingress

 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
 name: echo-ing
 namespace: ingress-pilot
 annotations:
 nginx.ingress.kubernetes.io/rewrite-target: /$1
 spec:
 ingressClassName: nginx
 rules:
 - host: echo.local
 http:
 paths:
 - path: /(.*)
 pathType: Prefix
 backend:
 service:
 name: echo-svc
 port:
 number: 80 

- Test safely

- Apply and verify:

 kubectl apply -f pilot.yaml
 kubectl -n ingress-pilot get ingress echo-ing -o wide
 kubectl -n ingress-pilot describe ingress echo-ing
 kubectl -n ingress-pilot get endpoints echo-svc 

- From a diagnostic pod, hit the Ingress using Host header:

 kubectl -n ingress-pilot run tester --rm -it --image=curlimages/curl:8.7.1 --restart=Never --command -- sh -lc \
 "curl -i -H 'Host: echo.local' http://<ingress-external-ip>/" 

- Expected: HTTP/1.1 200 and body hello .

- TLS pilot (optional)

- Create a TLS secret in the same namespace:

 kubectl -n ingress-pilot create secret tls echo-tls \
 --cert=server.crt --key=server.key 

- Add TLS to the Ingress:

 spec:
 tls:
 - hosts:
 - echo.local
 secretName: echo-tls 

- Validate certificate and SNI:

 openssl s_client -connect <ingress-external-ip>:443 -servername echo.local < /dev/null | openssl x509 -noout -subject -dates 

- Success criteria

- HTTP path / returns 200 with body hello .

- Changing Host header to a non-matching host yields 404.

- Removing all endpoints from the Service yields 502, then returns to 200 when endpoints are restored.

- With TLS configured, the certificate subject matches echo.local and the request returns 200.

## Conclusion

 When NGINX Ingress fails, treat it as a routing puzzle you can solve step by step: confirm DNS and Host, verify the Ingress class and rules, check path matching and rewrites, differentiate 404 from 502, and validate Services, Endpoints, and Pods. Use targeted annotations only when necessary and always test in a narrow pilot before making wider changes.

Next steps:

- Keep a diagnostic pod template ready to curl Services and the Ingress with Host headers.

- Standardize Ingress patterns for hosts, paths, and TLS so teams can reuse known-good building blocks.

- Capture a short checklist for 404 vs 502 triage and store it with your app runbooks.