Intro
NGINX Ingress is the most common entry point into Kubernetes clusters. When traffic does not reach your app or fails with HTTP 404 or 502, you need a reliable way to isolate the fault: is it DNS and host routing, the Ingress rule, TLS, the Service, or the Pods behind it?
This guide gives you a practical, step-by-step workflow with safe commands and minimal manifests. You will learn how to verify host rules and path matching, validate TLS, interpret 404 vs 502 errors, and confirm backends and endpoints. Each step is designed to be low-risk and observable so you can apply it in production-like environments with confidence.
Workflow Overview
Use this sequence to troubleshoot NGINX Ingress issues safely. Each step narrows the blast radius and avoids disruptive changes.
- Identify the exact failing request
- Note the full URL scheme, host, and path, for example: https://api.example.com/v1/ping
- Capture the status code and response headers if available
- Confirm DNS and host routing
- From a client, resolve the host:
nslookup api.example.com- From a node or diagnostic pod inside the cluster, test HTTP routing with the Host header:
kubectl -n default run curl --rm -it --image=curlimages/curl:8.7.1 --restart=Never --command -- sh -lc \
"curl -sS -o /dev/null -w '%{http_code} %{remote_ip}\n' -H 'Host: api.example.com' http://<ingress-external-ip>/v1/ping"- If the Ingress is behind a LoadBalancer, replace <ingress-external-ip> with the public IP or DNS of the load balancer.
- Verify the Ingress object and class
- List and inspect the Ingress in the target namespace:
kubectl get ingress
kubectl describe ingress api-ing- Check that spec.ingressClassName matches your controller (often "nginx" or "ingress-nginx"). If using the legacy annotation, it should be kubernetes.io/ingress.class: nginx.
- Confirm host rules and paths exactly match your request, including case and trailing slashes.
- Validate path matching and rewrites
- Understand pathType:
- Prefix: matches by path prefix. Example: /api matches /api, /api/v1.
- Exact: matches the exact path only.
- If your backend expects paths without the prefix, use a rewrite:
metadata:
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
rules:
- host: api.example.com
http:
paths:
- path: /api/(.*)
pathType: Prefix
backend:
service:
name: api-svc
port:
number: 80- After changes, re-run curl tests with and without trailing slashes to confirm behavior.
- Differentiate 404 vs 502 at the Ingress layer
- 404 Not Found from NGINX usually indicates no matching host or path rule for the request. Check:
- Host in the request matches spec.rules[].host
- Path and pathType match the incoming URI
- IngressClass is correct
- 502 Bad Gateway usually indicates the rule matched but the upstream failed. Common causes:
- Service has no endpoints (zero Pods Ready)
- Service port does not match containerPort
- NetworkPolicy or firewall blocking pod traffic
- Backend app closed connection or timed out
- Check Service, Endpoints, and Pods
- Confirm selector and ports:
kubectl get svc api-svc -o wide
kubectl describe svc api-svc
kubectl get endpoints api-svc -o wide- If endpoints are empty, verify labels on Pods match the Service selector:
kubectl get pods -l app=api -o wide
kubectl get pod <pod> -o jsonpath='{.metadata.labels}'- Check container ports and readiness:
kubectl describe pod <pod>- If readiness probes fail, the Pod will not appear as a ready endpoint and NGINX will return 502.
- Inspect the NGINX Ingress Controller
- Ensure the controller is running and healthy:
kubectl -n ingress-nginx get pods -o wide
kubectl -n ingress-nginx logs deploy/ingress-nginx-controller | tail -n 100- Look for configuration reloads, upstream errors, or TLS errors.
- Validate TLS configuration
- Check the TLS block and secret reference:
kubectl get ingress api-ing -o yaml | sed -n '/tls:/,/rules:/p'
kubectl get secret api-tls -o yaml- The secret type should be kubernetes.io/tls with tls.crt and tls.key.
- From a client, inspect the certificate chain:
openssl s_client -connect api.example.com:443 -servername api.example.com -showcerts < /dev/null | openssl x509 -noout -issuer -subject -dates- If you see a certificate for the wrong host, either SNI is not matching or the Ingress host does not match the request host.
- Timeouts and large headers or bodies
- Long backend responses may require tuning:
- nginx.ingress.kubernetes.io/proxy-read-timeout: "60"
- nginx.ingress.kubernetes.io/proxy-send-timeout: "60"
- nginx.ingress.kubernetes.io/proxy-body-size: "10m"
- Set annotations on specific paths or Ingresses rather than globally to minimize risk.
- Safe validation before rollout
- Use kubectl diff to preview changes:
kubectl apply -f ingress.yaml --server-side --dry-run=server
kubectl diff -f ingress.yaml- Validate by curling the load balancer with Host headers and by curling ClusterIP Services from a diagnostic pod to isolate Service vs Ingress issues:
# Test Service directly inside the cluster
kubectl -n default run diag --rm -it --image=curlimages/curl:8.7.1 --restart=Never --command -- sh -lc \
"curl -sS -i http://api-svc.default.svc.cluster.local:80/healthz"Local Pilot Plan
Create a narrow, measurable pilot in an isolated namespace so you can confirm routing, TLS, and backends end to end.
- Namespace and sample app
apiVersion: v1
kind: Namespace
metadata:
name: ingress-pilot
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: echo
namespace: ingress-pilot
spec:
replicas: 2
selector:
matchLabels:
app: echo
template:
metadata:
labels:
app: echo
spec:
containers:
- name: echo
image: hashicorp/http-echo:0.2.3
args: ["-text=hello", "-listen=:8080"]
ports:
- containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
name: echo-svc
namespace: ingress-pilot
spec:
selector:
app: echo
ports:
- port: 80
targetPort: 8080- Basic HTTP Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: echo-ing
namespace: ingress-pilot
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
ingressClassName: nginx
rules:
- host: echo.local
http:
paths:
- path: /(.*)
pathType: Prefix
backend:
service:
name: echo-svc
port:
number: 80- Test safely
- Apply and verify:
kubectl apply -f pilot.yaml
kubectl -n ingress-pilot get ingress echo-ing -o wide
kubectl -n ingress-pilot describe ingress echo-ing
kubectl -n ingress-pilot get endpoints echo-svc- From a diagnostic pod, hit the Ingress using Host header:
kubectl -n ingress-pilot run tester --rm -it --image=curlimages/curl:8.7.1 --restart=Never --command -- sh -lc \
"curl -i -H 'Host: echo.local' http://<ingress-external-ip>/"- Expected: HTTP/1.1 200 and body "hello".
- TLS pilot (optional)
- Create a TLS secret in the same namespace:
kubectl -n ingress-pilot create secret tls echo-tls \
--cert=server.crt --key=server.key- Add TLS to the Ingress:
spec:
tls:
- hosts:
- echo.local
secretName: echo-tls- Validate certificate and SNI:
openssl s_client -connect <ingress-external-ip>:443 -servername echo.local < /dev/null | openssl x509 -noout -subject -dates- Success criteria
- HTTP path / returns 200 with body "hello".
- Changing Host header to a non-matching host yields 404.
- Removing all endpoints from the Service yields 502, then returns to 200 when endpoints are restored.
- With TLS configured, the certificate subject matches echo.local and the request returns 200.
Conclusion
When NGINX Ingress fails, treat it as a routing puzzle you can solve step by step: confirm DNS and Host, verify the Ingress class and rules, check path matching and rewrites, differentiate 404 from 502, and validate Services, Endpoints, and Pods. Use targeted annotations only when necessary and always test in a narrow pilot before making wider changes.
Next steps:
- Keep a diagnostic pod template ready to curl Services and the Ingress with Host headers.
- Standardize Ingress patterns for hosts, paths, and TLS so teams can reuse known-good building blocks.
- Capture a short checklist for 404 vs 502 triage and store it with your app runbooks.