## Introduction Ingress routes external HTTP(S) traffic to Services inside your cluster, but the Ingress resource alone does not expose anything. A working request travels through multiple layers that each must be healthy. Most Kubernetes Ingress errors are not caused by a single YAML field; they are request-path failures across DNS, load balancers, controllers, Services, endpoints, and application Pods. Troubleshoot by testing every hop in order, not by randomly editing manifests. Client | DNS | External IP / Load Balancer | Ingress Controller Service | Ingress Controller Pod | Ingress Rule | Application Service | EndpointSlice | Application Pod What you will learn: - How to verify each layer with concrete commands. - How to diagnose Kubernetes Ingress 404/502/503/504 and TLS errors. - How to separate the Ingress API from the controller implementation. Primary concept: an Ingress is configuration consumed by an Ingress controller. Creating an Ingress resource does not install or configure a controller. ## Build a minimal working example Save the following as example.yaml. It creates a namespace, a simple HTTP echo Deployment, a ClusterIP Service, and an Ingress with host-based routing at path "/" using pathType: Prefix. Adjust and . apiVersion: v1 kind: Namespace metadata: name: --- apiVersion: apps/v1 kind: Deployment metadata: name: echo namespace: spec: replicas: 2 selector: matchLabels: app: echo template: metadata: labels: app: echo spec: containers: - name: echo image: hashicorp/http-echo:0.2.3 args: ["-text=ok"] ports: - containerPort: 5678 --- apiVersion: v1 kind: Service metadata: name: echo-svc namespace: spec: selector: app: echo ports: - name: http port: 80 targetPort: 5678 type: ClusterIP --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: echo namespace: spec: ingressClassName: nginx rules: - host: http: paths: - path: / pathType: Prefix backend: service: name: echo-svc port: number: 80 Apply and inspect: kubectl apply -f example.yaml kubectl get pods,svc,ingress -n kubectl describe ingress echo -n Healthy signs: - Pods Ready=1/1 and STATUS=Running. - Service has a stable ClusterIP and a port 80 -> 5678 mapping. - Ingress shows an ADDRESS (external IP or hostname) after your controller provisions it. ## Start with the request path Use this quick table to test each layer deliberately.
LayerTypical symptomDiagnostic commandSuccess looks likeLikely fix
DNSHost does not resolve<code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">dig +short <host></code>Returns IP or LB hostnameCreate/adjust DNS A/AAAA/CNAME
External LBConnection refused/timeout<code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">curl -v http://<host>/</code>TCP connect succeedsProvision LB, open firewall/SG, wait for provisioning
Controller ServiceEmpty Ingress ADDRESS<code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">kubectl get svc -A</code>Controller svc has EXTERNAL-IP or NodePortConfigure Service type, fix cloud/LB setup
Controller Pods503/timeout<code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">kubectl get pods -A</code>Controller Pods Running/ReadyInstall/repair controller
IngressClassIngress ignored<code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">kubectl get ingressclass</code>Matching class exists and is default or named in specSet <code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">spec.ingressClassName</code> correctly
Ingress resource404 default backend<code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">kubectl describe ingress</code>Accepted, rules listed, no errorsFix host/path, annotations, class
Backend Service502/503<code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">kubectl get svc <svc> -o yaml</code>Port/selector correctCorrect name/port/selector
EndpointSlice503/504<code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">kubectl get endpointslice -l kubernetes.io/service-name=<svc> -n <ns></code>Addresses with Ready=trueFix selector, Pod readiness
Application Pods502/504<code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">kubectl describe pod <pod></code>Ready, listening on 0.0.0.0Fix probes, bind address, container args
TLSBrowser warnings<code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">openssl s_client -connect <host>:443 -servername <host></code>Certificate matches host, not expiredFix Secret, SANs, chain
App responseApp 404<code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">curl -v -H "Host: <host>" http://<ingress-ip>/</code>App headers/body as expectedFix app routes or rewrites
## Error 1: No Ingress controller is installed An Ingress object is only configuration. Without a controller, nothing listens. kubectl get pods -A | grep -E "ingress|traefik|haproxy|alb|gateway" kubectl get deployments -A | grep -E "ingress|traefik|haproxy|alb|gateway" kubectl get ingressclass Look for ingress-nginx, Traefik, HAProxy, or your cloud vendor controller. Installation differs across bare metal, minikube/kind, managed Kubernetes, and VMs. Use your platform’s documented method. ## Error 2: IngressClass is missing or incorrect The controller handling your Ingress is chosen by spec.ingressClassName or the cluster’s default IngressClass. kubectl get ingressclass kubectl describe ingressclass kubectl get ingress echo -n -o yaml Example spec: spec: ingressClassName: nginx Use the class name your controller advertises; names are implementation-specific. ## Error 3: Ingress has no external address kubectl get ingress -A shows an empty ADDRESS when: - No controller handles the resource. - The controller Service has no EXTERNAL-IP ( ). - Load balancer is still provisioning or not supported on your infra. - Bare-metal LoadBalancer integration (e.g., MetalLB) is missing/misconfigured. - The controller is exposed via NodePort instead. Check: kubectl get svc -A kubectl describe svc -n kubectl get events -A --sort-by=.metadata.creationTimestamp If EXTERNAL-IP is , solve the Service or infrastructure issue first. ## Error 4: DNS does not point to the Ingress endpoint Validate DNS and host-based routing: dig +short nslookup curl -v http:/// To bypass DNS while preserving the Host header: curl -v --resolve :80: http:/// curl -vk --resolve :443: https:/// --resolve proves the Ingress and backend work even if public DNS is wrong. ## Error 5: Default backend returns 404 A controller-generated 404 typically means host or path did not match any rule, or the wrong controller handled the request. curl -v -H "Host: " http:/// kubectl describe ingress -n kubectl get ingress -n -o yaml Differentiate: - Ingress 404: headers often include controller-specific identifiers (e.g., nginx). Fix host/path, class, or annotations. - Application 404: request reached your app; adjust app routing. ## Error 6: Path matching and rewrites - pathType: Prefix matches /api and /api/v1 . - pathType: Exact matches the path literally. - ImplementationSpecific is controller-defined and not portable. Standard example without rewrite: paths: - path: / pathType: Prefix backend: service: name: echo-svc port: number: 80 Ingress-nginx-specific rewrite example (controller-specific, not portable): metadata: annotations: nginx.ingress.kubernetes.io/rewrite-target: / Do not copy nginx annotations to Traefik, HAProxy, or cloud controllers. ## Error 7: Wrong Service name or port The Ingress backend must reference the Service port, not the Pod’s containerPort. backend: service: name: api-service port: number: 8080 Check mappings and names: kubectl get svc -n -o yaml kubectl describe svc -n kubectl get deployment -n -o yaml Service.port maps to Service.targetPort, which maps to the containerPort (or a named port) inside the Pod. ## Error 8: Service has no endpoints A Service can exist with zero ready endpoints. Confirm: kubectl get endpoints -n kubectl get endpointslice -n -l kubernetes.io/service-name= kubectl describe svc -n kubectl get pods -n --show-labels Common causes: - Selector does not match Pod labels. - Pods are not Ready (failing probes). - Pods are terminating or in another namespace. Fix selectors to match labels; Ingress cannot reference Services in other namespaces via the standard backend. ## Error 9: Pods running but not Ready Running does not mean routable. Inspect probes and logs: kubectl get pods -n kubectl describe pod -n kubectl logs -n Example readiness probe: readinessProbe: httpGet: path: /healthz port: 5678 initialDelaySeconds: 2 periodSeconds: 5 Ensure the app listens on 0.0.0.0, not 127.0.0.1. ## Error 10–12: 502, 503, 504 - 502 Bad Gateway: often wrong targetPort, backend refusing connections, HTTP vs HTTPS mismatch. - 503 Service Unavailable: no ready endpoints, backend Service missing, or controller reload errors. - 504 Gateway Timeout: slow app, dependency latency, NetworkPolicy blocking, or controller timeout settings. Test the Service from inside the cluster to isolate Ingress vs backend: kubectl run curl-test --rm -it --restart=Never \ --image=curlimages/curl -n -- \ curl -v http://:/ Success here but failure through Ingress points to controller, routing rules, or protocol/TLS configuration. ## TLS errors: Secrets and certificates TLS requires a Secret of type kubernetes.io/tls in the same namespace as the Ingress. Hosts must match the certificate SANs. spec: tls: - hosts: - app.example.com secretName: app-example-tls Validate: kubectl get secret app-example-tls -n kubectl describe ingress -n openssl s_client -connect :443 -servername | openssl x509 -noout -subject -issuer -dates -ext subjectAltName Check for expiration, chain completeness, and correct hostname. ## Works with port-forward but not through Ingress kubectl port-forward deployment/ 8080: proves the Pod listens and responds. kubectl port-forward service/ 8080: proves the Service routes to endpoints. If both work but the public route fails, focus on controller, DNS, TLS, or external networking. ## Controller logs and generated configuration Identify controller Pods generically: kubectl get pods -A -l app.kubernetes.io/component=controller Then inspect logs in the correct namespace. Example (ingress-nginx only): kubectl -n ingress-nginx logs deploy/ingress-nginx-controller Look for admission errors, config reloads, and upstream health messages. ## A disciplined troubleshooting workflow - Confirm DNS: dig +short . - Confirm external reachability: curl -v http(s):/// . - Controller installed and Ready: kubectl get deployments,pods -A . - Correct IngressClass: kubectl get ingressclass and Ingress spec.ingressClassName . - Inspect Ingress and events: kubectl describe ingress . - Test host/path: curl -v -H "Host: " http:/// . - Verify Service name and port: kubectl describe svc . - EndpointSlices Ready: kubectl get endpointslice . - Test inside cluster: kubectl run ... curl http://:/ . - Check controller logs. - Check application logs and probes. - Validate TLS Secret and certificate hostname. - Review NetworkPolicies/firewalls. - Re-test externally. ## Practical end-to-end incident Symptom (example output): - curl -v https://app.example.com/ returns 503. Evidence: kubectl get ingress echo -n demo # ADDRESS set; rules OK kubectl get svc echo-svc -n demo # Port 80 -> 5678 kubectl get endpointslice -n demo -l kubernetes.io/service-name=echo-svc # No endpoints listed kubectl get deploy echo -n demo -o yaml | grep labels -A2 # Deployment pods labeled app=echo-app (mismatch!) Root cause: Service selector app=echo does not match Pod label app=echo-app , so there are zero ready endpoints. Fix (align labels): spec: selector: matchLabels: app: echo template: metadata: labels: app: echo Validate: kubectl apply -f deployment.yaml kubectl get endpointslice -n demo -l kubernetes.io/service-name=echo-svc # Endpoints now present curl -v https://app.example.com/ # 200 OK Prevention: alert on Services with no ready endpoints; lint selectors in CI; test routes after deploy. ## Quick decision table
SymptomMost likely layerFirst commandCommon causeNext action
DNS failureDNS<code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">dig +short <host></code>Missing/old recordFix DNS A/AAAA/CNAME
Connection refusedLB/Service<code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">curl -v http://<host>/</code>LB not ready, firewallVerify LB, open ports
TimeoutLB/Controller/App<code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">curl -v</code> + controller logsNetworkPolicy, slow appTest Service in-cluster
Controller 404Ingress rule<code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">kubectl describe ingress</code>Host/path mismatchCorrect rule/pathType
Application 404App<code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">curl -v</code> headersApp route missingFix app routes
502BackendIn-cluster curl to ServiceWrong targetPort/protocolAlign ports/protocols
503Endpoints<code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">kubectl get endpointslice</code>No ready endpointsFix selector/probes
504App/depsApp logs/metricsSlow backendOptimize or adjust timeouts
TLS warningTLS<code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">openssl s_client ...</code>Name/expiry/chainFix Secret/cert
Wrong certificateTLS/SNI<code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">openssl s_client -servername</code>Host mismatchProvide per-host cert
Redirect loopProxy/app<code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">curl -IL</code>Proto/header issuesTrust X-Forwarded-Proto, fix redirects
Empty external addressInfra<code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">kubectl get svc -A</code>Pending LBConfigure LB/NodePort
Works by IP onlyIngress rule<code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">curl --resolve</code>Host mismatchCorrect host rules
## Ingress versus Gateway API Ingress remains widely used. Gateway API offers richer routing and role separation, but still requires a compatible controller and the same disciplined troubleshooting across DNS, Services, EndpointSlices, Pods, and TLS. Migrating does not eliminate these fundamentals. ## Final troubleshooting checklist - Controller installed and Pods Ready - Controller Service reachable (EXTERNAL-IP/NodePort) - Correct spec.ingressClassName - DNS resolves to the Ingress endpoint - Host and path rules correct, proper pathType - Backend Service name and port correct - Service selector matches Pod labels - EndpointSlices contain Ready addresses - Application listens on 0.0.0.0 - TLS Secret exists in same namespace; cert matches hostname and is valid - NetworkPolicy/firewalls allow controller <-> Pods and external <-> controller - Controller and application logs inspected ## Conclusion Most Kubernetes Ingress errors are not YAML typos; they are broken links in the end-to-end request path. Start at DNS and the external endpoint, move through the controller and Ingress rules, and finish at the Service, EndpointSlices, and Pods. Distinguish controller 404s from application 404s, and validate TLS without guessing. Reliable fixes come from isolating the failing layer, proving each hop with commands, and only then changing configuration.