## Intro Kubernetes Ingress upgrades touch one of the most sensitive edges of your platform: how users reach your applications. A good plan lets you move to supported APIs, modern controllers, and better traffic policies without breaking routes or TLS. This guide provides a practical, low-risk approach to upgrading and migrating Kubernetes Ingress, from inventory and planning through verification, failure modes, and rollback. What you will get: - Clear prerequisites and environment inventory steps. - Three safe paths: API-only upgrade, in-place controller upgrade, and parallel migration. - Concrete YAML, commands, and expected results. - Failure modes, rollback, and a repeatable checklist. The first pilot should be narrow, measurable, and easy to inspect locally and in staging before any production change. Keep scope tight, verify thoroughly, and then scale out. ## Version and Environment Inventory Before changing anything, capture a precise picture of what you have. This tells you which upgrade path is viable and prevents surprises. ### Identify cluster and API capabilities - Check cluster version: kubectl version --short - Confirm available Ingress API groups and preferred versions: kubectl api-resources | grep -i ingress kubectl explain ingress | head -n 5 Expected: networking.k8s.io/v1 is available on modern clusters (1.19+). If you still rely on extensions/v1beta1 , plan an API migration first. ### Find your Ingress controllers and classes - List IngressClass resources: kubectl get ingressclass - Find controller deployments (commonly in namespaces like ingress-nginx , nginx-ingress , traefik , or vendor-specific): kubectl get deploy,ds,svc -A | grep -i ingress - Inspect the IngressClass for controller string (example for ingress-nginx): kubectl get ingressclass -o yaml Expected: You see something like spec.controller: k8s.io/ingress-nginx for the community ingress-nginx controller. ### Map traffic and TLS - List Ingresses and their addresses: kubectl get ing -A -o wide - Inventory hosts, paths, and TLS secrets (note: TLS secrets live in the same namespace as the Ingress): kubectl get ing -A -o jsonpath='{range .items[*]}{.metadata.namespace} {.metadata.name} {.spec.rules[*].host} {.spec.tls[*].secretName}{"\n"}{end}' - For each public Ingress, identify DNS records and current TTLs. Lower TTLs ahead of any cutover you plan (for example, from 3600 to 60 seconds). - Risk triage for incompatible features: - Note any annotations used for rewrites, timeouts, authentication, or canary. Some may behave differently between controller versions. - Identify backends using named ports vs numeric ports, and services with nonstandard health checks. ### Ingress API changes at a glance Area v1beta1 networking.k8s.io/v1 Notes API group extensions/v1beta1 networking.k8s.io/v1 v1beta1 removed in newer clusters Backend reference serviceName/servicePort backend.service.name/port.number or port.name Structure changed in v1 Path type implicit pathType: Prefix, Exact, or ImplementationSpecific You must set pathType in v1 Class selection annotation: kubernetes.io/ingress.class spec.ingressClassName Prefer ingressClassName in v1 TLS secret same namespace as Ingress same No change; verify namespace alignment ## Safe Configuration Path Choose the smallest change that achieves your goal. Three proven options: ### A) API-only upgrade (keep your controller) - When: Your controller already supports networking.k8s.io/v1 , and you only need to update resource manifests. - Benefit: Minimal moving parts. Ideal for first pilot. ### B) In-place controller upgrade (same class and Service) - When: You want new controller features or CVE fixes but will keep the same LoadBalancer or NodePort Service. - Benefit: No DNS change. Keep rollouts tight with surge and zero-downtime settings. ### C) Parallel controller migration (new class and Service) - When: You will replace the controller family or change major configuration. You run old and new controllers in parallel with different IngressClass values. - Benefit: Safe canary or host-by-host cutover. Easy rollback by switching ingressClassName or DNS weight. ### A) API-only upgrade example Suppose you have this legacy Ingress (constructed example): apiVersion: extensions/v1beta1 kind: Ingress metadata: name: web-legacy namespace: prod annotations: kubernetes.io/ingress.class: nginx spec: tls: - hosts: ["app.example.com secretName: tls-app rules: - host: app.example.com http: paths: - path: / backend: serviceName: web servicePort: 80 Migrate to v1 with explicit pathType and structured backend: apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: web namespace: prod spec: ingressClassName: nginx tls: - hosts: - app.example.com secretName: tls-app rules: - host: app.example.com http: paths: - path: / pathType: Prefix backend: service: name: web port: number: 80 Apply safely: kubectl apply -f web-ingress-v1.yaml kubectl describe ing -n prod web Expected: Address remains the same, TLS secret is found, and rules match your old Ingress. ### B) In-place controller upgrade example (ingress-nginx) - Confirm existing class and Service: kubectl get ingressclass kubectl get svc -n ingress-nginx - Stage a controlled deployment upgrade (constructed commands; adapt to your tooling): - Ensure controller Deployment has a PodDisruptionBudget and readinessProbes are green. - Set a safe rollout strategy (example): kubectl patch deploy -n ingress-nginx ingress-nginx-controller \ -p '{"spec":{"strategy":{"type":"RollingUpdate","rollingUpdate":{"maxUnavailable":0,"maxSurge":1}}}}' - Upgrade the controller image to the target version, then monitor: kubectl set image deploy/ingress-nginx-controller -n ingress-nginx \ controller=registry.k8s.io/ingress-nginx/controller:v1.9.6 kubectl rollout status deploy/ingress-nginx-controller -n ingress-nginx --timeout=3m kubectl logs deploy/ingress-nginx-controller -n ingress-nginx --tail=100 Expected: No drop in endpoints, Service external IP remains, and existing Ingresses re-sync cleanly. Rollback (if needed): kubectl rollout undo deploy/ingress-nginx-controller -n ingress-nginx ### C) Parallel controller migration example (class-based cutover) Goal: Run a new controller in parallel, bind only selected Ingresses to it, verify, then move more traffic. - Create a new IngressClass for the new controller (constructed example): apiVersion: networking.k8s.io/v1 kind: IngressClass metadata: name: nginx-new spec: controller: k8s.io/ingress-nginx - Deploy the new controller referencing this class (follow your vendor or chart docs). Ensure it creates a distinct Service such as ingress-nginx-new-controller with its own external IP. - Duplicate a single low-risk Ingress to target the new class: apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: web-new namespace: prod spec: ingressClassName: nginx-new tls: - hosts: - app-canary.example.com secretName: tls-app rules: - host: app-canary.example.com http: paths: - path: / pathType: Prefix backend: service: name: web port: number: 80 - Create DNS for app-canary.example.com pointing to the new controller Service IP with a short TTL (for example, 60 seconds). Verify end-to-end before moving production hosts. - Host-by-host cutover: change the production Ingress to ingressClassName: nginx-new and re-apply. Alternatively, use weighted DNS to split traffic across the old and new controller Services during the transition. ### Canary within a single controller (feature flag) If you only need to canary a new backend version (not a new controller), you can use a canary Ingress with ingress-nginx. Primary Ingress (stable): apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: web-stable namespace: prod spec: ingressClassName: nginx rules: - host: app.example.com http: paths: - path: / pathType: Prefix backend: service: name: web port: number: 80 Canary Ingress (10% traffic to web-canary service): apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: web-canary namespace: prod annotations: nginx.ingress.kubernetes.io/canary: "true" nginx.ingress.kubernetes.io/canary-weight: "10" spec: ingressClassName: nginx rules: - host: app.example.com http: paths: - path: / pathType: Prefix backend: service: name: web-canary port: number: 80 Increase the weight as you gain confidence. ## Verification and Diagnostics Validate each stage with explicit checks and expected signals. Keep TTLs low and observe for a full error budget window before proceeding. ### Core checks and expected signals
CheckCommandExpected result
API supportkubectl api-resourcesIngress shows networking.k8s.io/v1
Ingress statuskubectl get ing -A -o wideADDRESS column populated for each Ingress
Class mappingkubectl get ingressclass -o wideCorrect class names and controller strings
Eventskubectl describe ing NAME -n NAMESPACENo errors: TLS secret found, backends resolved
Controller logskubectl logs deploy/ingress-nginx-controller -n ingress-nginxSync cycles without error or crashloop
Backend healthcurl -I https://host/healthzHTTP 200/204; latency within normal bounds
### Additional diagnostics - Confirm generated config (ingress-nginx): # Identify a controller pod POD=$(kubectl get pods -n ingress-nginx -l app.kubernetes.io/component=controller -o jsonpath='{.items[0].metadata.name}') # Inspect a snippet of the generated NGINX config kubectl exec -n ingress-nginx $POD -- cat /etc/nginx/nginx.conf | head -n 50 - Validate TLS secret content and namespace alignment: kubectl get secret tls-app -n prod -o yaml | grep -E 'tls\.crt|tls\.key' - Confirm Service endpoints: kubectl get endpoints web -n prod -o yaml | grep addresses -A2 - Load test a small sample (constructed example): for i in $(seq 1 50); do curl -s -o /dev/null -w "%{http_code} %{time_total}\n" https://app.example.com/healthz; done | sort | uniq -c Expected: Stable 200s with consistent timings. ## Failure Modes and Recovery Plan for failure so you can recover quickly and confidently. ### 1. 404 Not Found after cutover - Cause: Wrong host, path mismatch, or missing default backend. In v1, pathType is required and changes matching behavior. - Fix: Ensure pathType: Prefix or Exact as intended, confirm host spelling, and that the backend service exists. - Rollback: Re-apply the last known-good Ingress manifest (keep it on hand), or switch ingressClassName back to the old class if you used parallel migration. ### 2. 502/504 from upstream - Cause: Service endpoints not ready, port mismatch, or readiness checks failing. - Fix: Verify service selectors, endpoints, and pod readiness. Confirm backend port number vs name. - Rollback: Scale canary weight to 0 or revert Ingress to stable backend. ### 3. TLS handshake or certificate errors - Cause: TLS secret not found, wrong namespace, or missing SAN for the host. - Fix: Ensure the secret exists in the Ingress namespace and contains valid tls.crt and tls.key for the host. - Rollback: Reinstate the previous TLS secret and re-apply. ### 4. Controller never assigns an address - Cause: LoadBalancer provisioning delay, cloud permissions, or misconfiguration. - Fix: Inspect Service events, cloud provider logs, and controller logs. Verify Service type and annotations. - Rollback: Keep old controller as the primary; avoid DNS change until the new Service has an address. ### 5. Annotation behavior change - Cause: Some annotations may differ between controller versions. - Fix: Compare supported annotations, test on a canary host, and remove or adjust deprecated keys. - Rollback: Restore the previous annotation set and controller version. ### 6. Unexpected routing after API migration - Cause: Assumptions from v1beta1 path matching no longer hold. - Fix: Set pathType explicitly; for regex or rewrites, ensure controller-specific annotations are correct. - Rollback: If supported by your cluster, revert to the previous manifest while you correct paths. ### Rollback playbooks ### A) API-only upgrade rollback # Re-apply the previously working manifest kubectl apply -f web-ingress-previous.yaml # Confirm status kubectl describe ing -n prod web ### B) In-place controller upgrade rollback # Roll back the deployment to the prior ReplicaSet kubectl rollout undo deploy/ingress-nginx-controller -n ingress-nginx # Watch pods return to Ready kubectl rollout status deploy/ingress-nginx-controller -n ingress-nginx --timeout=3m ### C) Parallel controller migration rollback - Repoint production Ingress from ingressClassName: nginx-new back to ingressClassName: nginx and apply. - If you used DNS split, set weight back to 100% old controller and 0% new controller. - Remove the canary or duplicate Ingresses targeting the new class once stable. Verification after rollback: kubectl get ing -A -o wide kubectl logs deploy/ingress-nginx-controller -n ingress-nginx --tail=100 curl -I https://app.example.com/healthz Expected: Previous behavior restored with healthy responses. ## Operations Checklist Use this checklist to guide each upgrade or migration. Mark items done; do not skip verification gates. ### Planning and inventory - [ ] Record cluster version and confirm networking.k8s.io/v1 availability. - [ ] List all Ingresses, hosts, paths, and TLS secrets. - [ ] Identify current Ingress controllers, classes, and Services. - [ ] Lower DNS TTLs for affected hosts. ### Pilot selection - [ ] Choose a single low-risk host or path as the first pilot. - [ ] Decide on approach: API-only, in-place upgrade, or parallel migration. - [ ] Prepare manifests for the pilot, including pathType and ingressClassName . ### Change execution - [ ] Apply changes to a non-production environment first; verify end-to-end. - [ ] For in-place controller upgrades, ensure surge/zero-unavailable rollout. - [ ] For parallel migration, create a new IngressClass and new controller Service. - [ ] If canarying, start at 5-10% weight and observe. ### Verification - [ ] Check Ingress status and ADDRESS fields. - [ ] Review controller logs for errors and config reloads. - [ ] Validate TLS certs and SANs for all hosts. - [ ] Probe health endpoints; confirm latency and error rates. ### Decision and scale-out - [ ] Hold steady state for an agreed observation window. - [ ] Increase canary weight or move additional hosts. - [ ] Document any annotation or behavior changes discovered. ### Rollback readiness - [ ] Keep prior manifests and controller versions noted. - [ ] Have DNS weights or ingressClassName toggles prepared. - [ ] Verify rollback commands in a staging environment. ## Conclusion Upgrading and migrating Kubernetes Ingress safely is about scoping the change, verifying at each step, and keeping a clean path to rollback. Start with an API-only upgrade where possible, then proceed to in-place controller upgrades or parallel migrations when you need new features or a different controller family. Keep pilots small, use explicit pathType and ingressClassName , validate with logs and probes, and store known-good manifests for instant recovery. With a short, repeatable checklist and a narrow first pilot, you can modernize Ingress with confidence and minimal disruption.