Retiring ingress-nginx across 30 clusters without a single outage

The community-maintained ingress-nginx controller, the one wired into a huge share of Kubernetes ingress paths in the wild, was retired in March 2026. Kubernetes SIG Network and the Security Response Committee made the call official on November 12, 2025: best-effort maintenance only until the March 2026 cutoff, and after that, no releases, no bugfixes, no CVE patches, permanently. The repository is now archived and read only in place, with its final release on March 19, 2026.

Get this distinction right, because I have watched it get muddled in more than one planning meeting. The Ingress API resource in Kubernetes, networking.k8s.io/v1, is not going anywhere. It stays GA, feature frozen, fully supported. The thing that was retired is the open source controller that implements it, a project a couple of volunteer maintainers carried for years until they ran out of runway.

This was never a purely academic problem. In March 2025, CVE-2025-1974 landed, an unauthenticated remote code execution through exposed admission webhooks, and it was a live demonstration of exactly how much blast radius a flaw in the ingress layer buys an attacker. Every request into every namespace behind that controller runs through it. Now that the project is past its retirement date, if something like that shows up again, there is no patch coming. And "EOL software sitting in the L7 data path" is not a theoretical audit finding, it is the line item that stalls SOC 2, PCI-DSS, ISO 27001, and HIPAA attestations and holds up production promotions until someone can show a remediation date.

Numbers first, because they set the constraint that shaped everything downstream. 30 clusters. 120 namespaces across healthcare and banking platforms with audit-sensitive production paths. Every one of them was fronted by ingress-nginx, most carrying live production traffic with real SLAs behind them, and a subset regulated in ways that made "redeploy it and see what breaks" not a plan anyone was going to sign off on.

A footprint that size changes the shape of the problem. It stops being "swap the ingress controller" and becomes "replatform the entry point to the org's Kubernetes estate while it stays open for business." Every nginx.ingress.kubernetes.io/* annotation any application team had ever copied into a manifest became technical debt with a deadline attached, because none of it carries over to a different controller as-is. Every websocket, every long-lived stream, every custom timeout someone tuned two years ago and forgot about, all of it had to be rediscovered before it could be moved safely.

We did not touch 30 clusters at once and we did not touch them in cluster-ID order. We grouped them into waves: a small canary wave to prove the framework itself worked, an early adopter wave of teams with enough test coverage and low enough traffic to tolerate a rough edge, a bulk wave that carried most of the estate, and a long tail wave of regulated and legacy clusters that needed extra soak time and, in a couple of cases, extra sign-off before we touched them at all.

The instinct in a lot of platform orgs, the moment "we have to replace our ingress controller" turns into "let's finally get a real service mesh," is understandable, and for us, wrong. A service mesh and an ingress controller solve different problems, and it is worth being blunt about the difference before picking a tool.

An ingress controller answers north-south questions: traffic coming from outside the cluster, how it gets routed, TLS terminated, load balanced, to the right service. A service mesh answers east-west questions: how services inside the cluster talk to each other, how that traffic gets encrypted and authorized, how you do fine-grained shaping and observability between your own workloads. We had an EOL problem in the north-south layer. We did not have an unsolved problem in the east-west layer, and importing one just because we were already elbow-deep in the networking stack would have been scope creep dressed up as ambition.

Istio's sidecar model means an Envoy proxy in every pod, in every one of 120 namespaces, a permanent tax on CPU and memory across the estate, plus a control plane, istiod, with its own upgrade cadence, cert rotation, and injection webhook to operate. Istio's newer ambient mode, GA since version 1.24 in November 2024, removes the per-pod sidecar in favor of a per-node ztunnel and optional waypoint proxies, and the public benchmarks on it are genuinely good, something in the range of a 70 percent memory reduction against the sidecar model. It is a real answer if what you want is a mesh without the sidecar tax. But you are still running istiod and still carrying the full Istio CRD surface to solve a problem, mTLS and east-west policy, that we did not actually have. Adopting a full mesh, sidecar or ambient, to fix an ingress EOL problem means buying a service nobody asked for and now owning it forever.

Contour was the other serious contender, an Envoy-based ingress controller with solid Gateway API support and a CNCF pedigree. It lost mostly on ecosystem depth and day-two experience, a smaller contributor base and a thinner middleware story than traefik's, at the exact moment we needed a platform team's worth of application teams to self-serve routing config without filing tickets.

Kong Ingress Controller earns a specific callout for the irony sitting inside it. Kong's own data plane is built on nginx and OpenResty. Choosing Kong to get away from an nginx problem does not actually get you away from nginx, it gets you a different nginx fork with a different maintenance model, and past a certain scale, a licensing conversation between Kong OSS and Kong Enterprise that has nothing to do with the technical problem you started with.

What tipped it to traefik, past the resource footprint argument, was forward compatibility. Kubernetes' own retirement notice for ingress-nginx explicitly points teams toward Gateway API as the modern standard, not toward another vendor's implementation of the legacy Ingress spec. Traefik has been contributing to Gateway API since its early SIG Network days, has been production ready on it since v3.1, and reports full core conformance across the recent spec releases, with a growing set of extended features. Picking traefik meant we were not doing this migration twice, once now to escape ingress-nginx and again in two years to escape the legacy Ingress API. We could run traefik on plain Ingress resources today and walk teams onto Gateway API's HTTPRoute and GRPCRoute resources on their own timeline, on a controller that already speaks both fluently.

"Migrate the ingress controller" is a sentence that hides an enormous amount of risk if you say it and then just do it. A bad deploy in the ingress layer does not take down one service, it takes down every namespace that controller fronts. With 120 namespaces behind a single layer, a naive cutover was never going to survive contact with a change advisory board, let alone with whoever was on call that week. Regulated clusters needed extra soak. New NLB IPs and IP-based firewall rules had to land before any class flip. 120 namespaces of annotation rewrites could not serialize through a platform review queue.

So the migration was not "delete nginx, install traefik." It was a parallel-run framework designed so that traefik was never the only thing standing between the internet and production, until the data said it was safe to be.

Both controllers ran live in every cluster at the same time, each with its own deployment and its own target group at the load balancer layer, each pointed at the same backend services through routing rules kept in parity for the duration of the transition. Traffic weight shifted to traefik progressively, cluster by cluster and in some cases namespace by namespace, starting small, single-digit percent, holding at each stage for a soak window, stepping up only when the numbers said it was safe to.

What decided "safe" was signal, not a clock. Every promotion was gated on the same reliability signals the platform already tracks, latency, error rate, saturation, and a burn rate breach during a soak window snapped the weight back to nginx automatically. Nginx stayed up the whole time, running at whatever traffic percentage the current stage called for, sometimes zero. Rollback meant changing a number in the load balancer config, not paging anyone.

A cluster only lost its nginx path once it had sat clean at 100 percent traefik through a full soak window, and even then we scaled ingress-nginx to zero replicas before deleting anything, because the cheapest insurance policy in a migration like this is the one you do not have to redeploy under pressure.

That was the platform soak. Once traefik was taking weight and looking clean, app teams moved their own hostnames with one field: ingressClassName from nginx to traefik. Each cluster already had a second NLB in front of traefik, new IPs, up to four per region. Platform external-dns is scoped to --ingress-class=traefik, so it does not fight team-owned external-dns installs, and it writes a specific A record for that hostname pointing at the new NLB. Specific beats wildcard, so traffic leaves the nginx NLB without anyone touching Route53. Flip the class back, the specific record goes away, traffic falls onto the cluster wildcard, and nginx is serving again in about a minute.

Traefik does not read nginx.ingress.kubernetes.io/* annotations. Host, path, TLS secret, and ingressClassName stay on the Ingress. Almost everything else, rewrites, CORS, IP allowlists, rate limits, custom headers, forward auth, body size, moves into a Middleware custom resource in the same namespace, then gets attached with one annotation:

traefik.ingress.kubernetes.io/router.middlewares: my-ns-strip-prefix@kubernetescrd

The format is <namespace>-<name>@kubernetescrd. Get the namespace wrong and traefik logs "middleware not found," and the route silently loses the rewrite, the allowlist, or the CORS headers. Chain several with commas; they run in the order listed. Create the Middleware CRs before you flip the class, or there is a window where traefik is live and the behavior is not.

Stay on Ingress plus Middleware CRs unless you actually need ServersTransport, TCP/UDP, or matchers Ingress cannot express. Traefik's timeout defaults are already more permissive than nginx's: responseHeaderTimeout is unlimited, dialTimeout is 30s. Copying a 120s nginx value onto a new CRD because the annotation existed is how you invent 504s on work that used to finish.

Once traefik was live in a cluster, the remaining work was pattern-driven and repetitive: read the nginx annotations, emit Middleware CRs, rewrite the Ingress, validate. 120 namespaces of that, serialized through a platform review queue, is how you miss the deadline.

We shipped app teams an AI skill wired to the annotation inventory from discovery. Structured translation rules, plus checkpoints the generated change had to pass: ingressClassName is traefik, nginx annotations are gone, the middleware exists in the same namespace, and the router.middlewares string matches namespace-name@kubernetescrd. Snippets the mapping could not parse got flagged for a human. The skill does not invent traefik config.

Teams opened their own PRs, reviewed the diff, merged on their own timeline. Platform engineering stopped being the reviewer on every namespace rewrite. That is the only reason under 30 days was possible at this scope. The skill itself is reproduced at the end of this writeup, the same artifact we put in their hands.

Somewhere in the middle of this, the platform team hit a wall that had nothing to do with traefik. A set of application namespaces needed to reach a centralized AuthN/AuthZ API gateway, the shared internal service fronting identity and authorization for a large slice of the org's applications, and the connection kept failing.

The existing pattern used a VPC endpoint, a private link into the gateway that avoided public routing entirely, the pattern you would design on a whiteboard if someone asked you to do this from scratch. In practice it was not working reliably, and it was not the kind of failure a single config change was going to fix. It took real design time, multiple sessions across teams, to work through the tradeoffs of the private-link path against the alternative: dropping the VPC endpoint and reaching the gateway through an explicitly opened, scoped firewall rule instead. Less elegant than a private link. Far easier to debug when it breaks. It worked.

This was not just our blocker. Every other team building against that same AuthN/AuthZ gateway had been fighting the identical VPC endpoint problem, mostly in isolation, each one assuming it was something specific to their own namespace or their own security group. Because the ingress migration forced us to actually solve it instead of routing around it, the fix and the reasoning behind it became something every other team could pick up instead of independently rediscovering the same design conversation. A plumbing problem on our project turned into an unblock for the board.

What we can say with certainty about the migration itself:

Beyond that, this is the kind of migration whose value shows up in telemetry that already exists elsewhere on the platform, so rather than round numbers to make a slide look fuller, here is what the framework should always be reporting, whether it is this migration or the next one:

1. 01A service mesh is not a consolation prize for an ingress migration. Do not buy operational surface you did not need yesterday just because the networking stack is already open on your laptop.
2. 02Keep the old controller alive and warm until the data says you are done, not the calendar. A rollback should be a weight change, not an incident.
3. 03Annotations do not migrate themselves. Inventory every nginx.ingress.kubernetes.io/* annotation before touching a cluster. configuration-snippet has no traefik equivalent. The middleware reference is namespace-name@kubernetescrd, and getting the namespace wrong looks like a successful cutover with missing behavior.
4. 04nginx's implicit 1MB body limit does not come with you. Traefik will accept a 2GB POST unless you put a buffering middleware on that ingress. Set maxRequestBodyBytes and memRequestBodyBytes to the same number; they add.
5. 05FQDN firewall rules follow DNS. IP allowlists against the old NLB do not. Check which kind you have before you flip ingressClassName.
6. 06Long-lived connections are where migrations quietly die. Test server-sent events, websockets, and streaming gRPC explicitly. Do not assume proxy buffering behaves the same across controllers, because it does not.
7. 07The boring cross-team blocker you fix along the way is sometimes worth more than the migration itself. Do not route around a shared problem just because it is not technically your ticket.

With traefik as the standard, the next move is walking teams off plain Ingress resources onto Gateway API's HTTPRoute and GRPCRoute where it actually earns its keep: multi-protocol routing, cleaner separation between platform-owned and team-owned config, native support for the streaming and gRPC paths that gave us the most trouble this time around. Traefik already passes full core conformance on the spec, so this is an adoption curve, not another controller swap. Most teams should stay on Ingress plus Middleware CRs until a feature actually requires IngressRoute or Gateway API. Do not convert the estate twice for sport.

The service mesh question has not gone away either, it has just been separated from an EOL deadline, which is where it belonged in the first place. If a specific set of services genuinely need east-west mTLS or fine-grained traffic policy between each other, that is a real decision worth making deliberately, on its own timeline, evaluated on what it actually costs to run, not inherited as a rider on a migration that was never about it.

This is the artifact from section 06, cleaned up and published as we actually used it. Not a chatbot with a wiki link. A skill with translation rules, deterministic checkpoints, and a prompt that emits a migration script from one Ingress YAML. It can create Middleware CRs and rewrite the Ingress. It cannot invent Traefik config the mapping does not cover. configuration-snippet and server-snippet get flagged for a human.