Note: the code lives in zosbase (pkg/gateway), filing here as the tracking repo. Same bug affects pkg/gateway_light.
What happens
When a gateway workload is deleted, we remove its traefik routing config but leave its TLS certificate in traefik's acme.json. Traefik keeps trying to renew that certificate every day, forever.
On a long-lived public node this piles up. Real log from a mainnet gateway — 19 renewals failing in one 60-second sweep, all for domains whose deployments are long gone:
09:39:52 error msg="Error renewing certificate from LE: {test.fqdn.main.gridtesting.xyz []}"
error="acme: error: 400 :: urn:ietf:params:acme:error:dns :: DNS problem: NXDOMAIN
looking up A for test.fqdn.main.gridtesting.xyz - check that a DNS record exists"
...
09:40:52 error msg="Error renewing certificate from LE: {lhumina.com []}"
error="acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: 168.119.77.253:
Invalid response from https://lhumina.com/.well-known/acme-challenge/...: 404"
17x NXDOMAIN (domain deleted), 2x unauthorized (domain moved to another host). Several are gridtesting.xyz CI leftovers.
Why
DeleteNamedProxy — the single delete path for both name and fqdn workloads — removes the dynamic config file and the reserved-domain entry, but never touches acme.json:
pkg/gateway/gateway.go:739pkg/gateway_light/gateway.go:764 (identical)
verifyDomainDestination (pkg/gateway/gateway.go:488) only runs at deploy time, so a domain that later disappears is never noticed.
Impact
Cosmetic today — live workloads renew fine, and LE's failed-validation limit is per-hostname so we aren't rate-limited. But the list only grows over a node's lifetime, and it makes gateway logs useless for spotting real problems. Every gateway node that has ever served a workload has this.
Suggested fix
Prune acme.json on gateway startup, in ensureGateway, before traefik starts: drop any entry whose domain has no matching file in the dynamic config dir.
Doing it at startup rather than in DeleteNamedProxy avoids racing traefik for the file (traefik owns acme.json while running and doesn't watch it for changes), and it self-heals nodes that have already accumulated junk. Same change needed in gateway_light.
What happens
When a gateway workload is deleted, we remove its traefik routing config but leave its TLS certificate in traefik's
acme.json. Traefik keeps trying to renew that certificate every day, forever.On a long-lived public node this piles up. Real log from a mainnet gateway — 19 renewals failing in one 60-second sweep, all for domains whose deployments are long gone:
17x NXDOMAIN (domain deleted), 2x unauthorized (domain moved to another host). Several are
gridtesting.xyzCI leftovers.Why
DeleteNamedProxy— the single delete path for both name and fqdn workloads — removes the dynamic config file and the reserved-domain entry, but never touchesacme.json:pkg/gateway/gateway.go:739pkg/gateway_light/gateway.go:764(identical)verifyDomainDestination(pkg/gateway/gateway.go:488) only runs at deploy time, so a domain that later disappears is never noticed.Impact
Cosmetic today — live workloads renew fine, and LE's failed-validation limit is per-hostname so we aren't rate-limited. But the list only grows over a node's lifetime, and it makes gateway logs useless for spotting real problems. Every gateway node that has ever served a workload has this.
Suggested fix
Prune
acme.jsonon gateway startup, inensureGateway, before traefik starts: drop any entry whose domain has no matching file in the dynamic config dir.Doing it at startup rather than in
DeleteNamedProxyavoids racing traefik for the file (traefik ownsacme.jsonwhile running and doesn't watch it for changes), and it self-heals nodes that have already accumulated junk. Same change needed ingateway_light.