Conversation
During the image migration, the temporary update-ee-<site> project's php container answers to the same php alias on the site network. nginx resolves that upstream only when it starts, so the recreated site nginx keeps two php IPs, and once the support containers are removed one of them is dead: requests routed to it fail with "No route to host" after ~3 s before nginx retries the live peer, intermittently, until the site nginx is reloaded. Reload the site nginx after its support containers are removed. A failed reload doesn't abort or roll back the upgrade; it logs a warning with the `ee site reload <site> --nginx` workaround. The support nginx has the same problem while it serves during the switch (it resolved the site's old php too), so it's reloaded, best effort, once the site's own containers are gone. HTML sites have no php upstream and are skipped.
Contributor
There was a problem hiding this comment.
Copilot review overview
🟢 Approval recommended
The reload ordering and best-effort failure handling correctly address stale upstream resolution without disrupting migrations.
Review effort: Balanced
Findings: None
What changed in this PR
Adds best-effort nginx reloads during WP/PHP container migrations to remove stale PHP upstream addresses.
Changes:
- Reloads support nginx after old site containers stop.
- Reloads site nginx after support containers stop.
- Centralizes support project-name generation.
| File | Description |
|---|---|
php/EE/Migration/SiteContainers.php |
Implements reload helpers and project naming. |
php/EE/Migration/Containers.php |
Adds reload steps for enabled WP/PHP sites. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
When a later step of the image migration fails, the rollback recreates each WP/PHP site's old containers while the support project's php still answers to the site's php alias, then removes the support project. The site nginx kept the dead support php as a peer, so some requests took ~3 s until `ee site reload <site> --nginx`. Add a first step per enabled WP/PHP site with a no-op up and a site nginx reload as its down. Undo runs in reverse, so the reload runs last for that site, after the support project is gone. A failed reload only warns.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
During the image migration, every enabled site whose images changed goes through
enable_support_containers()→disable_default_containers()→enable_default_containers()→disable_support_containers(). The temporaryupdate-ee-<site>project runs its own php container on the site network with the same<site>_phpalias that the site nginx'supstream phppoints to. nginx resolves that name only when it (re)loads its config:Requests nginx sends to the dead peer fail with
connect() failed (113: No route to host)after ~3 s, or hang until the client gives up, and then nginx retries the live peer. Users get intermittent slow requests on every recreated site until its nginx is reloaded or restarted: at most one perfail_timeout(10 s) window per worker. Any upgrade that bumps a site image hits this, and v4.13.0 recreates every WP and PHP site (postfix plus all php images are bumped). It was first seen on a realee cli update --nightlyof a populated 4.12.0 host, whereee site reload <site>cured it.The rollback path has the same problem. When a step fails late in the image migration (for example
update-compose, after every site was migrated), each site is undone in reverse: the support project comes back up, the old containers are recreated while the support php still answers to<site>_php, and only then is the support project removed. Nothing reloads the site nginx afterwards, so every rolled-back WP/PHP site keeps the dead support php as a peer.Fix
SiteContainers::reload_nginx()(nginx -t && nginx -s reload). The reload re-resolves<site>_phpto the one live container. It's a separate step per site (reload-site-nginx-<site>). If it fails, the upgrade isn't aborted or rolled back. Instead it logs a warning naming the site and the workaround:Could not reload nginx of <site> after recreating its containers. Some requests may take ~3 s until you run `ee site reload <site> --nginx`.reload-support-nginx-<site>, best effort, debug log on failure), so it doesn't stall requests while it serves during the switch.reload-site-nginx-on-rollback-<site>: a no-op up, and a site nginx reload as its down. Undo runs in reverse, so that reload runs last for the site, after its old containers are back and the support project is gone. It's only undone for sites whose steps ran, and a failed reload only warns, so the rollback of the other sites continues.Testing
GitHub Actions on a temp branch of my fork (deleted afterwards). The 4.12.0 release phar (md5
61d6600d…) is installed and creates 7 sites: WP, WP--cache, PHP--with-db, PHP--php=8.1, PHP--php=7.4, html, and one more PHP site for the failure test. Then the site is upgraded the wayee cli updatedoes it: the phar is replaced andee cli infois run. That's done to the published nightly4.13.0-nightly-edee0a1(before) or to a phar built from this branch withVERSION4.13.0-nightly-mig21fix(after). After the upgrade, each site gets 12 requests spaced 11 s apart (to crossfail_timeout), and its nginxerror.logis checked. Final run: https://github.com/mrrobot47/easyengine/actions/runs/36126366916 (an earlier run, 36124682270, gave the same picture).Requests over 2 s after the upgrade (of 12), and
No route to hostlines after the upgrade:--cache--with-dbee site reload <site> --nginxupstream: "fastcgi://10.0.5.5:9000", whilegetent hosts p81.test_phpin its nginx returns only10.0.5.2. The WP--cachesite didn't show it in either run; its nginx evidently resolved php only after the support php was gone.Executing reload-support-nginx-<site>/reload-site-nginx-<site>for the 6 WP/PHP sites and none for html. Every recreated site answered all 12 requests in under 0.5 s with no dead-peer errors.nginx -tfor one site right before its reload step. The upgrade still exits 0, the warning appears on stderr and in ee.log, and that site keeps the stale peer (as without the fix) untilee site reload <site> --nginx.php -lis clean on 7.4 and 8.5, and PHPCS shows no new violations on the changed lines.Rollback path
A small harness runs the real
RevertableStepProcessor,Containers::migrate_site_containers()andSiteContainerswith stubbed docker calls (php:7.4-cli and php:8.5-cli, no network). With a late failure, every WP/PHP site's reload runs once, afterenable-support-containers' undo, the support project's removal and the old containers' recreation, and nothing of that site is undone after it. html and disabled sites get no step. When a middle site's upgrade fails, only the sites that ran are reloaded. A failing reload logs the warning and the other sites still roll back. On success the step does nothing. 40/40 checks pass; on the previous head 20 of them fail.Live on a populated 4.12.0 host (WP, WP
--cache, PHP--with-db, PHP 8.1, PHP 7.4, html), upgraded likeee cli updatewith a forcedupdate-composefailure after every site was migrated. Each run rolled back cleanly: version 4.12.0, old image tags, the old compose files and containers, noupdate-ee-*left, content unchanged. Then each site got 12 requests spaced 11 s apart. Requests over 2 s (of 12), andNo route to hostlines during the measurement:af2bd2c1)--cache--with-dbSYN-SENT) to the support php's old IP (10.0.x.5), whilegetentreturned only the live one. After:reload-site-nginx-on-rollback-<site>ran once per WP/PHP site, after its support containers were removed, with no warnings. There were noNo route to hosterrors and no connections to a dead peer. Over two rounds (120 requests), two requests (one each on WP and PHP--with-db) hung 10 s on the live php with no error logged. That's php-fpm'sondemandidle reaping, which also happens on 4.12.0 and isn't related to this change.reload-support-nginx-<site>andreload-site-nginx-<site>steps run for the 5 WP/PHP sites, html is untouched, and 12 requests per site give 0 slow and 0No route to host. A nightly re-run changes nothing (containers, compose files, options, migrations,.backup) and recreates or reloads nothing.Alternatives considered
up -d --no-deps nginxwould leave the support nginx with no php once the site's containers stop. Giving the support project a different alias needs a separate compose file or override.resolver 127.0.0.11 valid=…plus a variable infastcgi_pass) would change the nginx templates in site-type-wp and site-type-php, needs a migration to rewrite existing sites' configs, and drops theupstreamblock. Much larger than a reload.This should land before the core v4.13.0 tag, since v4.13.0 recreates every WP and PHP site on upgrade.