Skip to content

fix(migration): reload site nginx after its containers are recreated - #1937

Open
mrrobot47 wants to merge 2 commits into
EasyEngine:developfrom
mrrobot47:fix/reload-site-nginx-after-migration
Open

mrrobot47 wants to merge 2 commits into
EasyEngine:developfrom
mrrobot47:fix/reload-site-nginx-after-migration

Conversation

@mrrobot47

@mrrobot47 mrrobot47 commented Sep 25, 2026 •

Copy link
Copy Markdown
Member

Problem

During the image migration, every enabled site whose images changed goes through enable_support_containers() → disable_default_containers() → enable_default_containers() → disable_support_containers(). The temporary update-ee-<site> project runs its own php container on the site network with the same <site>_php alias that the site nginx's upstream php points to. nginx resolves that name only when it (re)loads its config:

  • The recreated site nginx starts while the support php is still up, so it resolves two php IPs. Once the support containers are removed, one of them is dead.
  • The support nginx has the same problem in reverse: it starts while the site's old php is still up, and that php is removed right after.

Requests nginx sends to the dead peer fail with connect() failed (113: No route to host) after ~3 s, or hang until the client gives up, and then nginx retries the live peer. Users get intermittent slow requests on every recreated site until its nginx is reloaded or restarted: at most one per fail_timeout (10 s) window per worker. Any upgrade that bumps a site image hits this, and v4.13.0 recreates every WP and PHP site (postfix plus all php images are bumped). It was first seen on a real ee cli update --nightly of a populated 4.12.0 host, where ee site reload <site> cured it.

The rollback path has the same problem. When a step fails late in the image migration (for example update-compose, after every site was migrated), each site is undone in reverse: the support project comes back up, the old containers are recreated while the support php still answers to <site>_php, and only then is the support project removed. Nothing reloads the site nginx afterwards, so every rolled-back WP/PHP site keeps the dead support php as a peer.

Fix

  • After a site's support containers are removed, reload the site nginx with the existing SiteContainers::reload_nginx() (nginx -t && nginx -s reload). The reload re-resolves <site>_php to the one live container. It's a separate step per site (reload-site-nginx-<site>). If it fails, the upgrade isn't aborted or rolled back. Instead it logs a warning naming the site and the workaround: Could not reload nginx of <site> after recreating its containers. Some requests may take ~3 s until you run `ee site reload <site> --nginx`.
  • Once the site's own containers are gone, also reload the support nginx (reload-support-nginx-<site>, best effort, debug log on failure), so it doesn't stall requests while it serves during the switch.
  • For the rollback path, each enabled WP/PHP site's sequence starts with reload-site-nginx-on-rollback-<site>: a no-op up, and a site nginx reload as its down. Undo runs in reverse, so that reload runs last for the site, after its old containers are back and the support project is gone. It's only undone for sites whose steps ran, and a failed reload only warns, so the rollback of the other sites continues.
  • Only WP and PHP sites get these steps; html sites have no php upstream. The support project name is computed in one helper now, instead of being repeated.

Testing

GitHub Actions on a temp branch of my fork (deleted afterwards). The 4.12.0 release phar (md5 61d6600d…) is installed and creates 7 sites: WP, WP --cache, PHP --with-db, PHP --php=8.1, PHP --php=7.4, html, and one more PHP site for the failure test. Then the site is upgraded the way ee cli update does it: the phar is replaced and ee cli info is run. That's done to the published nightly 4.13.0-nightly-edee0a1 (before) or to a phar built from this branch with VERSION 4.13.0-nightly-mig21fix (after). After the upgrade, each site gets 12 requests spaced 11 s apart (to cross fail_timeout), and its nginx error.log is checked. Final run: https://github.com/mrrobot47/easyengine/actions/runs/36126366916 (an earlier run, 36124682270, gave the same picture).

Requests over 2 s after the upgrade (of 12), and No route to host lines after the upgrade:

Site Before (nightly) After (this branch)
WP 4 slow, 4 errors 0, 0
WP --cache 0, 0 0, 0
PHP --with-db 3, 3 0, 0
PHP 8.1 4 (one hung past the 10 s client timeout), 3 0, 0
PHP 7.4 5 (one hung 10 s), 3 0, 0
html (not recreated) 0, 0 0, 0
PHP (forced reload failure) 4, 3 5, 3; then 0 of 6 after ee site reload <site> --nginx
  • Before: nginx still used a peer that's gone. For example, PHP 8.1's error log shows upstream: "fastcgi://10.0.5.5:9000", while getent hosts p81.test_php in its nginx returns only 10.0.5.2. The WP --cache site didn't show it in either run; its nginx evidently resolved php only after the support php was gone.
  • After: the ee.log has Executing reload-support-nginx-<site> / reload-site-nginx-<site> for the 6 WP/PHP sites and none for html. Every recreated site answered all 12 requests in under 0.5 s with no dead-peer errors.
  • During the upgrade (0.5 s probe per site): before, each WP/PHP site had repeated 3 s and 8 s (probe timeout) stalls through and after its switch. After, there were at most 1–2 hiccups per site, right when its old containers stopped.
  • Forced failure: the test breaks nginx -t for one site right before its reload step. The upgrade still exits 0, the warning appears on stderr and in ee.log, and that site keeps the stale peer (as without the fix) until ee site reload <site> --nginx.
  • Core Behat suite on the branch (with the Behat coverage suite and workflow merged in): https://github.com/mrrobot47/easyengine/actions/runs/36124682188. On PHP 7.4 and 8.5: main 242 passed, migration 1, uninstall 1; the known-bug lane fails its 19 scenarios, the same set as on develop.
  • php -l is clean on 7.4 and 8.5, and PHPCS shows no new violations on the changed lines.

Rollback path

A small harness runs the real RevertableStepProcessor, Containers::migrate_site_containers() and SiteContainers with stubbed docker calls (php:7.4-cli and php:8.5-cli, no network). With a late failure, every WP/PHP site's reload runs once, after enable-support-containers' undo, the support project's removal and the old containers' recreation, and nothing of that site is undone after it. html and disabled sites get no step. When a middle site's upgrade fails, only the sites that ran are reloaded. A failing reload logs the warning and the other sites still roll back. On success the step does nothing. 40/40 checks pass; on the previous head 20 of them fail.

Live on a populated 4.12.0 host (WP, WP --cache, PHP --with-db, PHP 8.1, PHP 7.4, html), upgraded like ee cli update with a forced update-compose failure after every site was migrated. Each run rolled back cleanly: version 4.12.0, old image tags, the old compose files and containers, no update-ee-* left, content unchanged. Then each site got 12 requests spaced 11 s apart. Requests over 2 s (of 12), and No route to host lines during the measurement:

Site Before (previous head af2bd2c1) After (this head)
WP 5 (one hung 10 s), 4 1 (hung 10 s on the live php, see below), 0
WP --cache 0, 0 0, 0
PHP --with-db 4, 4 0, 0
PHP 8.1 4, 4 0, 0
PHP 7.4 4, 4 0, 0
html (not recreated) 0, 0 0, 0
  • Before: during the measurement, each nginx sent connections (SYN-SENT) to the support php's old IP (10.0.x.5), while getent returned only the live one. After: reload-site-nginx-on-rollback-<site> ran once per WP/PHP site, after its support containers were removed, with no warnings. There were no No route to host errors and no connections to a dead peer. Over two rounds (120 requests), two requests (one each on WP and PHP --with-db) hung 10 s on the live php with no error logged. That's php-fpm's ondemand idle reaping, which also happens on 4.12.0 and isn't related to this change.
  • Retry with the fault removed: the upgrade succeeds, the forward reload-support-nginx-<site> and reload-site-nginx-<site> steps run for the 5 WP/PHP sites, html is untouched, and 12 requests per site give 0 slow and 0 No route to host. A nightly re-run changes nothing (containers, compose files, options, migrations, .backup) and recreates or reloads nothing.

Alternatives considered

  • Keeping the support php off the shared alias doesn't work with the same compose file. up -d --no-deps nginx would leave the support nginx with no php once the site's containers stop. Giving the support project a different alias needs a separate compose file or override.
  • Having nginx re-resolve per request (resolver 127.0.0.11 valid=… plus a variable in fastcgi_pass) would change the nginx templates in site-type-wp and site-type-php, needs a migration to rewrite existing sites' configs, and drops the upstream block. Much larger than a reload.

This should land before the core v4.13.0 tag, since v4.13.0 recreates every WP and PHP site on upgrade.

During the image migration, the temporary update-ee-<site> project's php container answers to the same php alias on the site network. nginx resolves that upstream only when it starts, so the recreated site nginx keeps two php IPs, and once the support containers are removed one of them is dead: requests routed to it fail with "No route to host" after ~3 s before nginx retries the live peer, intermittently, until the site nginx is reloaded.

Reload the site nginx after its support containers are removed. A failed reload doesn't abort or roll back the upgrade; it logs a warning with the `ee site reload <site> --nginx` workaround. The support nginx has the same problem while it serves during the switch (it resolved the site's old php too), so it's reloaded, best effort, once the site's own containers are gone. HTML sites have no php upstream and are skipped.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The reload ordering and best-effort failure handling correctly address stale upstream resolution without disrupting migrations.

Review effort: Balanced
Findings: None

What changed in this PR

Adds best-effort nginx reloads during WP/PHP container migrations to remove stale PHP upstream addresses.

Changes:

  • Reloads support nginx after old site containers stop.
  • Reloads site nginx after support containers stop.
  • Centralizes support project-name generation.
File Description
php/​EE/​Migration/​SiteContainers.php Implements reload helpers and project naming.
php/​EE/​Migration/​Containers.php Adds reload steps for enabled WP/PHP sites.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

When a later step of the image migration fails, the rollback recreates each WP/PHP site's old containers while the support project's php still answers to the site's php alias, then removes the support project. The site nginx kept the dead support php as a peer, so some requests took ~3 s until `ee site reload <site> --nginx`.

Add a first step per enabled WP/PHP site with a no-op up and a site nginx reload as its down. Undo runs in reverse, so the reload runs last for that site, after the support project is gone. A failed reload only warns.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants