Skip to content

Commit 6cf5154

Browse files
ralyodioclaude
andauthored
dev2-edge: pit.world and the Moshpit names on Caddy, every other site on nginx (#199)
* dev2-edge: pit.world and the Moshpit names on Caddy, every other site on nginx nginx's stream module takes :443 and routes by TLS server name (ssl_preread): pit.world, *.pit.world and the Moshpit names in dev2/edge/edge.json go to Caddy on 127.0.0.1:8453, everything else to nginx's https servers on 127.0.0.1:8444, both with a PROXY protocol header so client addresses survive. - dev2/dev2-edge: render / apply / route on|off / rollback / sweep / status - dev2/edge/edge-box: the box side; apply backs up /etc/nginx, sweeps every name twice, checks Caddy answers like nginx on its own port before any traffic moves, switches with one nginx reload, sweeps again and rolls back on any change - caddy-edge-certs (+ .path/.timer): Caddy loads bundles of the certificates certbot, pit-proxy-cert and moshpit-renew already renew; Caddy issues nothing - nginx-edge-listen (+ nginx.service drop-in, .path): rewrites any `listen 443` a tool writes later (install.sh, setup-origin.sh, certbot --nginx), which would otherwise pass nginx -t and then silently fail to bind on reload - dev2-site vhost renders the 8444 form when the edge is on - test/dev2-edge.test.ts Rehearsed on dev3 against a copy of dev2's nginx config: 287 names same status before/after, SNI routing, client IP in nginx and Caddy upstreams, websockets, http->https, no dropped requests across the switch and the rollback, renewal pickup, stray listen 443 repair, rollback restores /etc/nginx byte for byte. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * edge-box: backup tarball stays root 0600 (it holds the private keys) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * edge-box: only restore a backup apply made, only run a shipped bundle; route waits for the reload Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * edge-box: backups live in one fixed place Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * edge-box: the last-backup pointer is a fixed path Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * edge-box: a re-apply reloads a running Caddy instead of restarting it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 166e1bc commit 6cf5154

16 files changed

Lines changed: 1515 additions & 0 deletions

‎dev2/README.md‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -223,3 +223,56 @@ deploy-poll.sh --dry-run what one pass would deploy
223223

224224
Logs: `~/.local/state/deploy-poll/poll.log` and `<site>.log`.
225225

226+
227+
## The edge: pit.world on Caddy, everything else on nginx (`dev2-edge`)
228+
229+
Since 2026-10-19 nginx's `stream {}` module owns :443 on dev2 and routes each
230+
connection by TLS server name (`ssl_preread`; TLS is not terminated there):
231+
232+
| name | goes to |
233+
| --- | --- |
234+
| `pit.world`, `www.pit.world`, `*.pit.world` (gateways, `bbs.`/`social.`), the Moshpit names in `edge/edge.json` | Caddy, `127.0.0.1:8453` |
235+
| everything else (and `live.streaming.pit.world`, see `keep_on_nginx`) | nginx's own https servers, `127.0.0.1:8444` |
236+
237+
Both hops carry a PROXY protocol header, so nginx (`real_ip_header proxy_protocol`,
238+
`conf.d/00-edge-realip.conf`) and Caddy (`proxy_protocol` listener wrapper) see the
239+
visitor's address; rate limits, logs and ThreatCrush are unchanged. Port 80 is not
240+
touched: nginx still answers ACME and http->https for every name. UDP 443 (HTTP/3) is
241+
out of scope.
242+
243+
```
244+
dev2-edge render [--out DIR] [--off] the box files for edge.json (nothing touches dev2)
245+
dev2-edge apply --yes backup, sweep, Caddy up + checked on its own port, nginx switched, sweep again
246+
dev2-edge route off|on every name back to nginx (or to Caddy again): one nginx reload
247+
dev2-edge rollback [TARBALL] restore /etc/nginx from apply's backup (~anthony/backups/nginx-<ts>.tgz), stop Caddy
248+
dev2-edge sweep | status
249+
```
250+
251+
`apply` (via `edge/edge-box` on the box) takes the status of `https://<name>/` (verified
252+
and `-k`) and `http://<name>/` for every name nginx and Caddy serve, twice (names that
253+
differ between the two are set aside as flaky), starts Caddy and checks it answers each
254+
routed name exactly as nginx did before any visitor reaches it, switches nginx with one
255+
reload (the :443 socket passes from the http servers to the stream without a dropped
256+
connection), sweeps again, and rolls everything back if any name's status changed.
257+
258+
**Certificates.** Caddy issues nothing (`auto_https off`). `caddy-edge-certs` copies
259+
what the existing renewals keep current into `/etc/caddy/edge-certs/<host>.pem`
260+
(root:caddy 0640): certbot for `pit.world`, `pit-proxy-cert` for each `<name>.pit.world`,
261+
`moshpit-renew` for the Moshpit names. `caddy-edge-certs.path` runs it within a second
262+
of any change to those directories (daily timer as a backstop) and reloads Caddy only
263+
when a bundle changed. A new `<name>.pit.world` therefore still costs one Let's Encrypt
264+
certificate, issued by `pit-proxy-cert` on the name's first plain-http visit, as before.
265+
266+
**`listen 443` is now a bug, and is fixed for you.** With the stream on :443 an http
267+
server's `listen 443` passes `nginx -t`, but the reload cannot bind, so nginx keeps the
268+
old configuration without saying so (a restart fails outright). OpenInstall's
269+
`bin/install.sh`, moshpit-proxy's `setup-origin.sh` (every Moshpit renewal) and
270+
`certbot --nginx` still write it. `nginx-edge-listen` rewrites such lines to
271+
`listen 127.0.0.1:8444 ssl proxy_protocol;` (and drops `[::]:443`): it runs before every
272+
nginx start and reload (`nginx.service.d/dev2-edge.conf`) and whenever a vhost file
273+
changes (`nginx-edge-listen.path`). `dev2-site vhost` renders the 8444 form itself when
274+
the edge is on. `nginx-edge-listen --check` lists anything left.
275+
276+
**Cloudflare.** One `real_ip_header` per level: `conf.d/cloudflare-realip.conf`'s
277+
`CF-Connecting-IP` line is commented out. No dev2 name was proxied by Cloudflare on
278+
2026-10-19; one that is would log Cloudflare's edge address again.

0 commit comments

Comments
 (0)