Keep the proxy up without HaRP and move the shared key into a file - #9
Conversation
A literal proxy_pass hostname stops nginx from starting whenever the appapi-harp container is absent, which takes every instance down; resolve it per request instead. The shared key now lives in an untracked data/harp.key mounted as HP_SHARED_KEY_FILE, with the rotation order that actually works. verify.py gains a proxy check and reads the key file. Signed-off-by: Oleksander Piskun <oleksandr2088@icloud.com>
|
will rebase this PR after we merge this one: #6 |
…n the triage script A live rotation showed that docker compose up -d does not pick up a changed key file and that restart crash-loops HaRP (its generated FRP configs keep the old token); only a recreate works. The key one-liner needs LC_ALL=C on macOS. harp-triage.sh now reads HP_SHARED_KEY_FILE, and the harness accepts any proxy_pass that carries a variable. Signed-off-by: Oleksander Piskun <oleksandr2088@icloud.com>
kyteinsky
left a comment
kyteinsky
left a comment
There was a problem hiding this comment.
nice!
yeah this is one nasty issue
sorry I should've documented that in the harp readme instead of implicitly using the IP. Would you mind adding a note there too, or I can add it later?
| - The HaRP shared key must be byte-identical between the compose override (`HP_SHARED_KEY`) and the | ||
| `daemon:register --harp_shared_key` value; a mismatch is the number one install failure. | ||
| files (`.env`, `docker-compose.override.yml`, `data/nginx/vhost.d/`, `data/harp.key`). | ||
| - The HaRP shared key lives in one untracked file, `data/harp.key`: HaRP mounts it as `HP_SHARED_KEY_FILE` and |
There was a problem hiding this comment.
maybe better to put this in data/ssl/ so it's also ignored by .gitignore
There was a problem hiding this comment.
data/ssl/ is mounted into the proxy container as its certs dir (docker-compose.yml line 12), so the key would be readable from inside nginx. I would rather keep it out of any mounted dir and pay the one exclude line.
| and the ExApps cannot be removed the normal way because that goes through HaRP. The order that works: | ||
|
|
||
| ```bash | ||
| ./scripts/occ.sh nextcloud -- app_api:app:unregister <appid> --force --silent # per ExApp; drops the row without asking HaRP |
There was a problem hiding this comment.
the ex-app container might still be there if this unregister does not go through with harp
manual removal by container name nc_app_<appid> could be done
or the exapps should be removed before the key is rotated.
There was a problem hiding this comment.
Good point, unregistering the ExApps before the key changes is the cleaner order. Rewrote the section that way; the --force path is now the recovery case and names docker rm -f nc_app_<appid>.
|
|
||
| ```bash | ||
| (umask 077; LC_ALL=C tr -dc A-Za-z0-9 </dev/urandom | head -c 32 > data/harp.key) # any ASCII string works; LC_ALL=C keeps BSD tr (macOS) happy | ||
| echo data/harp.key >> .git/info/exclude |
There was a problem hiding this comment.
ah it's ignored here, might be still better to use already ignored dir data/ssl/
There was a problem hiding this comment.
Same reason as above, data/ssl/ is the proxy's certs mount.
Unregistering them while the old key still works lets HaRP remove the containers; the forced path becomes the recovery case and names the container to remove by hand. Signed-off-by: Oleksander Piskun <oleksandr2088@icloud.com>
|
Sure, note for the HaRP README is here: nextcloud/HaRP#121 |
2 participants
Two things found while re-running the dev-setup runbook end to end (NC 36 dev, AppAPI 36.0.0-dev.0, HaRP 0.4.5, docker-dev d0d1016).
Proxy snippet. With a literal
proxy_pass http://appapi-harp:8780/exapps/, nginx resolves the name at startup and refuses to start when the container is absent:[emerg] host not found in upstream "appapi-harp". Every URL of every instance then refuses the connection,status.phpincluded, while Nextcloud's logs look fine. Any proxy restart while HaRP is stopped triggers it (host reboot,up -dafter adown).set $harp_upstream ...; proxy_pass $harp_upstream;resolves per request through theresolver 127.0.0.11nginx-proxy already generates; HaRP being down then degrades to a 502 on/exapps/only, and recovery needs no proxy restart. Tested both ways.Shared key. Moved from
HP_SHARED_KEYin the override to an untrackeddata/harp.keymounted asHP_SHARED_KEY_FILE; Stage 6 reads the same file, so the two cannot drift, anddocker compose configno longer prints it. The "unregister first to change values" note did not work for a key change:daemon:unregisterrefuses while the daemon holds ExApps and the ExApps cannot be removed through HaRP any more. Documented the order that does work.tests/verify.py: newproxy_survives_harp_absencecheck (nginx -tplus the snippet must use a variable), andharp_inforeads the key file instead of skipping. All nextcloud-dev-setup and harp-operations checks pass here.