From 2c59e88473d2ab950129e02ece53a6126f9d37ed Mon Sep 17 00:00:00 2001 From: Jackson Atassi Date: Wed, 10 Jun 2026 22:01:46 -0600 Subject: [PATCH] docs(runbooks): corrections from the 2026-06-10 update-loop run MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - update-loop §1.7: tail the real signal sources (.err.log / monitor.log / heartbeat age) — the .out.log files are empty by design; note that §1.4 shutdown residue (KeyboardInterrupt / cancelled-connect TimeoutError) is expected and only post-start tracebacks matter. - update-loop §1.1: scan pulled commits touching scripts/services/ for deploy notes — installer edits don't reconfigure installed services. - troubleshooting §8.12: new gotcha for live SCM/NSSM config drifting from the installers (observed deploying ALP-945's sc.exe config step). Co-Authored-By: Claude Fable 5 --- docs/runbooks/troubleshooting.md | 15 +++++++++++++++ docs/runbooks/update-loop.md | 29 +++++++++++++++++++++++------ 2 files changed, 38 insertions(+), 6 deletions(-) diff --git a/docs/runbooks/troubleshooting.md b/docs/runbooks/troubleshooting.md index 71e9c928..ac66dbbe 100644 --- a/docs/runbooks/troubleshooting.md +++ b/docs/runbooks/troubleshooting.md @@ -264,3 +264,18 @@ monitor's auto re-protection job (ALP-938, § 8.10). Only investigate if the sam symbol rejects across consecutive invocations — that means fill integration is not catching the projection up to broker reality. +### 8.12 Pulled an installer-script change, but the live service still has the old config + +`scripts/services/install_*_service.ps1` changes only affect *future* installs — +the update loop never re-runs installers, so an already-registered service keeps +whatever SCM/NSSM config it was installed with. Observed 2026-06-10: PR #350 +deleted the command center's `DependOnService` edges from its installer, but +`sc.exe qc AlphaMindCommandCenter` still showed both edges until the one-time +`sc.exe config AlphaMindCommandCenter depend= /` from the commit's deploy note +was applied (with services stopped). + +When a pulled commit touches `scripts/services/`, read its commit message for a +deploy note and apply the live-config step it names; verify with `sc.exe qc +` / `nssm get ` afterward (update-loop.md +§ 1.1 carries the scan step). + diff --git a/docs/runbooks/update-loop.md b/docs/runbooks/update-loop.md index df2a2e37..6e9b4137 100644 --- a/docs/runbooks/update-loop.md +++ b/docs/runbooks/update-loop.md @@ -30,6 +30,12 @@ git pull --ff-only origin main If `git status` shows you on a feature branch or with uncommitted changes, stop — investigate before pulling. Prod runs `main`, full stop. +Scan what came in: `git log ..HEAD --oneline`, then read the full +message of any commit touching `scripts/services/` — installer-script changes +do **not** reconfigure already-installed services, so such commits carry a +deploy note naming a one-time live-config step (`sc.exe config ...` / +`nssm set ...`) to run while services are stopped (troubleshooting.md § 8.12). + ### 1.2 Sync dependencies ```powershell @@ -121,13 +127,24 @@ that, tail its `.err.log` immediately. ### 1.7 Verify green +The `.out.log` files are empty by design — every daemon logs to stderr — so the +startup signal lives in the `.err.log` files and the monitor's dedicated +`monitor.log`. Shutdown residue from § 1.4 is expected at the tails: a +`KeyboardInterrupt` traceback in the collector/safety-core/watchdog `.err.log`s, +and a cancelled websocket-connect `TimeoutError` in `monitor.err.log`. A +traceback only matters if it is timestamped **after** the § 1.6 starts. + ```powershell -# Tail the recent end of each daemon's log; look for "startup complete" / -# "supervisor started" / equivalent. No traceback should appear in the last -# 60 s of any .err.log. -Get-Content "$env:USERPROFILE\AlphaMind\logs\pipeline.out.log" -Tail 30 -Get-Content "$env:USERPROFILE\AlphaMind\logs\monitor.out.log" -Tail 30 -Get-Content "$env:USERPROFILE\AlphaMind\logs\command_center.out.log" -Tail 30 +# Scheduler: expect "Application startup complete" + a next-fire line per trigger. +Get-Content "$env:USERPROFILE\AlphaMind\logs\pipeline.err.log" -Tail 15 +# Monitor: expect a fresh "monitor session start" + control surface on 8766, +# and a heartbeat younger than the watchdog's 150 s stall bound. +Get-Content "$env:USERPROFILE\AlphaMind\logs\monitor.log" -Tail 10 +$hb = [double](Get-Content "$env:USERPROFILE\AlphaMind\logs\monitor.heartbeat") +"heartbeat age: $([math]::Round([DateTimeOffset]::UtcNow.ToUnixTimeMilliseconds()/1000 - $hb, 1)) s" +# Command center: expect uvicorn on 8090 + 200 OK GETs to both upstream /events +# (8765 pipeline, 8766 monitor). The dormant alert-rule WARNINGs are known noise. +Get-Content "$env:USERPROFILE\AlphaMind\logs\command_center.err.log" -Tail 15 ``` Optional gate (runs in ~1 min, exercises 7 command-center checks):