Start the stone and session listener once, across processes

[MatiasFernandez]

Closes #112.

Problem

Three callers started NetLDI without a lock: the window (startProcesses), the GemDB Shell (cliMain.ts), and runStop's restart after a declined force-stop. When two of them raced, the loser's startnetldi failed. The window then reported "Start session listener failed" and stopped bring-up, although both processes were up (closed PR #89). Smaller problems fed into it:

• A stone at Startup counted as running.
• A caller that had only waited still sent databaseStarted.
• The Shell started the stone without the window's pre-start guards.

What changed

• ensureProcesses() (processes.ts) lists and starts the stone and NetLDI inside withStoneLock. It returns the flags for what this call started, so a window that only waited sends no databaseStarted. The window and the Shell use it. startNetldi is no longer exported, and startStone() is gone: the integration tests use a fixture helper instead.

• Already up, no lock. One listing before the lock decides: when the stone is ready (or in a status we let through, below) and NetLDI is listed, ensureProcesses returns without taking the lock. Inside the lock it always lists again. The already-running path makes one gslist call. A start that only needs the listener makes two, because the in-lock listing doubles as the ready-wait's first reading. A cold start makes three.

• Declined force-stop puts back only the listener. runStop calls ensureListener(), which under the lock starts NetLDI only if the stone is listed and not shutting down, and never starts the stone. runStop also re-checks the stone after the user declines, and logs "GemDB stopped." if it went away meanwhile.

• NetLDI race tolerance. If startnetldi fails but gemdbldi is listed with status OK within 5 s, that counts as "another process started it". The 5 s matter for an overlap (exit 3, "could not start server" over the winner's lock file), where the winner's listener may not be answering yet when ours gives up. A start that really fails reports startnetldi's own error, up to 5 s later. This is still needed because 1.5.x and 1.6.0 start NetLDI with no lock.

• "Listed" and "ready" are separate questions, with a status model. "Listed" decides whether to start a stone. The status decides what to do next, compared without case (the engine prints Startup, Shutdown, OK, frozen, exe deleted, …):

  • OK: ready.
  • startup, recovery: wait, up to a minute, then an error naming the stone log. Never started again.
  • shutdown: fail at once with "The database is shutting down…".
  • anything else listed (exe deleted, frozen, full, …): go ahead and let the login judge it, as before.

  findStone/isRunning are unchanged, so the status bar and maintenance still ask "is there a process".

• Waiting on the stone lock. The holder now keeps the lock through the guards, startstone, the ready-wait and startnetldi. A waiter still gives up after a minute when the holder shows nothing; past that, a stone listed and not shutting down counts as progress and keeps it waiting for 30 s after it was last seen, up to 2.5 minutes. When the holder is this same window, the message says so instead of "another process", and still names the lock file to remove. A crashed process whose pid we happened to get again looks the same.

• The gemdb wrapper gets the same model: one gslist -cvl per check (stone_status), branching on the lowercased status. Its waits are timed with date +%s instead of counted polls; a frozen stone makes every gslist -cvl take about 5 s, which stretched the old "minute" to most of an hour. The wrapper's fingerprint changes, so existing bin/gemdb files are rewritten once.

• External databases in the wrapper get no lock and no start block: not listed → the administrator error, shutdown → error, starting → bounded wait, anything else → go ahead. startstone is unreachable for them.

• The pre-start guards move to assertSafeToStart() (NFS, engine version, space limits, room for the extent). They run inside the lock, and the Shell now gets them too.

Commit 1 (the tolerance) stands alone and can be cherry-picked.

Worth a reviewer's attention

• External database. ensureProcesses refuses external databases, so the window and the Shell both check one before calling it, sharing externalDatabaseDownError. Without that, every Shell start against an external database would throw. Centralising the check is noted on gemdb refuses to start a database without Python support #115, which edits the same function.
• Wrapper re-check inside the lock. It decides on "listed", not "OK". With "OK" there, a stone still at Startup would get a second startstone beside it: the two-stones-on-one-extent case the lock exists for. A gslist row of unexpected shape also counts as listed, so it can never cause a start.
• The stopping tail. A stopping stone reports OK → Shutdown → Startup (20–40 ms) → gone. Shutdown seen inside the lock fails at once; a first look that lands in the Startup tail still reports "stopped while it was starting". Coordinating Stop with start is Stop takes no lock, so a start in another window fails #125.

Testing

• Unit: startProcesses.test.ts (new), plus updates to databaseStarted, externalLifecycle, externalDatabase, gslist, cli and stop. They cover:
  • every status class, in ensureProcesses and in the generated wrapper (exe deleted/frozen/full go ahead with one gslist call, Shutdown fails at once, Startup/recovery wait);
  • the already-up path taking no lock;
  • a lock holder that takes 70 s (startstone 20 s, then 50 s at Startup): the second waiter no longer fails at 60 s;
  • a declined force-stop after the stone went away starts nothing;
  • the external wrapper containing no startstone and no lock.
• Integration: src/__integration__/processes.test.ts (new) races real node processes against a real engine:
  • two ensureProcesses calls: one started the stone, one started the listener, and pgrep counts exactly one netldid (two simultaneous startnetldi can both succeed and leave an orphan that gslist doesn't show, Two simultaneous startnetldi leave an orphaned NetLDI #127);
  • a raw, unlocked startnetldi (as 1.6.0 does) against ensureProcesses.
• Full integration suite passed locally on macOS arm64 (114/114) before the review fixes; after them, processes, database and cli pass again on a throwaway root.
• After the second review round (ea64653: the 5 s listener retry, the reused listing, the lock message): unit tests only, 945/945. The overlap test was checked against the code before the fix and fails there. Integration not re-run yet.
• Manual, macOS arm64, before the review fixes: GemDB Shell opened while a window is starting the database. A hand-held stone lock (live pid) made the race deterministic: Start in the window and bin/gemdb both waited on it, and on release they raced for it. Result: one gemdb stone and one gemdbldi, both OK, the Shell reached its prompt, and the window's Start finished without "Start session listener failed".
• Manual, macOS arm64, before the review fixes: Start while GemDB 1.6.0 in another editor is starting the database. Both editors queued on a hand-held stone lock. 1.6.0 won, started the stone, then started gemdbldi unlocked. This window waited and found both OK, started nothing and reported no error. The NetLDI tolerance branch was not reached by hand; the unit and integration tests cover it.
• Manual, macOS arm64, before the review fixes: kill -9 the stone, then gemdb -c .... The wrapper started the stone, waited through crash recovery at Startup, then ran the code and exited 0. Three gemdb -c run in parallel after a kill -9: one started the stone, the others waited, all three ran, and pgrep showed a single stoned gemdb.

Known gaps, planned for later

A second review of this PR checked each finding against main. Those that are not regressions are tracked in #134, so they are deliberately not fixed here:

• Follow-ups from the review of #118 (concurrent database starts) #134.1: gemdb file.py starts the stone without the pre-start guards. After an engine pin move it gets a raw 4045 instead of the "delete this directory" message. This was already the case on main; this PR guards the Shell only. It touches the same lines as gemdb refuses to start a database without Python support #115.
• Follow-ups from the review of #118 (concurrent database starts) #134.2: the stone start logic lives in both TypeScript and the generated bash, and the copies already differ. The plan is to have file mode run ensureProcesses through the staged Node bundle, which also closes Follow-ups from the review of #118 (concurrent database starts) #134.1 and Follow-ups from the review of #118 (concurrent database starts) #134.3.
• Follow-ups from the review of #118 (concurrent database starts) #134.3: external database at Startup. The wrapper waits, the window and Shell don't (unchanged from main). The external-is-down check is also duplicated between lifecycle.ts and cliMain.ts.
• Follow-ups from the review of #118 (concurrent database starts) #134.4: the overlap integration test doesn't assert that an overlap happened.
• Follow-ups from the review of #118 (concurrent database starts) #134.5: a stone lock carrying our own pid is never considered stale. This was already on main; the message now at least names the lock.
• Follow-ups from the review of #118 (concurrent database starts) #134.6: a stone that vanishes during the wait is reported as "stopped while it was starting", even when this call didn't start it. Same outcome as main, a different message. See also Stop takes no lock, so a start in another window fails #125.
• Nits, also in Follow-ups from the review of #118 (concurrent database starts) #134: a killed row in the wrapper, the log line after a declined force-stop at Shutdown, $STATUS reuse in the wrapper, the duplicate ### Fixed headings, and the overlap between ensureListener and ensureProcesses.

Also planned:

• The wrapper gives up on a live stone lock after 10 s (now real seconds), while the extension waits on a holder that makes progress. Agreeing on one policy needs a heartbeat: The wrapper gives up on a live stone lock after 10 s #126, after One lock mechanism for TS and bash, with a nonce and a heartbeat #124.
• Stop takes no lock, so a start in another window can fail while the database stops: Stop takes no lock, so a start in another window fails #125.
• Two simultaneous startnetldi can leave an orphaned NetLDI. The lock prevents it between these processes, not against 1.5.x/1.6.0: Two simultaneous startnetldi leave an orphaned NetLDI #127.
• The Shell does not write the statmonitor config, because that needs editor settings. system.conf keeps whatever the window's last start wrote.
• No telemetry for "waited for another process". That belongs with the planned waitCancelled work.
• The integration race is not deterministic. The 1.6.0 test lines the two startnetldi calls up, but which one wins varies. The tolerance branch ran in about 1 of 8 local runs; the unit tests cover it every time.

🤖 Generated with Claude Code