Skip to content

Persist admin sessions so a restart does not force another login - #34

Merged
maiphucgiang merged 6 commits into
maiphucgiang:mainfrom
moshouhot:fix/persist-admin-sessions
Sep 20, 2026
Merged

maiphucgiang merged 6 commits into
maiphucgiang:mainfrom
moshouhot:fix/persist-admin-sessions

Conversation

@moshouhot

@moshouhot moshouhot commented Sep 19, 2026

Copy link
Copy Markdown
Contributor

Summary

The management console keeps its session table in memory only, so restarting the
gateway invalidates the cb_admin_session cookie and the WebUI asks for the API key
again. This is easy to hit during an update or a plain restart.

This persists the session table to admin-sessions.json under the managed auth
directory (the same directory as control.sqlite3 / logs.sqlite3, which is already
git-ignored) and restores it on startup, so a restart no longer forces another login.

Why expiry moves off the monotonic clock

time.monotonic restarts with the machine and cannot express an absolute deadline
across restarts, so a persisted monotonic timestamp would be meaningless. Session
expiry now uses the wall clock. Login throttling still uses time.monotonic, which
is the appropriate clock for a sliding window and is unaffected by wall-clock jumps.

Revocation is unchanged

Persisting a login must not weaken the existing guarantees. The design keeps
revocation as the single mechanism — clearing the stored snapshot:

  • The file carries a keyed fingerprint naming the key epoch it belongs to.
  • A snapshot whose fingerprint does not match the current key is revoked, not
    adopted. That covers key rotation, a disabled key, and a key that is set back to a
    previous value, across restarts and in-process.
  • Logout and expiry clear the entry, and the file is rewritten.
  • A snapshot that fails validation is removed rather than ignored, so it cannot
    be adopted by a later start.

The fingerprint is an epoch label, not an integrity MAC — revocation is performed by
clearing the file, and the header identity remains ephemeral as before.

Hardening of the restored snapshot

The snapshot is untrusted input read from disk, so it is:

  • read with a bounded read, via O_NOFOLLOW with a regular-file check, so a symlink
    is not followed and an oversized file is not loaded;
  • parsed as strict JSON (duplicate keys rejected, Infinity / NaN rejected);
  • validated field by field: exact top-level and per-entry key sets, integer version,
    bounded url-safe tokens, and a finite in-range deadline with bool rejected;
  • size bounded in both entry count and bytes.

Writes go through a mode-0600 temporary file and an atomic os.replace. If the write
cannot complete, the previous snapshot is revoked as the fallback, so a stale
on-disk table cannot outlive a revocation. Note that chmod does not establish
owner-only ACLs on Windows; the mode is best effort there.

When no session_path is configured the previous in-memory behaviour is unchanged.

Tests

tests/test_admin_boundaries.py gains a PersistedSessionTests class covering:
survival across a restart, the restored CSRF token and deadline, expiry under an
injected wall clock, key rotation, a disabled key both at startup and in-process,
logout, a failing write, symlinks, and a table of untrusted snapshots
(Infinity / NaN expiry, duplicate and extra fields, boolean version, non-ASCII
fingerprint, oversized sid / csrf / file).

python -m pytest tests/test_admin_boundaries.py tests/test_admin_api.py → 62 passed.
Full suite on Windows shows the same 20 pre-existing failures before and after this
change (symlink / fsync / Docker / permission tests); verified by diffing the failure
list against pristine main. No new failures.

Summary by Sourcery

Persist admin sessions across restarts while preserving durable revocation and failing safely when session storage is unavailable.

New Features:

  • Persist administrator sessions to disk and restore them across gateway restarts without requiring another API-key login.

Bug Fixes:

  • Ensure session expiry remains correct across restarts and that key rotation, disabled keys, logout, expiry, and invalid snapshots revoke sessions durably.
  • Fail closed when a superseded session snapshot cannot be revoked, preventing stale sessions from being resurrected.

Enhancements:

  • Harden persisted session handling with bounded, strict validation and atomic owner-protected snapshot updates.
  • Expose session-storage health in the administration API and surface failed revocations in the WebUI.

Tests:

  • Add coverage for restart persistence, expiry, revocation, storage failures, symlink handling, malformed snapshots, bounds, and UI storage-status behavior.

The management console kept its session table in memory only, so every
restart of the gateway invalidated the cb_admin_session cookie and the
WebUI asked for the API key again. This is easy to hit when updating or
restarting the service.

Sessions are now written to admin-sessions.json under the managed auth
directory and restored on startup.

Persisting a login must not weaken the existing revocation guarantees.
Revocation works by clearing the stored snapshot; the file additionally
carries a keyed fingerprint naming the key epoch it belongs to, and a
snapshot whose fingerprint does not match the current key is revoked
rather than adopted. That covers key rotation, a disabled key, logout and
expiry, in-process and across restarts.

Session expiry moves from time.monotonic to a wall clock, because a
monotonic clock restarts with the machine and cannot express an absolute
deadline across restarts. Login throttling still uses the monotonic clock,
which is the appropriate choice for a sliding window.

The snapshot is size bounded, read with a bounded read and a regular-file
check without following symlinks, parsed as strict JSON, and validated
field by field; a snapshot that fails validation is removed so it cannot
be adopted later. Writes go through a mode-0600 temporary file and an
atomic replace, with revocation of the previous snapshot as the fallback
when the write cannot complete. Note that chmod does not establish
owner-only ACLs on Windows.

When no path is configured the previous in-memory behaviour is unchanged.
@sourcery-ai

sourcery-ai Bot commented Sep 19, 2026

Copy link
Copy Markdown

Reviewer's Guide

Admin sessions are now optionally persisted to a hardened, atomically written JSON snapshot and restored across restarts only when tied to the current API-key epoch; wall-clock expiry enables restart safety, while revocation and in-memory fallback behavior remain intact.

Sequence diagram for persisted admin session restoration

sequenceDiagram
    participant Gateway
    participant AdminAuth
    participant Snapshot as admin-sessions.json
    participant WebUI

    Gateway->>AdminAuth: _key()
    AdminAuth->>Snapshot: _restore(key)
    Snapshot-->>AdminAuth: validated sessions for matching fingerprint
    AdminAuth->>AdminAuth: session(request)
    AdminAuth-->>WebUI: restored session and CSRF token
Loading

Sequence diagram for admin session persistence and revocation

sequenceDiagram
    actor Admin
    participant WebUI
    participant AdminAuth
    participant Snapshot as admin-sessions.json

    Admin->>WebUI: Login with API key
    WebUI->>AdminAuth: login(request, key)
    AdminAuth->>Snapshot: _persist()
    Snapshot-->>AdminAuth: atomic replacement
    AdminAuth-->>WebUI: session cookie and CSRF token
    Admin->>WebUI: Logout
    WebUI->>AdminAuth: logout(request)
    AdminAuth->>Snapshot: _persist()
    AdminAuth-->>WebUI: session revoked

    AdminAuth->>AdminAuth: _key()
    AdminAuth->>Snapshot: _revoke()
    Snapshot-->>AdminAuth: snapshot cleared after key change
Loading

File-Level Changes

Change Details Files
Persist admin session state across gateway restarts while preserving key-epoch revocation semantics.
  • Resolve an optional session file and configure runtime management to use admin-sessions.json in the managed auth directory.
  • Persist versioned session records with a keyed fingerprint, wall-clock deadlines, CSRF tokens, and bounded session count.
  • Restore only snapshots matching the active API-key epoch; clear snapshots on key changes, disabled keys, logout, and expiry.
  • Retain in-memory-only behavior when no session path is configured.
app/admin_auth.py
app/runtime_management.py
Harden persisted-session loading and writing against untrusted or unreliable filesystem state.
  • Read bounded regular files without following symlinks and reject malformed, duplicate-key, non-finite, oversized, or structurally invalid JSON.
  • Validate versions, fingerprints, tokens, deadlines, entry counts, and exact object fields.
  • Write via a 0600 temporary file followed by atomic replacement, and revoke the prior snapshot if persistence fails.
app/admin_auth.py
Switch session expiry to a restart-safe clock without changing throttling behavior.
  • Use wall-clock time for session deadlines and injected expiry tests.
  • Keep monotonic time for login rate limiting.
app/admin_auth.py
tests/test_admin_boundaries.py
Add boundary coverage for persistence, revocation, and snapshot hardening.
  • Test restart survival, CSRF/deadline restoration, expiry, rotation, disabled keys, logout, write failures, symlinks, missing paths, capacity limits, and file permissions.
  • Exercise malformed and adversarial snapshots including duplicate fields, non-finite values, invalid types, oversized tokens/files, and extra fields.
tests/test_admin_boundaries.py

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 4 issues

Prompt for AI Agents
Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="app/admin_auth.py" line_range="146" />
<code_context>
+    @staticmethod
+    def _deadline(value):
+        """Accept only finite, in-range numeric deadlines; bool is not a deadline."""
+        if type(value) not in (int, float):
+            return None
+        number = float(value)
+        return number if math.isfinite(number) and 0 < number < 1e11 else None
+
</code_context>
<issue_to_address>
**issue (bug_risk):** A persisted JSON expiry containing an integer too large for `float()` raises `OverflowError` during `_restore`, and that exception is not caught, so `AdminAuth.enabled()` crashes instead of revoking the untrusted snapshot.

**Triggers:** When an attacker or corrupted file supplies a very large integer for `expires`.

**Suggested fix:** Catch `OverflowError` around the numeric conversion or reject integer values by checking their range before converting to float.

```suggestion
        try:
            number = float(value)
        except OverflowError:
            return None
```
</issue_to_address>

### Comment 2
<location path="app/admin_auth.py" line_range="187-188" />
<code_context>
+        if len(raw) > _SESSION_FILE_BYTES:
+            return False
+        try:
+            document = _strict_json(raw)
+        except (ValueError, UnicodeDecodeError):
+            return False
</code_context>
<issue_to_address>
**issue (bug_risk):** Deeply nested but size-bounded JSON can make `json.loads` raise `RecursionError`, which `_restore` does not catch, so a malformed snapshot can crash startup rather than being removed.

**Triggers:** When `admin-sessions.json` contains sufficiently deeply nested JSON within the 256 KiB size limit.

**Suggested fix:** Catch `RecursionError` when parsing the snapshot, or impose a structural nesting limit before/while parsing.

```suggestion
            document = _strict_json(raw)
        except (ValueError, UnicodeDecodeError, RecursionError):
```
</issue_to_address>

### Comment 3
<location path="app/admin_auth.py" line_range="170-171" />
<code_context>
+        if self._path is None:
+            return True
+        try:
+            fd = os.open(self._path, os.O_RDONLY | getattr(os, "O_NOFOLLOW", 0) | getattr(os, "O_NONBLOCK", 0))
+        except FileNotFoundError:
+            return True
+        except OSError:
</code_context>
<issue_to_address>
**🚨 issue (security):** On platforms without `os.O_NOFOLLOW` such as Windows, `os.open` follows a symlink, so `_restore` can adopt session data from the symlink target despite the regular-file check and the stated no-symlink guarantee.

**Triggers:** When `session_path` is a symlink and the platform does not provide `O_NOFOLLOW`.

**Suggested fix:** Reject symlinks using a platform-supported no-follow/open-reparse-point mechanism, or explicitly compare the opened file's identity and path before accepting it.
</issue_to_address>

### Comment 4
<location path="app/admin_auth.py" line_range="214-215" />
<code_context>
+            expires = self._deadline(item["expires"])
+            if token is None or csrf_token is None or expires is None:
+                return False
+            if expires > now:
+                restored[token] = {"csrf_token": csrf_token, "expires": expires}
+        self.sessions.update(restored)
+        return True
+
</code_context>
<issue_to_address>
**issue (broader_impact):** Expired entries are filtered out of `self.sessions` but the snapshot is not rewritten, so the expired records remain on disk and can be restored later if the wall clock moves backward before a subsequent restart.

**Triggers:** When startup occurs after expiry and the wall clock subsequently moves earlier before the next process restart.

**Suggested fix:** Remove expired entries from the persisted document and call `_persist()` after restoration, or otherwise revoke and rewrite the snapshot whenever expired records are discarded.

```suggestion
        self.sessions.update(restored)
        if len(restored) != len(entries):
            self._persist()
        return True
```
</issue_to_address>

Sourcery assessment

Needs a human reviewer. 4 findings to address first, and if persistence or revocation is wrong, a previously issued admin cookie could retain administrative access across a restart or a key rotation, or an incorrectly trusted snapshot could grant access. Reverting prevents future restoration, but any sessions already restored remain usable until they expire or are explicitly revoked.

Blocking findings: app/admin_auth.py:146, app/admin_auth.py:188, app/admin_auth.py:171, app/admin_auth.py:215


Sourcery is free for open source - if you like our reviews please consider sharing them ✨

Comment thread app/admin_auth.py Outdated
Comment thread app/admin_auth.py Outdated
Comment thread app/admin_auth.py
Comment thread app/admin_auth.py
Four defects found in review of the persisted session snapshot:

- `_deadline` called `float()` on a JSON integer without guarding the
  conversion, so a value beyond the float range raised `OverflowError` out of
  `enabled()`. The exception escaped `_restore`, which also skipped the
  revoke-on-invalid path, leaving the file in place for the next start.
- `_strict_json` can raise `RecursionError` on deeply nested input well inside
  the 256 KiB size bound, with the same effect.
- `os.O_NOFOLLOW` does not exist on Windows, where `getattr(..., 0)` silently
  degraded the no-symlink guarantee to nothing and the link target was adopted.
- Expired records were filtered out of memory but left on disk, so a wall clock
  that moved backward before the next restart could revive them.

`_deadline` now rejects non-convertible numbers, `_restore` treats
`RecursionError` as malformed input, the opened file is compared against the
path itself so a link is rejected where `O_NOFOLLOW` is unavailable, and a
snapshot that dropped expired records is rewritten.

Tests: regression cases for each, including a symlink pointing at a valid
snapshot so following it would be observable. `tests/test_admin_boundaries.py`
and `tests/test_admin_api.py` pass (63); the full suite shows the same failures
as before the change.
@maiphucgiang

Copy link
Copy Markdown
Owner

@codex review

@maiphucgiang maiphucgiang left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please address the two session-revocation regressions below before merging, and restore the overwritten symlink test. I reviewed a4da831 and ran the focused admin tests (63 passed). Additional isolated reproductions against this head and main@426e8a5 confirmed that both revocation scenarios return HTTP 200 for an old cookie on this branch, versus HTTP 401 on main. Only synthetic credentials and temporary storage were used; no upstream services were contacted.

Comment thread app/admin_auth.py Outdated
Comment on lines +289 to +293
if restoring:
# Adopt only a snapshot that matches the current epoch; anything else is
# revoked, so switching back to an old key cannot resurrect its sessions.
if not self._restore(key):
self._revoke()

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Revoke obsolete snapshots at startup, not on the first admin request

_key() is lazy: AdminAuth.__init__() and install_admin() do not call it, and inference requests bypass AdminMiddleware. Consequently, log in with key A, restart with key B (or an empty key), make only inference requests, and restart again with A before the cookie expires: the original cookie is restored and accesses a protected admin endpoint with HTTP 200. I reproduced both intermediate-key variants using the real admin and inference middleware; main returns 401. The intermediate process never clears A's snapshot, so the deterministic fingerprint cannot distinguish the reused key from the old epoch.

Please reconcile/revoke the persisted snapshot during startup after the effective configuration is resolved, including when management is disabled. Add restart tests for A → B → A and A → empty → A that deliberately make no admin request during the intermediate run.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 79386ba. install_admin now calls AdminAuth.reconcile(), which resolves the key epoch during startup instead of on the first /admin request.

You were right about the mechanism: _key() is lazy, AdminAuth.__init__() and install_admin() never called it, and AdminMiddleware only covers /admin/, so an inference-only process never touched the file and the next start adopted it. reconcile() is called before the middleware is installed, so it also runs when management is disabled and the key is empty.

Verified against your scenario through the real ASGI app: A → B (inference only) → A now returns 401 from /admin/settings for the old cookie, where it previously returned 200. Both restart tests you asked for are in test_startup_reconciles_without_any_admin_request (A → B → A and A → empty → A, each asserting no admin request is made during the intermediate run), plus test_startup_reconcile_happens_for_the_installed_admin to keep the wiring from regressing.

Your point about the deterministic fingerprint is exactly why this could not be fixed by inspecting the file: with A and B both valid epochs, nothing on disk distinguishes "reused key" from "same epoch". The fix has to be that the intermediate process actually runs its reconciliation.

@maiphucgiang maiphucgiang Sep 20, 2026

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 79386ba. install_admin now calls AdminAuth.reconcile(), which resolves the key epoch during startup instead of on the first /admin request.

You were right about the mechanism: _key() is lazy, AdminAuth.__init__() and install_admin() never called it, and AdminMiddleware only covers /admin/, so an inference-only process never touched the file and the next start adopted it. reconcile() is called before the middleware is installed, so it also runs when management is disabled and the key is empty.

Verified against your scenario through the real ASGI app: A → B (inference only) → A now returns 401 from /admin/settings for the old cookie, where it previously returned 200. Both restart tests you asked for are in test_startup_reconciles_without_any_admin_request (A → B → A and A → empty → A, each asserting no admin request is made during the intermediate run), plus test_startup_reconcile_happens_for_the_installed_admin to keep the wiring from regressing.

Your point about the deterministic fingerprint is exactly why this could not be fixed by inspecting the file: with A and B both valid epochs, nothing on disk distinguishes "reused key" from "same epoch". The fix has to be that the intermediate process actually runs its reconciliation.

Thanks — I verified the writable-storage rotation cases, failed-logout retry, and duplicate-test fix on 79386ba; those now pass. One P1 failure path in this thread remains.

[P1] Startup still succeeds when revoking the previous epoch fails

reconcile() at lines 331–340 calls _key(), but _key() still discards the failed _revoke() result at line 317. install_admin() therefore completes successfully even though the previous epoch's snapshot survives.

I reproduced this with a real read-only bind mount, not mocked I/O errors:

  1. Start with a valid persisted session for key A, then make only admin-sessions.json read-only. The containing directory and sibling SQLite database remain writable.
  2. Install the real admin/inference middleware with key B (also tested with an empty key), and make only inference requests. Startup succeeds and inference returns 200; session storage merely reports degraded: true.
  3. Start again with A and replay the old cookie against a protected admin endpoint: 200, versus 401 on the non-persistent baseline.

So the original issue is fixed when revocation succeeds, but a storage failure still permits resurrection. A settings-page warning cannot prevent the next process from adopting the surviving file, and that page is unavailable when the intermediate key is empty.

Please propagate durable-revocation failure through _key()reconcile()install_admin(). If an obsolete snapshot cannot be removed or replaced, startup must explicitly fail rather than activate the new key epoch, unless an independent durable revocation mechanism guarantees the old snapshot cannot return. Add failure-path tests for A → B → A and A → empty → A with an undeletable snapshot and writable SQLite, while retaining the successful-reconciliation tests.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in e0bd154. You are right, and thank you for the follow-up: my 79386ba only recorded the failure as degraded state, so the rotation still completed and the snapshot still survived — the file was left for a later start to adopt. Reproduced exactly as you describe, with a read-only admin-sessions.json and the directory and sibling SQLite stores left writable: install_admin for the new key returned normally, and a later start under the old key accepted the cookie.

_key() now raises SessionStoreError when neither the rewrite nor the _revoke() fallback made the revocation durable, and rolls _configured_key back so the epoch stays unactivated. That rollback is deliberate: without it the next call would see the new key already configured, skip the revocation branch, and report success — the same "recorded as done" bug one level down. Because the epoch is not activated, a retry after the permissions are fixed does the revocation for real instead of finding nothing to do.

Propagation, since a stored flag was exactly the thing that failed:

  • converter.main catches it and aborts startup with a readable reason (exit 2) rather than serving traffic — verified in a subprocess: it refuses with the undeletable snapshot and still starts and serves /health when the snapshot is valid under a matching key.
  • AdminMiddleware catches it and answers 503, so if an epoch change degrades after startup, management traffic fails closed rather than open. Confirmed that repeated requests all return 503, not just the first.

The tests use real permissions rather than mocked I/O, per your "distinct storage-failure mode" point: a read-only snapshot file, plus a read-only directory where POSIX allows the unlink (Windows blocks the unlink on the read-only file alone), with the sibling stores writable. test_startup_fails_when_a_superseded_snapshot_cannot_be_revoked covers rotation and disabled-key cases, the failed retry, and recovery after permissions are restored; test_startup_failure_reaches_the_installed_admin covers propagation, and test_a_superseded_snapshot_that_cannot_be_revoked_fails_closed covers the 503. I checked the startup test against the previous _key() behaviour and it fails there, so it does pin the fix.

The old write-failure tests still pass unchanged, so the mocked path and the real-permission path agree.

Verification: tests/test_admin_boundaries.py, tests/test_admin_api.py, tests/test_runtime_endpoints.py, tests/test_inference_admission.py — 127 passed; pnpm test — 132 passed; the full Python suite matches its baseline apart from the pre-existing flaky trial-ledger concurrency test, which also fails on main. I also re-ran the duplicate-method scan, since shadowed tests were the P3 finding: none.

One judgement call worth flagging: I made this fatal at startup rather than degrading, because the alternative — serving with the epoch switch recorded as complete — is what created the resurrection path. That means a permissions problem in the auth directory now stops the gateway from starting. If you would rather it start read-only and simply refuse management traffic, that is a small change and I am happy to switch it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two follow-ups since that reply, both on the same head:

3a8a3e5 — the fixture was not portable, and would have failed on your CI. Your "distinct storage-failure mode" point made me re-check my own helper, and it had the same class of bug I was fixing: it verified the lock by unlinking the snapshot itself. On Windows the read-only file mode blocks that unlink, so the probe doubled as the check; on POSIX the file mode does not affect removal, so it deleted the fixture and the test asserted nothing while still passing. It also re-created the probe after chmod'ing it read-only, which raises on POSIX. The check now runs against a throwaway probe file, the directory is locked only after that check shows the file mode was not enough, and platforms where removal cannot be blocked skip rather than pass silently. Verified against simulated POSIX semantics (unlink gated on directory write permission) and against a platform where every unlink succeeds: the POSIX path locks the directory, keeps the snapshot undeletable, refuses the reconcile and restores the mode; the unblockable path skips. I cannot run Linux here, so that verification is by simulation rather than on your runner — worth watching the CI result for this one.

a5edf2a — merged main. Upstream added the usage-snapshot publish before runtime_management.install, which collides with the SessionStoreError guard in converter.py; the PR had gone CONFLICTING. Both changes are kept: publish cached usage, then install under the guard. It is MERGEABLE again, and tests/test_usage_snapshots.py and tests/test_credential_cooldowns.py from that PR pass alongside the admin tests (198 passed, 1 skipped).

Full-suite note: 937 passed, and the failure set is identical to the pre-change baseline apart from the pre-existing flaky test_trial_rewards concurrency test. One further flake I saw once and chased down rather than dismissing: test_observability.py::test_first_effective_output_not_role_or_usage asserts first_token_ms >= 20 after asyncio.sleep(0.02), and on Windows that sleep quantises to one timer tick — I measured time.monotonic() steps of exactly 16.0 ms and 4/400 samples below 20 ms, and the failure I captured was 15.999999988707714. app/observability.py and tests/test_observability.py are byte-identical to main on this branch, so it is not from this change; flagging it only because it is a real Windows-only flake in a test you run.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Following up on the one caveat I left open: CI has now run on the merged head and the Linux test job is green, so the POSIX path is verified on a real runner rather than only by simulation. Specifically, the three tests that exercise the storage-failure path all report ok on Linux and none of the admin tests skipped — test_startup_fails_when_a_superseded_snapshot_cannot_be_revoked, test_startup_failure_reaches_the_installed_admin, and test_logout_reports_a_revocation_that_could_not_be_persisted. The directory-mode branch in the fixture is therefore exercised for real, which was the part I could not check locally.

All checks on a5edf2a pass: test, web, image, CodeQL (all three languages) and Sourcery. The PR is MERGEABLE; CHANGES_REQUESTED is still standing on your side, so it is yours to clear when you are satisfied.

Comment thread app/admin_auth.py Outdated
Comment thread tests/test_admin_boundaries.py Outdated
Three regressions found while reviewing the persisted session snapshot.

Startup never resolved the key epoch. `_key()` is lazy and only `/admin`
requests reach it, so a restart that served inference traffic alone left the
previous key's snapshot on disk; restarting back to that key adopted it and
resurrected a cookie the rotation was meant to revoke. `install_admin` now
calls `AdminAuth.reconcile()`, so the epoch is resolved during startup,
including when management is disabled.

`logout` discarded the result of `_persist()`. When the session directory is
not writable both the replacement write and the `_revoke()` fallback fail, so
the session stayed valid on disk while `DELETE /admin/session` answered
`authenticated: false` and the client dropped its cookie. The entry is now
kept when the snapshot cannot be rewritten, so a retry can still revoke it, and
`session_delete` answers 503 instead of acknowledging a revocation that did not
happen. The failure is recorded as session-store state, surfaced in
`/admin/settings` as `session` alongside the existing `audit` state, and shown
in the settings page.

The strengthened symlink test was shadowed: `PersistedSessionTests` defined
`test_symlinked_snapshot_is_never_followed` twice, and the later definition
replaced the earlier one, so the case pointing at a valid snapshot never ran.
Both cases are now covered by one test that asserts the target is neither
adopted nor modified.

Tests: restart cases for A -> B -> A and A -> empty -> A that deliberately make
no admin request during the intermediate run, an installed-admin reconcile
check, logout with both writing and unlinking failing plus its retry, the 503
logout path over HTTP, the settings session state, and the WebUI warning panel.
`tests/test_admin_boundaries.py` and `tests/test_admin_api.py` pass (68);
`pnpm test` passes (132). The full Python suite shows the same failures as
before the change apart from the pre-existing flaky trial-ledger concurrency
test, which fails on `main` too.
`_key()` discarded the result of the revocation it performs when the key epoch
changes, so a read-only `admin-sessions.json` made the rotation a silent no-op:
the file survived, the new epoch was recorded as active, and a later start under
the superseded key adopted it. Restarting A -> B -> A with the file left
read-only therefore accepted the original cookie again.

`_key()` now raises `SessionStoreError` when neither the rewrite nor the unlink
fallback made the revocation durable, and rolls `_configured_key` back so the
epoch stays unactivated and a later attempt retries rather than reporting the
work as done. `converter.main` aborts startup with a readable message instead of
serving traffic, and `AdminMiddleware` answers 503 if an epoch change degrades
after startup, so management traffic fails closed rather than open.

Tests use real permissions rather than mocked I/O: a read-only snapshot file
(plus a read-only directory where POSIX allows the unlink) with the sibling
SQLite stores left writable, covering rotation and disabled-key cases, the
failed retry, recovery once permissions are restored, propagation through
`install_admin`, and the 503 fail-closed path. The new startup test fails
against the previous `_key()` behaviour.

Verified end-to-end: `python converter.py` exits 2 with the reason while the
snapshot is undeletable, and still starts and serves `/health` with a valid
snapshot under a matching key. `tests/test_admin_boundaries.py`,
`tests/test_admin_api.py`, `tests/test_runtime_endpoints.py` and
`tests/test_inference_admission.py` pass (127); `pnpm test` passes (132); the
full Python suite matches its baseline apart from the pre-existing flaky
trial-ledger concurrency test.
The fixture verified its own lock by unlinking the snapshot, which removes the
file it is supposed to protect. On Windows the read-only file mode blocks that
unlink and the probe doubled as the check, but on POSIX the file mode does not
affect removal, so the snapshot was deleted and the test asserted nothing while
still passing. It also re-created the probe after chmod'ing it read-only, which
raises on POSIX.

The check now runs against a throwaway probe file, the directory is locked only
after that check shows the file mode did not protect removal, and the probe is
re-created before being protected. Platforms where removal cannot be blocked
skip instead of silently passing.

Verified against simulated POSIX semantics (unlink gated on directory write
permission) and against a platform where every unlink succeeds: the POSIX path
locks the directory, keeps the snapshot undeletable, refuses the reconcile, and
restores the directory mode; the unblockable path skips. Windows behaviour is
unchanged and the test still fails against the previous `_key()`.
Upstream added a usage-snapshot publish before `runtime_management.install`
(PR maiphucgiang#35), which conflicts with the `SessionStoreError` guard added here. Keep
both: publish cached usage first, then install under the guard.
@maiphucgiang
maiphucgiang merged commit 3c0be21 into maiphucgiang:main Sep 20, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants