Skip to content

fix(privacy): declassify takes the write lock before it reads - #294

Merged
Broccolito merged 1 commit into
mainfrom
fix/chip-0e9826a6
Sep 12, 2026
Merged

Broccolito merged 1 commit into
mainfrom
fix/chip-0e9826a6

Conversation

@Broccolito

Copy link
Copy Markdown
Collaborator

privacy::declassify::declassify() opened its transaction with pool.begin() (SQLite DEFERRED) and its first statement was a read. That pins a WAL read snapshot, so the classification_audit INSERT had to upgrade to a writer — and if any other connection committed in between, SQLite refuses the upgrade with SQLITE_BUSY_SNAPSHOT, for which the busy handler is not consulted. The pool's five-second busy_timeout was never in the path. declassify_session maps every Err to INTERNAL_SERVER_ERROR, so a user whose declassification merely lost a race got a bodyless 500.

Step zero: the defect is live on main (5a404ec)

The new test concurrent_declassifications_survive_racing_writes — 120 declassifications of real private chats, barrier-synced against a second, independent store that never stops committing — on unmodified origin/main:

118 of 120 declassifications failed (a `database is locked`, code 5 or its
SQLITE_BUSY_SNAPSHOT variant 517, here means the transaction read before it wrote)

Three runs on the read-first shape lost 118, 120 and 120 of 120.

The mechanism, measured directly

Two raw SQLite connections (WAL, timeout=5.0), B committing between A's BEGIN DEFERRED/SELECT and A's UPDATE:

read-first UPDATE failed after 0.0000s: OperationalError('database is locked') sqlite_errorcode=517

517 is SQLITE_BUSY | (2<<8) — SQLITE_BUSY_SNAPSHOT — and 0.0000s is the busy handler never being asked.

The fix

The one SessionStorage::delete_session already documents: take the write lock up front. Everything declassify must do before it can write is a read — the provenance decides the grade, and the grade decides whether there is a write at all — so the lock is taken by a statement that matches no row and exists only for its lock. Verified with the exact statement shipped:

  • while it is held, a second connection's BEGIN IMMEDIATE blocked for the full 2.2052s timeout → the file write lock really is taken;
  • when another writer holds the lock, the statement itself waited 2.1774s and reported code 5, not 517 → the busy handler is back in the path, so contention costs a wait and never an error.

A BEGIN IMMEDIATE on an acquired connection (the ensure_privacy_schema shape) was rejected deliberately: this runs under an axum handler that is dropped when the client disconnects, and sqlx 0.8.0's pool only pings a returned connection (return_to_pool in sqlx-core-0.8.0), so a hand-rolled BEGIN would hand the pool a connection still holding the write lock. A real Transaction rolls back on drop. sqlx 0.8.0 has no Pool::begin_with.

Before / after

read-first (main) write-first (this PR)
concurrent_declassifications_survive_racing_writes 118, 120, 120 of 120 lost 0 lost, 8 consecutive runs
concurrent commits landing alongside 1139–4245 per run, 0 busy appends

Removing only the new statement and changing nothing else turns both new tests red:

120 of 120 declassifications failed ...
a racing declassification must not error: error returned from database: (code: 5) database is locked

The same-row race improves too

two_declassifications_of_one_chat_serialize_into_one_ledger_row pins the behaviour change. The loser used to hold the same snapshot showing private and surface as a 500; it now parks on the write lock and, by the time it reads, sees public and returns the tidy AlreadyPublic a sequential second call gets. One ledger row either way — the single-ledger-row invariant is now held by serialization rather than by a snapshot conflict. The doc paragraph that described the old behaviour as designed is rewritten to say what is now true.

Deliberately not changed

  • Point 2 of the suggestion — the route's Err → empty 500 mapping. With the lock taken up front a transient conflict now waits up to the five-second busy_timeout instead of failing instantly, so the class of failure that mapping obscured is gone at the source. Reworking the mapping means editing crates/biorouter-server/src/routes/session.rs, a heavily contended file this week, for a case that no longer occurs; left for a separate change.
  • #[serial] coverage in biorouter-server's tests. The suggestion calls it a workaround and it is — the transaction shape was the fix, and the previously-intermittent routes::session::declassify_tests::the_route_needs_more_than_the_secret_key passes here.
  • session_id_high_water and anything in fix(session): a deleted chat's side rows go with it, and its id is never minted again #264: untouched.

Gates

  • cargo fmt --check -p biorouter clean; ./scripts/clippy-lint.sh → all baseline checks passed.
  • cargo test -p biorouter --lib4047 passed, 0 failed (includes all 248 privacy:: tests and concurrent_deletes_survive_racing_writes).
  • cargo test -p biorouter-server --lib declassify → 7 passed, 0 failed.
  • One file touched: crates/biorouter/src/privacy/declassify.rs.

🤖 Generated with Claude Code

`declassify` opened a DEFERRED transaction whose first statement was the
`SELECT privacy_tier, privacy_reason, provider_name`. That pins a WAL read
snapshot, so the `classification_audit` INSERT had to upgrade to a writer,
and any commit landing in that window makes SQLite refuse it *instantly*
with SQLITE_BUSY_SNAPSHOT. The busy handler is not consulted for that
error, so the pool's five-second `busy_timeout` was never in the path.
`declassify_session` maps every `Err` to a bodyless 500, so a user whose
declassification merely lost a race was told nothing at all.

Measured against two raw SQLite connections (WAL, timeout=5.0): B commits
between A's BEGIN DEFERRED/SELECT and A's UPDATE, and A fails in 0.0000s
with errorcode 517 — SQLITE_BUSY_SNAPSHOT.

The fix is the one `SessionStorage::delete_session` already documents:
take the write lock up front. Everything this function does before it can
write is a read (the provenance decides the grade, the grade decides
whether there is a write at all), so the lock is taken by a statement that
matches no row and exists only for its lock. Verified with the exact
statement shipped: while it is held, a second connection's BEGIN IMMEDIATE
blocked for the full 2.2s timeout, and when another writer holds the lock
the statement itself waits 2.18s and reports code 5, not 517 — the busy
handler is back in the path.

A `BEGIN IMMEDIATE` on an acquired connection was rejected deliberately.
This runs under an axum handler that is dropped when the client
disconnects, and sqlx 0.8's pool only pings a returned connection, so a
hand-rolled BEGIN would return a connection to the pool still holding the
write lock. A real `Transaction` rolls back on drop.

Fail-before, in `concurrent_declassifications_survive_racing_writes`: 120
declassifications barrier-synced against a second store that never stops
committing. On origin/main it lost 118, 120 and 120 of 120 across three
runs; with the fix, eight consecutive runs lost none, against a measured
1139-4245 concurrent commits. Removing only the new statement turns both
new tests red.

The same-row race improves too, and
`two_declassifications_of_one_chat_serialize_into_one_ledger_row` pins it:
the loser used to surface as a 500 and now parks on the lock and returns
AlreadyPublic. One ledger row either way — by serialization now rather
than by a snapshot conflict. The doc comment that described the old
behaviour as designed is updated.
@Broccolito
Broccolito merged commit adabf9f into main Sep 12, 2026
16 checks passed
@Broccolito
Broccolito deleted the fix/chip-0e9826a6 branch September 12, 2026 18:30
Broccolito added a commit that referenced this pull request Sep 14, 2026
…r chat's failure report

Repairs the D3a regression an independent tester found in this PR's repair
round. Renderer-only: no Rust changed, and neither the write-first lock
ordering (#294) nor who may declassify is touched.

Reproduced in the dev app first (sandboxed config, this branch's daemon):
chat 20260809_21 failed with sessions.db write-locked (POST -> 503, "The chat
store was busy"), then 20260809_23 failed the same way and one toast was still
on screen. Retrying 20260809_23 answered 200: "Chat marked public" was the only
toast, and 8 s later there were none, with the DB reading _21 private and _23
public.

Cause: DeclassifySessionDialog keeps outstanding failure reports by chat, but
toastError deduplicated them by title + message, and the busy sentence is the
same for every chat. Both reports were one toast id, so _23's retraction
dismissed _21's report too. A different failure on one chat, or a public row
read about one chat, did the same.

Fix: toastError takes an optional dedupeScope, and the dialog raises each
report under declassify:<session id>, with a title that names its chat
("The chat store was busy — chat 20260809_21" for a placeholder name,
"— “Subagent delegation request”" otherwise). Identical failures on the same
chat still share an id, so a retry replaces its report instead of stacking.

Rejected alternative: reference-counting the shared id. It keeps _21's report
alive, but as one toast standing for several chats and saying "this chat". After
_23 succeeded, it would sit beside "Chat marked public" still saying "this chat
was not marked public", about a chat it doesn't name.

Measured after the fix, same steps and chats: two toasts, each naming its chat.
After _23's 200, _21's report was still on screen at +8 s and +30 s, and _23's
was gone. A second busy failure on _21 left one _21 toast, not two. _21's own 200
then retracted it, and no toasts were left at +8 s.

Tests: DeclassifySessionDialog.toastLayer.test.tsx (new, 6) renders the real
toasts.tsx and react-toastify container, with only the exit animation replaced,
because jsdom runs none. On the unfixed code 5 fail ("expected [] to have a
length of 1 but got +0", "expected [ Array(1) ] to have a length of 2"). The
same-chat retry guard passes before and after.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant